Alibaba released Qwen 3.8-Max this week and marketed the preview as second solely to Claude Fable 5 (their launch-day table was extra equivocal: the mannequin leads on considered one of 12 coding-agent rows). However an unbiased harness got here near the alternative conclusion: a benchmark run, apparently utilizing the Preview model, put Qwen 3.8-Max's finest effort setting mid-pack, and its default setting final.
Each outcomes are actual and defensible. The hole between them is about token and time budgets, and that issues as a result of these figures aren’t normally headline numbers. Alibaba's footnotes give its coding numbers a five-hour timeout, and as much as 12 hours per run on PaperBench. The unbiased harness, VulcanBench, allowed between 45 and 60 minutes of wall clock time. A time price range between 5 and 16 occasions bigger on Alibaba’s facet explains the massive distinction in outcomes.
It’s time to do two issues to start out accounting for these variations when selecting fashions. First, the metric to make use of is price per profitable process: complete spend, together with every part you spent on makes an attempt that failed, divided by the duties that really handed your acceptance verify. Second, you might want to make time or token budgets an express a part of your acceptance standards, not a hidden element.
Value per token has stopped predicting the invoice
The comparability everybody printed in Qwen 3.8-Max's first week was a value comparability, as a result of that was the one knowledge out there. It isn’t an affordable mannequin. DeepSeek-V4-Flash-0731, which entered public API beta on July 31, lists at 14 cents per million input tokens and 28 cents output. Qwen 3.8-Max lists at $2 and $6. Kimi K3 sits at $3 and $15.
These costs let you know lower than they used to, for a cause particular to reasoning fashions like Qwen: attending to a consequence prices pondering tokens. A mannequin that spends most of its token allowance on reasoning can attain a token cap earlier than it writes the reply, supplying you with an empty consequence indistinguishable from a complete failure at the price of a full run.
Synthetic Evaluation has the cleanest published measurement of how this may have an effect on actual agent spend: operating its Intelligence Index on DeepSeek-V4-Flash at most effort took 210 million output tokens in opposition to a category median of 100 million. Absolute price stayed low anyway, as a result of the tokens have been so low cost. However verbosity prices time, not simply cash, and relying in your use case that may sink you.
What you want is a quantity that counts every part you spent, together with the makes an attempt that got here again empty, in opposition to the duties that really acquired completed within the time and token price range you specified. That is what a cost-per-success metric helps you see.
Your failure price is partly a configuration setting
A run that produces a fallacious reply and a run that runs out of price range are completely different occasions with completely different fixes. Virtually no harness distinguishes them, and nearly no leaderboard reviews the cut up. I hit this constructing an agent benchmark of my own: the harness logged a failure and nothing about why, and I had so as to add the excellence myself. Whenever you do separate them, price range exhaustion seems to dominate.
Long-Horizon-Terminal-Bench, printed in July, ran 17 frontier fashions throughout 46 duties by a shared harness with one 90-minute try every. Timeouts accounted for 79% of unresolved runs, in opposition to 19% for brokers that stopped on their very own and three% for harness errors. The authors are cautious about what that does and doesn’t imply: the timed-out runs weren’t near ending, with imply reward between 0.10 and 0.35, so you can’t assume extra time would have resulted in success. However the lesson is: benchmarks are implicitly measuring time effectivity, whether or not or not they shout about that.
The clearest printed instance of the mechanism comes from VulcanBench, the identical open-source harness behind the Qwen chart. In a report dated July 26, Claude Opus 5's lowest-effort setting was its finest, fixing 20 of 23 duties in opposition to 18 at excessive effort. The additional reasoning wasn’t ineffective: excessive effort returned the fewest fallacious solutions of any setting, one in opposition to three. It ran out of clock as a substitute, and a timeout scores zero. Two of its three regressions have been cutoffs on duties that low effort solves, and given limitless time on each it solely ties its least expensive setting, at 3.1 occasions the fee.
That has a direct consequence for anybody constructing a routing ladder. The usual design escalates to extra reasoning when an affordable try fails, on the belief that the subsequent rung is healthier and merely prices extra. For a significant share of mannequin and process mixtures that assumption is fallacious, and also you pay the upper rung's value to escalate right into a timeout or hitting a cap.
Who’s already measuring this
A number of teams have landed on price per profitable process independently in the previous couple of months, which is the strongest sign it's turning into commonplace.
VulcanBench reviews {dollars} per solved process as a headline column and has since its earliest reports. Lengthy-Horizon-Terminal-Bench publishes per-task price subsequent to accuracy, and its most instructive row is GPT-5.4 at roughly $26 per process with a a lot decrease move price than Grok 4.5 at about $11. TestEvo-Bench runs brokers underneath a price cap, and Claude Code's test-generation rating falls from 71% to 44% on the tighter cap.
Distributors are already on board with the thought of measuring per profitable process. HubSpot moved its Breeze Buyer Agent in April to 50 cents per resolved conversation, down from $1 per dealt with dialog. Zendesk bills per automated resolution. Fin costs 99 cents per outcome and payments solely on end-to-end decision.
What to alter this week
-
Emit a failure cause on each agent run as a required area, with price range exhaustion, verifier failure and harness error as distinct values somewhat than one failure flag. Till you’ll be able to separate a timeout from a fallacious reply, your move price is measuring two issues directly and you can’t inform which one to repair.
-
Compute price per profitable process per effort stage, not simply per mannequin. Whole spend together with failed makes an attempt, divided by duties that handed your acceptance verify. The rating won’t match the speed card, and the most cost effective setting could effectively win.
-
Cap on tokens somewhat than wall clock until latency is genuinely in your service stage goal. A wall-clock cap scores your supplier's serving velocity as mannequin high quality.
-
Verify the default effort setting on every part you could have deployed. Qwen 3.8-Max runs at its highest reasoning setting when the effort field is unset, and its highest setting was its worst performer in unbiased testing. A group that by no means touches that parameter is operating the configuration that prices essentially the most per solved process.
