Note, added November 2026: This post was written before we fixed how benchmark statements are grouped. Until 21 October 2026, statements that read alike were merged even when they were about different models, tasks or snapshots, so a figure here may belong to a different model, task or snapshot than the one named beside it, and some sources may come from other snapshots. We have left the post as it was published. Current figures are on the benchmark.

Set a quality bar, not a leaderboard: the catalyst-analysis model that clears it for $0.02

If you build on LLM APIs, the most useful number in a benchmark is rarely the top score. It is the cheapest model that lands within a stated distance of that score. On financial catalyst and scenario analysis, the cheapest model that clears a 90%-of-best bar costs less than half of what the leader does per run, and the benchmark’s own head-to-head judging says the field below the top is too tight to call anyway.

Decide what good enough means first

The benchmark’s routing rule is simple. A model qualifies on a task when its quality is within 90% of the best measured model; on some tasks that bar is raised to 95% and on others lowered to 80% [^1][^2], with the bar set per task and stated. Every score behind it is one judge grading one output against a published rubric on a 0–10 scale, and where one model is reported to have beaten another, both answered the same live input and were scored independently [^1][^2].

That framing turns model choice into a threshold question rather than a ranking question. You decide what quality your product needs, and then pay for exactly that.

The catalyst-analysis numbers

On analyzing catalysts and scenarios, Gemini 3.5 Flash scores 8.8 out of 10 (±0.17) and costs $0.05 per task run in sync mode, measured on this model’s own usage and repriced at current rates [^3][^4]. That is the best measured quality on this task point.

Set the bar at 90% of best, and 90% of 8.8 is 7.92. The cheapest measured option at or above that line is GPT-5.4 Nano at 7.94 out of 10 (±0.21) for $0.02 per task run in sync mode [^3][^4]. Thinking Machines Inkling also clears it at 7.99 out of 10 (±0.42) and costs $0.03 per task run in sync mode, measured on this model’s own usage [^3][^4].

The gap between best quality and cheapest qualifier is 0.86 points on quality and $0.03 on quoted per-task price [^3][^4]. That is a material quality step for less than half the sync price, so the choice is a real one, not a rounding error. Just below the line, GLM-5.3 Flash scores 7.83 out of 10 (±0.38) and costs $0.0063 per task run in sync mode [^4], missing the bar by 0.09 points while being the cheapest near-miss, which is a reminder that where you set the bar matters as much as which model tops the table.

Why the scores overstate the gap

Head-to-head judging on the same catalyst work does not resolve the field below the top into a clear winner. The top two answers scored exactly the same in 20.0% of 30 judged occasions, and finished within half a point in 73.3%, with a median gap of 0.2 on a 10-point scale [^3][^4]. When the leader and the runner-up are that often indistinguishable on a single live input, an average difference between models is real but small relative to the spread between individual answers.

There is also a length effect worth knowing about. Judges in that work preferred longer answers, with the answers judges picked longer than the ones they passed over, at 4,745 against 4,177 output tokens [^3][^4]. Longer outputs are also more output tokens, so answer length sits on both sides of the quality-versus-cost trade.

Check the execution mode before comparing prices

Sync and batch prices are not like-for-like, and a bar-based route should be priced in the mode you will actually run. Running Claude Opus 5 on that same catalyst work through batch costs $0.07 per task against $0.15 synchronously, 50% less [^4]. Only 50 of 68 benchmark tasks are batch-eligible [^1][^2], so for the rest any batch-priced comparison would be fiction.

Price on usable output, not tokens billed

One more thing a leaderboard hides is that some of what you pay for never becomes an answer. On a neighbouring task in the same category, producing trading recommendations spent $25.16 and $15.30 of it, 60.8%, bought nothing: the tokens were generated, billed, and could not be parsed [^5][^6]. A cheap model with a high parse-failure rate is not cheap, so measure cost per usable result rather than the quoted per-task price before you commit to a route.

Where to look next

The numbers above are task points from one snapshot, and the catalyst ranking should not be extended to filing extraction, trading recommendations or reference-heavy writing without checking those tasks directly. Check the live numbers at https://fronset.ai/benchmark/ before you commit a route, and the per-task pages such as https://fronset.ai/benchmark/task/catalyst-and-scenario-analysis/ for the current scores, intervals and prices. Set the quality bar your system actually needs there and compare batch against sync pricing yourself.