Most Models Get It Right First Try. Budget for the Rest.

For eleven models measured on the Fronset benchmark, the share of attempts that return usable output with no repair or retry ranges from 95.7% to 99.8% [^1]. On that measure the models are close to tied. What a pipeline needs to watch is what happens to the rest: for two models, some outputs needed a second, separately billed repair call that their price cards do not mention.

Usable, salvaged or repaired

An attempt is usable as returned when the model’s output can be used with no repair and no retry. When output comes back but needs work, the fix takes one of two forms. Salvage corrects the output locally, without another provider call. Repair is a second, separately billed call, which adds real cost and latency that a headline price does not show.

First-try rates sit in a narrow band

At the top, Meta Muse Spark 1.1 and Meta Muse Spark 1.3 contributor both return usable output on 99.8% of attempts, across 2,871 and 40,720 attempts respectively [^1]. NVIDIA Nemotron-3 Nano 30B-A3B reaches 99.4% over 143,949 attempts [^2], and Tencent Hy3 99.1% over 26,758 [^3]. At the bottom of the list are DeepSeek V4 Pro at 96.5% and Qwen 3.6 Plus at 95.7% [^1].

No intervals are supplied, so the models from Claude Sonnet 5.5 at 97.9% [^4] up to 99.8% cannot be told apart. The two lowest figures sit close to that group rather than clearly below it.

Two limits apply. These are each model’s own record and are not paired with quality scores on the same tasks, so they cannot be used to choose a model on quality. Nor are they a schema-by-schema check: they support the statement that output was usable as returned, not that it was clean against a particular structured contract.

Where the repair calls are

Two models show the repair path directly. Thinking Machines Inkling Small needed a separately billed repair call on 2.9% of 9,639 attempts, on top of 1.0% fixed locally [^5]. Gemini 3.1 Pro Preview needed one on 1.7% of 15,389 attempts, on top of 1.2% fixed locally [^1]. Neither price card mentions this.

The two were not measured on the same workload, so they cannot be ranked against each other. Even so, a cost comparison built on token prices alone will understate the bill for both. The billed unusable tokens and the added latency do not appear in the headline figure, so a route that is cheap per task can be expensive end-to-end.

Beyond those figures, the evidence is thin. It gives no repair cost or repair latency for any model, does not say whether a repair is performed by the same model or another one, and reports no fallback outcomes. It also does not show that any higher-priced model avoids enough repair to be cheaper end-to-end.

What a first-try figure misses

First-attempt usability says nothing about how long an attempt takes. Tencent Hy3 is uneven on speed in claim-consistency classification: over 194 calls its median response was 40 seconds, but its 90th-percentile response was 171 seconds [^6]. That is the kind of tail latency a first-attempt figure does not capture.

Designing the pipeline around it

What follows is interpretation of the evidence, not a measured result. For workflows that cannot absorb a retry, filter on first-attempt usability before anything else and treat it as a gate rather than a verdict. Because no intervals separate the top band, cost and latency are the legitimate tiebreakers within it.

On structured paths, validate parsing against your own output contract and count only output that is usable as returned. The benchmark figure does not do that check for you.

Then model the whole completion path rather than the headline price. Where a repair call is separately billed and absent from the price card, a per-task figure understates both cost and elapsed time, and the size of that understatement has to be measured directly.

Check the live benchmark at https://fronset.ai/benchmark/ to set the first-pass bar your workflow actually needs, and compare batch against sync pricing alongside each model’s task results before routing production traffic.