Judges Don't Simply Reward the Longer Answer
When judges (graders that score one output at a time against a published rubric [^1]) pick the best of several answers to the same inputs, length tilts the choice, but not in one consistent direction. In one set of 38 comparisons in the Fronset benchmark, the answer the judges picked was longer than the ones they passed over in 23 tasks and shorter in 15. A judged preference therefore can’t be read as a clean measure of quality.
The tilt runs both ways
Scores run from 0 to 10 and are absolute, not head-to-head [^1]. A “picked” answer is the one judges chose as best among several answers to the same inputs, and its length in output tokens is set against the answers passed over. These results rest on 46,711 scored judgments across 906 measured model-and-task combinations, covering 27 models on 54 tasks.
The gaps can be wide in either direction:
| Task | Picked (output tokens) | Passed over (output tokens) | Picked answer was |
|---|---|---|---|
| Profiled document analysis | 6,422 | 3,405 | longer [^2] |
| Filings analysis | 2,230 | 1,549 | longer [^3] |
| Atomic fact claim extraction | 4,315 | 2,975 | longer [^4] |
| Translation | 1,150 | 1,699 | shorter [^5] |
| Query validation | 566 | 845 | shorter [^6] |
| Language identification | 86 | 196 | shorter [^7] |
No single length rule transfers across this work. The direction has to be read task by task.
Near-tied scores, same tilt
Batch text translation is the one comparison here that measures how far apart the top answers scored. The top two answers scored exactly the same on 40.0% of 70 judged occasions and finished within half a point on 91.4%, with a median gap of 0.1 on the 10-point scale [^8]. The benchmark concludes that on this task the choice of model barely moved the result.
The picks on the same task still leaned shorter, at 1,856 output tokens against 2,581 [^8]. One reading is that a preference for shorter answers was operating among answers that scored almost identically. That reading is interpretation, not a measured result, but it does suggest that on near-tied work a judged pick tells you little.
What the evidence does and doesn’t show
The evidence supports saying that output length is a confound in judged selections and that its direction is task-specific. It does not support “judges reward verbosity” or “judges reward brevity”. Either slogan would describe some of the tasks above and contradict others.
The limits matter as much as the finding. The evidence doesn’t say how much of any model’s measured quality the length tendency accounts for. It also doesn’t show whether a length-controlled comparison would change any of these picks, and that question remains open.
What to do about it
Read a judged win next to output length, not instead of it. Where judges leaned toward longer answers, such as filings analysis and atomic fact claim extraction, a verbose model’s edge may partly reflect length. Check whether the extra length carries information before treating the pick as a quality signal. Where judges leaned shorter, such as translation and query validation, the scoring points toward concision.
For close margins, treat a gap smaller than the reported interval as a tie. On near-tied work like batch text translation, don’t let a judged pick settle the choice on its own. A team with a strict quality bar should compare candidates on length-controlled samples of its own work before committing.
The live benchmark at https://fronset.ai/benchmark/ lets you set the quality bar you need, since the figures here are one reading of a bar you may want to move. Method is documented at https://fronset.ai/benchmark/methodology/.