When the Model Doesn't Matter, and When It Really Does

On some tasks in the Fronset Benchmark, the model you choose barely changes the result; on others, it changes it a lot. Language identification is the first kind: several leading models tie near a perfect score [^1]. Content domain suggestion is the second: scores are low and the best answers diverge widely, so it is the task where testing candidates on your own inputs pays off most [^2].

The same models on two tasks

Three models measured on both tasks show the contrast. Scores are out of 10, and the ± figure is the interval reported around a mean score; where two intervals overlap, the gap is a tie.

Only the task changed between the two columns. No interval is given for these content domain scores, so the gaps between them should not be read as a ranking. What the table does show is that how much your choice of model matters depends first on the task.

Near the ceiling, models tie

On language identification, the intervals of the two Gemini models overlap, so they tie, and the same holds for GLM-5.3 at 9.9, ±0.22, and GLM-5.3 Flash at 9.77, ±0.24 [^1]. A separate comparison looks at the two best answers on each judged occasion, meaning one input on which several models’ answers were scored side by side. On this task, those two answers scored exactly the same in 100.0% of 62 judged occasions [^1].

Part of this is the scale itself. A score out of 10 cannot rise above 10, so when several models sit near the top, little room is left for a difference to show.

When quality is this close to the ceiling, cost and run mode (batch or sync) are what separate the options. GPT-6 Luna costs $0 per task run in batch mode and $0.0001 per task run in sync mode [^1]. These are the only cost figures reported for that model.

Far from the ceiling, the choice matters

Content domain suggestion is the opposite case. Claude Sonnet 5.5 scores 5.01, ±0.3, and no model reported here does well on absolute score [^2]. Two separate results for the task also show the best answers pulling apart.

In one, the top two answers scored exactly the same in 12.7% of 63 judged occasions, finished within half a point in 47.6%, and had a median gap of 0.8 [^2]. In the other, they scored the same in 8.0% of 87 judged occasions, finished within half a point in only 20.7%, and had a median gap of 1.9 [^3]. The two results differ in magnitude but point the same way.

Win shares are also high for some models and zero for others. Thinking Machines Inkling Small was judged the best answer in 28 of the 33 occasions it competed in, and meta/muse-spark-1.1 in 39 of 45 [^3]. Tencent Hy3 was chosen best in none of 41 judged occasions [^3]. These come from different pools of occasions, so they do not rank the first two models against each other.

How to choose

Before comparing models on a task, check whether it is saturated: the leading models’ intervals overlap near the top, and their best answers usually score the same. If it is, quality will not separate the options, so decide on cost and run mode.

If the task is spread out, test the candidates on your own inputs before committing. Look at how often a model’s answer is judged best, not only at its average, and do not carry a model’s standing on one task over to another. A stricter quality bar, a different run mode or a tighter latency budget would change these answers.

The live benchmark at https://fronset.ai/benchmark/ lets you set the quality bar you need and compare batch against sync pricing yourself. The method is documented at https://fronset.ai/benchmark/methodology/.