In LLM Benchmarks, an Empty Cell Is Not a Weak Model
In LLM Benchmarks, an Empty Cell Is Not a Weak Model
The Fronset benchmark describes individual model-and-task cells in considerable detail and the grid around them hardly at all. Most model-and-task pairings hold no published result, and the results do not say why. Before you rely on a number, or read meaning into a missing one, check that the cell you care about actually exists.
The measured grid and the possible one
The current results rest on 46,711 scored judgments across 906 measured model-and-task combinations, covering 27 models on 54 tasks. Those 27 models across 54 tasks imply a possible grid of 1,458 pairings. That 1,458 is arithmetic on the scope line, not a coverage rate: nothing in the results establishes that every unmeasured pairing was feasible or eligible for measurement.
Most published results cover a single model on a single task, and cross-model comparison on the same task is rare. Four models share language identification [^1] and three share claim continuity classification [^2]. Beyond those, shared work is scarce.
Appearing once is not the same as measured once
Apart from a handful of models that turn up on several kinds of work, most named models appear once. DeepSeek V4.1 Flash appears only on voice profile generation [^3], Claude Haiku 5.5 only on language identification [^1], and GLM-5.3 Flash only on topic section assignment [^4]. The figures do not state the full set of tasks each model was run on, so “appears once here” is not the same claim as “measured on one task only”.
Tasks have the same problem from the other side. Publication title package generation, voice profile generation and topic section assignment, among others, each carry a single model, which yields a number with nothing to compare it against. Labels need care too: “structured summarisation” and “structured content summarisation” are worded differently, and the evidence does not confirm they are the same work.
Whole measures are missing too
Latency appears only twice. Tencent Hy3 on claim continuity classification has a median response time of 40 seconds but a 90th percentile (the time within which 90% of calls finish) of 171 [^2], and Qwen 3.7 Plus on content summarisation has a median of 52 seconds and a 90th percentile of 178 [^5]. Those are the only tail figures available, and for interactive use the tail matters more than the median.
Cost is absent entirely: these figures carry no per-task cost for any model, and no observation dates, repricing notes or estimator flags that would show whether a price is freshly observed, repriced or modelled. No pairing carries both a batch and a sync price. The only cost-adjacent finding is that on 17.2% of MiniMax-M2.5’s attempts, a repair triggered a second, separately billed provider call that its price card does not mention [^6].
Why a gap is not a verdict
A missing cell could mean a pair that was unavailable, not run, or run and not published, and nothing in the results says which. So a model’s absence from your task is not evidence that it would do badly there; it is only an absence of evidence. For the same reason, the results cannot rank all 27 models across all 54 tasks, or name any model as broadly or narrowly measured across the set. Nor can a score from a neighbouring task be carried over to fill the gap.
Check the cell before you rely on it
Three habits keep the gaps from turning into wrong conclusions:
- Find your exact cell. Look up your model on your task, under the task’s exact label, before relying on any figure. If the cell is missing, the benchmark has no answer for you, not a negative one.
- Treat a single-model task as one data point. A number with nothing beside it cannot tell you whether another model would do better or worse.
- Measure what isn’t published. Where your decision depends on price, speed, or a task that appears for only one model, these figures cannot settle it; run the workload on your own material before committing.
The measurement behind these figures is continuous and ongoing, so check whether your cell exists now in the live figures at https://fronset.ai/benchmark/.