8.95 on One Item, 5.69 on a Set: The Relevance-Scoring Cliff

Models that score the relevance of a single item well do much worse when asked to score a set of content or a batch of short posts. Qwen 3.7 Plus tops the single-item list at 8.95 out of 10 but scores 5.69 on content sets,[^1][^2] and no measured model reaches 7.5 on sets.[^2] If your pipeline scores many items at once, a single-item benchmark result tells you little about how it will perform.

Same models, three relevance tasks

The Fronset benchmark splits relevance scoring into several different tasks. Three are compared here: scoring a single item of generated content, scoring a set of content, and scoring a batch of short posts. Scores are quality ratings out of 10 from independent judgments, each with a stated ± range; where two ranges overlap, the scores do not separate the models.

On single items, the leading models sit in overlapping ranges that do not separate them. Their best answers often coincide: the top two answers tied exactly in 63.4% of 41 judged occasions, meaning inputs on which several models’ answers were judged.[^1]

Sets are a different picture. The highest content-set figure is Xiaomi MiMo V2.6 Pro’s 6.97 (±0.38), and NVIDIA Nemotron-3 Nano 30B-A3B’s 2.5 (±0.36) is the lowest result reported in this category.[^2] Every model in the table that is measured on both tasks scores lower on sets than on single items.

Batches of short posts are the second weak area. MiniMax M3 has the highest figure and the narrowest interval, 7.29 (±0.12), and the rest of the reported field sits below it, down to Qwen 3.8 Flash at 5.44 (±0.47).[^3]

The ranking reshuffles, not just the scores

The drop is uneven, so the order of models changes too. Qwen 3.7 Plus goes from the top of the single-item list to the bottom of a four-model comparison on content sets, where it ties with MiniMax M3 and Gemini 3.8 Flash and Xiaomi MiMo V2.6 Pro is clearly ahead.[^2] DeepSeek V4.1 Flash holds up better, at 8.59 (±0.35) on single items and 6.94 (±0.41) on sets, next to the highest content-set figure.

Even the two multi-item tasks disagree with each other. MiniMax M3 ties at the bottom of that content-set comparison at 5.46 (±0.28), yet has the highest figure on short-post batches.[^2][^3] A good result on one multi-item form is no guarantee on the other.

Price does not buy a way out. On short posts, GLM-5.3 Flash at 6.81 (±0.32) for $0.04 and GLM-5.3 at 6.42 (±0.4) for $0.40 are a tie, and Claude Opus 5.5 shows no measured quality advantage for its price on this work.[^3]

What it means for a pipeline

The benchmark reports these drops but not their cause, so the safe reading is a narrow one. Ability on the strong form of relevance scoring does not transfer to the weak ones, and on the weak ones the open question is not which model leads but whether any model clears a high bar at all.

Evaluate the form you deploy. If your pipeline scores a set of content or a batch of posts in one call, a model’s single-item score is not evidence for that job, and the single-item leader may be a poor choice for it.

Set a bar the task can support. The highest quoted figure is 6.97 on content sets and 7.29 on short-post batches, so an application that needs more on multi-item relevance will not find it among the measured models.[^2][^3] Plan review or fallback for that work from the start, instead of assuming a model will reach a single-item standard.

If your design allows scoring items one at a time, that is the form where the measured scores are high. The material reports no test of whether splitting a set into single items recovers that quality, so treat it as a hypothesis and check it on your own data.

Current figures for all three tasks are on the live benchmark at https://fronset.ai/benchmark/, where you can set the quality bar your application needs for the exact task.