The Best Summarizer Is Not the Best Synthesizer
The Best Summarizer Is Not the Best Synthesizer
On the Fronset benchmark, the model with the highest quoted score on structured content summarization falls back on the neighbouring summarization and synthesis tasks, and models from lower in that ranking lead them instead. A ranking from one of these tasks does not carry over to another, even inside the same category. Pick a model for executive summaries or decision synthesis from a summarization leaderboard and you can end up with the wrong one.
The summarization leaders slip
Structured content summarization asks a model to turn a document into a short faithful summary. Scores are mean quality ratings out of 10 across independent judgments, and the ± figure is the stated margin around that mean. Models clearing about 8.5 on this task form its top band, and Meta Muse Spark 1.3 holds the highest quoted score: 8.97 (±0.16) after 69 independent judgments [^1].
That lead does not hold elsewhere in the category. Meta Muse Spark 1.3 scores 7.72 (±0.38) on executive summaries [^2], 8.13 (±0.18) on multi-perspective decision synthesis [^3] and 7.98 on analytical writing that keeps source references intact [^4].
Tencent Hy3 tells the same story. Its structured summarization result is both cheap and in the top band: 8.69 (±0.17) at $0.0034 per task run in sync mode, meaning ordinary calls rather than batch [^1]. On decision synthesis it drops to 7.71 (±0.25) [^3], and on reference-preserving writing to 7.29 [^4], one of the weak results on that task.
Mid-ranked summarizers lead elsewhere
The reversal also runs the other way. GLM-5.3 is a mid-band result on structured summarization at 8.36 (±0.24) [^1]. In 40 judged occasions, head-to-head comparisons in which reviewers pick the best answer, it was never once chosen [^1]; that shows it was not the top answer when measured, not that it summarizes badly. Yet it leads executive summaries at 8.6 (±0.18) [^2] and ties the top of decision synthesis at 8.59 [^3].
GLM-5.3 Flash follows suit. It scores 8.46 (±0.18) on structured summarization [^1], just outside the top band, but leads reference-preserving analytical writing at 9.22 (±0.2) for $0.04 per task run [^4]. That puts it in a three-way tie at the top with Grok 4.7 and Kimi K3 [^4], and makes this one of the few places in the category where a cheap model is at the front rather than merely acceptable.
One category, several different jobs
The results do not explain why the rankings diverge, only that they do. The practical consequence is that this category has no champion: summarizing one document, writing an executive summary, weighing several perspectives into a decision and writing analysis with intact references each rank the models differently. A model strong on summaries is not automatically strong at synthesizing.
Read the gaps with one caution. The top-band figures come from separate result sets, and where margins overlap the evidence supports a tie rather than an order, so no model is shown leading across all of this work. The task-to-task disagreements above are still large enough to change a routing decision.
The same rule holds one step outside the category. Structured extraction scores higher than summarization for the same models: Xiaomi MiMo V2.6 Flash reaches 9.76 (±0.22) on extraction [^5] against 7.98 on structured summarization [^1]. These are different tasks, and extraction results should not be assumed to carry over.
What to do about it
Treat each task as its own routing decision. Look up the results for the task your pipeline actually runs, and never carry a structured-summary ranking onto another task in this category.
When the leaders on your task tie on overlapping ranges, let price and batch availability decide. On reference-preserving writing, for example, the three-way tie means price decides, though recent declines on that task are worth re-checking before you commit.
If one model has to serve several of these tasks, test it on each of them with your own content. Confirm completion rate and latency there too before fixing it as a production default.
The live task-level numbers are at https://fronset.ai/benchmark/, where you can set the quality bar your workload actually requires and compare batch against sync pricing yourself.