Journal — Fronset Benchmark

Moving Up a Model Family Is Not a Reliable Upgrade

Sun, 01 Nov 2026 21:30:24 +0000

Moving Up a Model Family Is Not a Reliable Upgrade

When a small model’s output falls short, the usual fix is to switch to its larger sibling in the same family. On the infrastructure and utility jobs in this benchmark, that step sometimes buys a clear gain, often buys nothing measurable, and in one case points the wrong way. The larger sibling is not necessarily slower, either.

So a model family should be treated as a set of separate candidates rather than a ladder you climb until the output looks right. Every step has to be checked on the job it will actually run.

A Model's Best Task Says Little About Yours

Sun, 01 Nov 2026 21:30:23 +0000

A Model’s Best Task Says Little About Yours

Claude Opus 5.5 scores 3.75 points higher, out of 10, on turning source material into structured fields than on scoring relevance across a whole content set [^1][^2]. Tencent Hy3 goes from 9.85 on identifying a document’s language to 6.92 on rewriting metadata paragraphs [^3][^4]. Same model, same benchmark, different job.

So a model’s reputation, its headline result or any ranking that blends tasks is a weak guide to how it will handle your work. The figure that bears on the choice is the model’s score on the task you are routing, set against the other models measured on that task. This matters most when one model is chosen for a whole pipeline because it did well on the first step someone checked.

When Claude Opus 5.5 Earns Its Price Over Claude Sonnet 5.5

Sun, 01 Nov 2026 21:30:19 +0000

When Claude Opus 5.5 Earns Its Price Over Claude Sonnet 5.5

Here are three kinds of work where the Fronset benchmark scored Claude Sonnet 5.5 and Claude Opus 5.5 on the same task. On two, the scores are tied and Claude Sonnet 5.5 costs far less. On the third, Claude Opus 5.5 is clearly better, and it is the only one of the three where the extra spend buys quality.

Relevance Scores Depend on How the Job Is Framed

Sun, 01 Nov 2026 21:30:18 +0000

Relevance Scores Depend on How the Job Is Framed

A model that judges the relevance of a single piece of content well can score far lower when asked to score a whole set, and the model at the top changes with the framing. On this benchmark, relevance scoring is three different jobs with three different ceilings.

That makes the framing your product uses a design choice with measurable consequences. A relevance leaderboard measured on one framing tells you little about another, so benchmark the one you will ship.

Cheap Models Fill Structures Well; No Model Extracts Claims Well

Sun, 01 Nov 2026 21:30:15 +0000

Cheap Models Fill Structures Well; No Model Extracts Claims Well

On turning source material into structured facts, Xiaomi MiMo V2.6 Flash at $0.0015 per sync run ties Claude Sonnet 5.5 at $0.07 [^1]. On pulling individual factual claims out of free text, no model measured scores above 7.7 out of 10 [^2], and several models that score above 9 on filling structures fall below 7.

The benchmark files both jobs under structured data and fact extraction, but a model’s score on one says little about the other. Reading them as one leaderboard leads to two mistakes: overpaying for structure-filling, and trusting a claim extractor that has not earned it.

GLM-5.3 Flash Ties Grok 4.7 for a Third of the Price, When It Answers

Sun, 01 Nov 2026 21:30:11 +0000

GLM-5.3 Flash Ties Grok 4.7 for a Third of the Price, When It Answers

On reference-preserving analytical writing, the most heavily measured long-form task in the Fronset benchmark, three models tie at the top. The cheapest of them, GLM-5.3 Flash, costs a third of what Grok 4.7 does per run, but its score comes from only 52% of attempts [^1].

That moves the decision away from quality, which is tied, and onto whether a call returns an answer at all, which the score does not show. The task is one component of long-form work: no result in the benchmark measures a whole document generated end to end.

Nearly Two-Thirds of Campaign Spend Went on Unparseable Output

Sun, 01 Nov 2026 21:30:09 +0000

Nearly Two-Thirds of Campaign Spend Went on Unparseable Output

On promotional campaign generation, nearly two-thirds of the money spent went on output that could not be parsed [^1]. On community content promotion, a task of the same genre, the share was under 7% [^2]. Neither a model’s price card nor its quality score shows this cost, so it has to be measured task by task.

Same genre, very different waste

For each task, the benchmark reports total spend and how much of it produced output that could not be parsed. Four promotional tasks show how wide the spread is.

When LLM Score Bands Overlap, Buy the Cheapest Model in the Group

Sun, 01 Nov 2026 21:30:00 +0000

Models whose scores tie within their uncertainty bands can differ in price by more than a hundredfold [^1]. Inside such a tie the extra spend buys nothing the benchmark can measure, so the cheapest tied model is the pick, and paying more is justified only where the bands separate.

How much money sits inside a tie

Every score here carries a ± band, and when two bands overlap the scores do not rank the models. On writing hypothesis-style posts, five models land between 8.7 and 8.77 out of 10 [^1]. Tied groups like that one span wide price ranges.

Batch Saves Money, but a Cheaper Model at Equal Quality Saves More

Sun, 01 Nov 2026 21:29:59 +0000

Batch Saves Money, but a Cheaper Model at Equal Quality Saves More

Batch serving costs less than sync for the same model on the same work, with no change in reported score. But on tasks where several models tie on quality, the choice of model moved cost more than the choice of mode: a cheaper model at its sync price undercut a pricier one in batch.

Batch is a discount, not a quality trade-off

That parity comes from paired measurements: the same model, the same work, both modes. In these results, sync buys responsiveness and nothing else. Some results exist in one mode only, and there the accuracy cost of a batch saving is unknown.

Models Tie on Language ID but Split on Content Domains

Sun, 01 Nov 2026 21:29:54 +0000

On identifying what language a text is written in, the leading models are tied at or near 10 out of 10 [^1]. On suggesting which content domain a piece belongs to, the same field spreads out, and only a small group pulls clearly ahead [^2]. On the first kind of task, choosing a model is a question of price and latency; on the second, it is a question of quality.

Budget LLM Latency on the 90th Percentile, Not the Median

Sun, 01 Nov 2026 21:29:53 +0000

Across 863,807 synchronous calls in the Fronset benchmark, the median answer took 5.4 seconds and the 90th percentile took 50 seconds, more than nine times as long, under a 600-second per-call deadline [^1]. If you size a user-facing step on the median, you are planning for the typical call and ignoring the slow ones, and in this data the slow ones are much slower.

Why the median is the wrong number for an interactive step

A median tells you that half of calls finish sooner. It says nothing about how long the other half take, and a step where a person waits on one call is exposed to all of them.

When Paying More for a Model Stops Buying a Better Score

Sun, 01 Nov 2026 21:29:53 +0000

When Paying More for a Model Stops Buying a Better Score

On several kinds of work in the Fronset benchmark, a far pricier model scores no better than a cheap one. Where the gaps are real, the gain can arrive one price tier up, with nothing measurable added beyond it.

The clearest case is judging whether a claim stays consistent. Claude Sonnet 5.5 scores 8.69 (±0.26) at $0.03 per run, and Claude Opus 5.5 scores 8.6 (±0.24) at $0.36 [^1]. The model that costs twelve times as much buys no measurable score.

Confidence Tiers Track Evidence, Not Model Quality

Sun, 01 Nov 2026 21:29:50 +0000

Nine results on the Fronset benchmark, spread across seven models, have dropped a confidence tier, and Gemini 3.8 Flash has been promoted twice. None of these moves reports that a model got better or worse at the work. A tier records how firmly a placement is known, and reading it as a quality signal is the easiest way to misread the results.

What a confidence tier measures

Each result carries a 95% interval on a model’s mean quality score for one kind of work. Three tiers appear in these results, in rising order: medium, high and ranked. When the interval closes far enough, the placement moves up a tier.

First-Shown Answers Score Higher, so Small Gaps Are Noise

Sun, 01 Nov 2026 21:29:50 +0000

First-Shown Answers Score Higher, so Small Gaps Are Noise

On the Fronset benchmark, the order in which a judge sees candidate answers moves the score. Across 105,928 position-stamped judgments, the candidate shown first averages 7.82 and the one shown last averages 7.55 [^1]. A gap between two models that is smaller than that 0.27-point drift is weak evidence that either one is better.

Why answer order leaks into a score

An LLM judge comparing several candidates reads them one after another, and its verdict is not fully independent of that sequence. Unless stronger answers were somehow shown first more often, the higher average for the first slot comes from the judging, not from the models.

Gemini 3.5 Flash Is Fast on One Task and Slow on the Next

Sun, 01 Nov 2026 21:29:50 +0000

Gemini 3.5 Flash takes a median of 8.7 seconds to judge whether a claim stays consistent across a document [^1]. Asked to fix broken line breaks in Markdown output, the same model takes a median of 88 seconds [^2]. A latency figure describes a model on one job, not the model in general.

One model, two very different speeds

The claim-consistency median comes from 276 calls, and both tasks were measured under the same 600-second per-call deadline [^1] [^2]. The gap is not an artefact of different time limits.

Read the Interval, Not the Judgment Count

Sun, 01 Nov 2026 21:29:50 +0000

Read the Interval, Not the Judgment Count

A score backed by more judgments is not necessarily a firmer score. On the Fronset benchmark, 129 judgments leave MiniMax M3’s result on synthesizing several perspectives into a decision at ±0.14 [^1]. Qwen 3.7 Plus gets to ±0.13 on fixing broken line breaks in Markdown with 46 [^2].

Why the count is a poor proxy

An interval narrows as judgments accumulate, but its width also depends on how much those judgments disagree. Where judgments cluster tightly, a score settles quickly. Where they scatter, many more are needed to reach the same width.

Timeouts Are a Separate Failure From Slow Answers

Sun, 01 Nov 2026 21:29:50 +0000

Timeouts Are a Separate Failure From Slow Answers

Every synchronous call in the Fronset benchmark runs under a 600-second deadline, and an attempt that misses it meters no tokens [^1][^2][^3]. For three of the five models with a reported miss rate, more than a quarter of attempts miss, and for one it is more than two-fifths [^1][^3]. In this data, how often a model misses does not track how quickly its answers usually arrive.

Answer Length Sways Judges, and the Direction Depends on the Task

Sun, 01 Nov 2026 21:29:49 +0000

Judges in these benchmark measurements do not treat answer length neutrally, and which way they lean depends on the task. On scoring, detection, selection, repair and translation, the answer judges picked was generally shorter than the one they passed over. On analysis, summarizing, extraction and long-form writing, it was generally longer.

That matters for anyone choosing models by judged quality. The same writing habit can help a model on one task and count against it on the next.

High Quality Scores Often Come With Low Answer Rates

Sun, 01 Nov 2026 21:29:48 +0000

GPT-6 Sol scores 9.15 out of 10 on generating publication title packages, its highest quoted score, yet only 10% of its attempts at that work return an answer [^1]. On the Fronset benchmark, quality and answer rate are separate numbers, and the most impressive scores often sit next to the weakest answer rates.

What the quality score leaves out

The quality score is graded on the calls that came back. Attempts that produced nothing do not pull it down, so it tells you how good an answer is when you get one, not how often you will get one [^1][^2].

A Well-Scored Model Can Still Win Zero Head-to-Heads

Sun, 01 Nov 2026 21:29:46 +0000

A Well-Scored Model Can Still Win Zero Head-to-Heads

On summarizing structured content, GLM-5.3 scores 8.41 (±0.24) after 34 independent judgments, tied with Grok 4.7 [^1]. Across 40 occasions where judges compared it with rivals on the same work, it was never picked as the best answer [^1]. Both can be true at once, because they measure different things.

Two numbers, two questions

A mean score says how good a model’s answers are on a 10-point scale. A win share says how often its answer was judged the best among the rivals in the same contest. The second is winner-take-all: a close second counts the same as last place.

How MiniMax M3 and gemini-3.5-flash Fail Differently on Summaries

Sun, 01 Nov 2026 21:29:46 +0000

MiniMax M3 fails less often than gemini-3.5-flash when turning content into structured summaries: a judged failure rate of 23.0% against 52.3%, on the same task under the same rubric [^1]. The more useful difference is how each model fails, because that decides whether a bad output costs you a retry or slips through as a plausible wrong answer.

Two ways to get a summary wrong

When gemini-3.5-flash fails at summarization, it mostly invents content. When MiniMax M3 fails, it is mostly too brief [^1]. The rubric counts both as failures, but they behave very differently once the output leaves the model.

GPT-6 Luna Ties GPT-6 Sol on Financial Scenario Analysis

Sun, 01 Nov 2026 21:29:45 +0000

GPT-6 Luna Ties GPT-6 Sol on Financial Scenario Analysis

On catalyst and scenario analysis, the closest thing to a trading judgment that the Fronset benchmark measures, GPT-6 Sol has the top score and GPT-6 Luna is statistically tied with it. In sync mode, GPT-6 Sol costs more than twenty times as much per run [^1].

The lesson is less about one model than about what price tells you here. The expensive models do not buy a lead, and the cheapest ones are not safe either.

Before Blaming the Model, Check Whether the Whole Task Moved

Sun, 01 Nov 2026 21:29:43 +0000

Before Blaming the Model, Check Whether the Whole Task Moved

Since 2026-10-16, scores on the benchmark have drifted by fractions of a point, and more moved down than up. Most of that drift lines up by task, not by model. On several tasks, every model measured moved the same way. A model that slips on every task it runs is a different pattern, and that is the one worth re-testing.

Cluster-Labeling Scores Do Not Predict Theme Discovery

Sun, 01 Nov 2026 21:29:40 +0000

Cluster-Labeling Scores Do Not Predict Theme Discovery

The model that names your topic clusters best is not necessarily the one that should find them. On the Fronset benchmark, every model measured on both steps scores lower at discovering themes in text than at labeling clusters it is handed, and some models tied near the top of labeling are among the weakest at discovery [^1][^2].

Two steps that look alike

Labeling gives a model a group of related content and asks for a name. Theme discovery gives it raw text and asks it to decide what the groups are. Both fall under topic organization, so it is tempting to pick one model for the whole pipeline from whichever leaderboard you check first.

A Near-Zero LLM Failure Rate Can Still Hide a Heavy Repair Load

Sun, 01 Nov 2026 21:29:39 +0000

A Near-Zero LLM Failure Rate Can Still Hide a Heavy Repair Load

A model that almost never fails can still hand back output you cannot use. On the Fronset benchmark, the MiniMax record with the lowest failure rate also has the smallest share of output usable as returned [^1][^2].

A failure rate counts one outcome out of three

In these records, every attempt ends in one of three ways. It fails outright, it returns output that needs repair, or it returns output that can be used as it arrives, with no repair call and no retry. A failure rate counts only the first, so output that needs repair never shows up in it.

Price per token vs cost per task across 53 tasks

Wed, 21 Oct 2026 07:23:59 +0000

Every provider publishes a price per million tokens. It is the number most people compare when they choose a model. Our benchmark ranks models differently: by what one run of a task actually costs. That figure covers the tokens the model really used, its reasoning, cache reads, the attempts that failed and were still billed, and what the provider’s invoice adds over its list price (methodology).

So we asked: how well does the price per token predict the cost of a run? We took the snapshot of 16 October 2026 and compared the two for every model on every task, 871 model-and-task pairs across 53 tasks.

What one month of Nova, our internal customer, cost on Fronset

Tue, 20 Oct 2026 15:52:36 +0000

Nova (https://nova.kapualabs.com) is our internal customer: a content and financial-analysis publication run by the team that builds Fronset. Its pipeline has run on Fronset since July. In September it made 198,167 calls across 25 kinds of step, most of them small: extracting claims, identifying languages, summarising, scoring relevance.

Without measurement, a team sends every step like that to one model it trusts. A mid-tier choice would have been Claude Sonnet 5; a cautious one, Claude Opus 5. We priced the month both ways and against what Fronset actually charged:

Clustering's Hard Part Is Finding Topics, Not Naming Them

Mon, 19 Oct 2026 09:57:43 +0000

Clustering’s Hard Part Is Finding Topics, Not Naming Them

In the Fronset benchmark, naming topic clusters is a step where most models land close together; finding the themes in the first place is not. Every model with results on both steps scores lower on thematic topic discovery than on topic cluster labeling [^1][^2]. And on discovery, the leading group includes routes at $0.04 and $0.45 per task run with tied scores, so paying more did not buy better themes [^2].

In LLM Benchmarks, an Empty Cell Is Not a Weak Model

Mon, 19 Oct 2026 09:57:43 +0000

In LLM Benchmarks, an Empty Cell Is Not a Weak Model

The Fronset benchmark describes individual model-and-task cells in considerable detail and the grid around them hardly at all. Most model-and-task pairings hold no published result, and the results do not say why. Before you rely on a number, or read meaning into a missing one, check that the cell you care about actually exists.

The measured grid and the possible one

The current results rest on 46,711 scored judgments across 906 measured model-and-task combinations, covering 27 models on 54 tasks. Those 27 models across 54 tasks imply a possible grid of 1,458 pairings. That 1,458 is arithmetic on the scope line, not a coverage rate: nothing in the results establishes that every unmeasured pairing was feasible or eligible for measurement.

When to Escalate From the Small Model: Watch How It Fails

Mon, 19 Oct 2026 09:57:41 +0000

When to Escalate From the Small Model: Watch How It Fails

In the Fronset benchmark’s model portfolio, every activity gets a default model and, only where the evidence supports one, a fallback. The small model can sit in either slot, and on several activities the fallback is left empty. What should move a call from the default to the fallback depends on how the default fails, not only on how well it scores.

A Three-Way Tie for Best Long-Form Writer. Let Price Decide.

Mon, 19 Oct 2026 09:57:40 +0000

A Three-Way Tie for Best Long-Form Writer. Let Price Decide.

On the Fronset benchmark’s long-form writing work, the top of the table is a tie, not a leader. On reference-preserving analytical writing, the task closest to sustained, source-grounded drafting, three models sit inside overlapping score ranges, and so does a broad band below them [^1]. When the scores cannot separate models, price and pricing mode are what is left to decide on.

Batch Is Cheaper — Except When It Isn't

Mon, 19 Oct 2026 09:57:39 +0000

The batch discount is real, but check it task by task

Where the Fronset benchmark measured the same model on the same work in both batch and synchronous mode, batch cost less per task run in nearly every pair, and the quality score did not change. But the size of the saving swings widely by model and task, and in two pairs batch cost more. The batch price has to be checked for each model and task, not assumed.

Fails Less, Fails Silently: Choosing a Fact Extractor

Mon, 19 Oct 2026 09:57:38 +0000

Fails Less, Fails Silently: Choosing a Fact Extractor

On the Fronset benchmark’s fact-extraction tasks, a lower failure rate does not on its own make a model the safer pick, because the way a model fails decides whether anyone notices. On refining factual claims, gemini-3.5-flash fails less often than MiniMax M3, but it mostly fails by dropping material [^1]. A visibly bad answer can be caught and retried; a missing claim looks like a finished one.

The Best Summarizer Is Not the Best Synthesizer

Mon, 19 Oct 2026 09:57:38 +0000

The Best Summarizer Is Not the Best Synthesizer

On the Fronset benchmark, the model with the highest quoted score on structured content summarization falls back on the neighbouring summarization and synthesis tasks, and models from lower in that ranking lead them instead. A ranking from one of these tasks does not carry over to another, even inside the same category. Pick a model for executive summaries or decision synthesis from a summarization leaderboard and you can end up with the wrong one.

Claude Opus 5.5's Best Score Is a Tie. Its Real Win Is Elsewhere

Mon, 19 Oct 2026 09:57:37 +0000

Claude Opus 5.5’s Best Score Is a Tie. Its Real Win Is Elsewhere

Claude Opus 5.5 scores 9.93 on structured output extraction [^1], its highest quality figure in this material, yet on that task the score marks a tie at the ceiling rather than a lead. Its clearest win is elsewhere: matching profiles to a pool, where it was judged best on 18 of 26 occasions [^2]. Telling those two kinds of result apart shows where Opus 5.5 earns a route on quality and where cost and latency should decide instead.

When Averages Tie, Count the Wins

Mon, 19 Oct 2026 09:57:37 +0000

When Averages Tie, Count the Wins

On the Fronset benchmark’s social-content selection tasks, average quality scores bunch so tightly that they often cannot tell models apart. Head-to-head results, which record how often a model’s answer is judged the best of several answers to the same input, can separate models where the scores cannot. They also expose models that are never judged best at all, which an average score can hide.

Nano, Lite, Haiku: When the Smaller Sibling Costs You Quality

Mon, 19 Oct 2026 09:57:30 +0000

Nano, Lite, Haiku: When the Smaller Sibling Costs You Quality

For plumbing work, the usual reflex is to reach for the smallest member of a model family: the Lite, the Nano, the Haiku. On several utility tasks in the Fronset benchmark, that step down brings a clear drop in score, while on others it costs nothing measured. Which case applies depends on the task, so the family gap has to be checked task by task.

The Leaderboard Moved. The Models Mostly Didn't.

Mon, 19 Oct 2026 09:57:22 +0000

The Leaderboard Moved. The Models Mostly Didn’t.

Since 2026-09-24, the Fronset benchmark records far more movement in how precisely scores are known than in the scores themselves. Confidence intervals closed across many tasks and confidence tiers shifted in both directions, while the means that did move mostly slipped, usually by small amounts [^1][^2][^3][^4]. So if you re-read the leaderboard and see a model promoted or demoted, what most likely changed is how much is known about it, not how well it answers.

8.95 on One Item, 5.69 on a Set: The Relevance-Scoring Cliff

Mon, 19 Oct 2026 09:57:19 +0000

8.95 on One Item, 5.69 on a Set: The Relevance-Scoring Cliff

Models that score the relevance of a single item well do much worse when asked to score a set of content or a batch of short posts. Qwen 3.7 Plus tops the single-item list at 8.95 out of 10 but scores 5.69 on content sets,[^1][^2] and no measured model reaches 7.5 on sets.[^2] If your pipeline scores many items at once, a single-item benchmark result tells you little about how it will perform.

A Pricier Model Only Pays Off When You Raise the Bar

Mon, 19 Oct 2026 09:57:18 +0000

A Pricier Model Only Pays Off When You Raise the Bar

On engagement triage in the Fronset benchmark, Claude Opus 5.5 at $0.01 per task in sync mode is statistically tied with Claude Haiku 5.5 at $0.0003 [^1]. At a bar that accepts any of the leading scores, that is the usual result: the leaders tie, and the defensible pick is the cheapest model in the tied group. A pricier model earns its price only when you raise the bar to require a clear separation, and only on the tasks where the scores actually separate [^2].

3.4 Seconds or 246: Same Model, Different Task

Mon, 19 Oct 2026 09:57:12 +0000

3.4 Seconds or 246: Same Model, Different Task

How long an LLM call takes in these findings depends first on the kind of work, and only after that on which model runs it. Qwen 3.8 Max answers language identification in 3.4 seconds [^1] but takes 246 seconds on topic sequence optimization [^2]. A latency decision therefore has to be made for each model-and-task pairing, not for a model alone.

One model, many speeds

Every figure here is a median: the time by which half of a model’s calls on a task have returned. Measured that way, the same model can move a long way between tasks. Qwen 3.8 Max also takes 187 seconds on thematic topic discovery [^3]. Grok 4.7 runs from 2.3 seconds on language identification [^1] to 138 seconds on reference-preserving analytical writing [^4].

When the Model Doesn't Matter, and When It Really Does

Mon, 19 Oct 2026 09:57:05 +0000

On some tasks in the Fronset Benchmark, the model you choose barely changes the result; on others, it changes it a lot. Language identification is the first kind: several leading models tie near a perfect score [^1]. Content domain suggestion is the second: scores are low and the best answers diverge widely, so it is the task where testing candidates on your own inputs pays off most [^2].

The same models on two tasks

Three models measured on both tasks show the contrast. Scores are out of 10, and the ± figure is the interval reported around a mean score; where two intervals overlap, the gap is a tie.

The Cheap Model Often Ties the Expensive One

Mon, 19 Oct 2026 09:57:04 +0000

On many tasks in the Fronset benchmark, the leading models cannot be separated on quality, so price decides the choice. On engagement opportunity triage, Claude Haiku 5.5 scores 8.52 out of 10 (±0.33) at $0.0003 per task run in sync mode, while Claude Opus 5.5 scores 8.56 (±0.3) at $0.01 [^1]. The exceptions are real, but they are narrower than the price gaps suggest, and they change from task to task.

Most Models Get It Right First Try. Budget for the Rest.

Mon, 19 Oct 2026 09:56:58 +0000

Most Models Get It Right First Try. Budget for the Rest.

For eleven models measured on the Fronset benchmark, the share of attempts that return usable output with no repair or retry ranges from 95.7% to 99.8% [^1]. On that measure the models are close to tied. What a pipeline needs to watch is what happens to the rest: for two models, some outputs needed a second, separately billed repair call that their price cards do not mention.

Median 5.5 s, p90 51 s: Set Timeouts on the Tail

Mon, 19 Oct 2026 09:56:57 +0000

Most latency figures in the Fronset benchmark are medians, and the median understates how long callers actually wait. Across 841,549 synchronous calls, the median is 5.5 seconds, but the 90th percentile is 51 seconds [^1]. A model with the lower median can also have the longer tail, so picking a model or setting a timeout on the median alone can go wrong.

Two numbers, two different promises

The median is the response time that half of all calls beat; the other half take longer. The 90th percentile, or p90, is the time that nine calls in ten beat, so one call in ten takes longer still. A latency budget set on the median is exceeded by roughly half the calls, and one set on the p90 by roughly one in ten.

Check the Error Bars Before You Call It a Lead

Mon, 19 Oct 2026 09:56:56 +0000

On the Fronset benchmark, a higher score is a real lead only when the two models’ confidence intervals do not overlap. In the busy middle of the scale, between roughly 8 and 9, they usually do, and the higher score is then a tie rather than a lead. The numbers worth planning against are the ones that pair a high mean with a narrow interval.

How to read the ± figure

Each score is a mean quality on a 10-point scale, and the ± figure beside it is the 95% confidence interval on that mean. The benchmark treats a model as ranked on a task only when that interval is within ±0.2 points [^1]. Inside that line, the mean is known tightly enough to place; outside it, the score is a rough centre rather than a position.

8.98 Out of 10, on the 10% of Calls That Answered

Mon, 19 Oct 2026 09:56:55 +0000

On the Fronset benchmark, a model’s quality score is measured only on the calls that came back, so a high score can describe a small slice of the work you send it. GPT-6 Sol scores 8.98 out of 10 on publication title package generation but answers on only 10% of attempts [^1]. Read every score next to its answer rate, the share of attempts that return an answer, or you will not know how much of that quality you actually receive.

Judges Don't Simply Reward the Longer Answer

Mon, 19 Oct 2026 09:56:49 +0000

When judges (graders that score one output at a time against a published rubric [^1]) pick the best of several answers to the same inputs, length tilts the choice, but not in one consistent direction. In one set of 38 comparisons in the Fronset benchmark, the answer the judges picked was longer than the ones they passed over in 23 tasks and shorter in 15. A judged preference therefore can’t be read as a clean measure of quality.

LLM Finance Scores Grade the Analysis, Not the Trade

Mon, 19 Oct 2026 09:56:43 +0000

LLM Finance Scores Grade the Analysis, Not the Trade

No result in Fronset’s financial-analysis category measures whether a trading decision was right. None measures returns or the accuracy of a financial conclusion either; the scores rate the answers models wrote on work next to that decision. For a developer building financial-analysis features, that makes the benchmark a way to build a shortlist, not evidence that a model’s calls are correct.

The Right-Sized Model Portfolio

Sat, 26 Sep 2026 14:37:03 +0000

What We Measure, and What We Don't

Sat, 26 Sep 2026 14:37:03 +0000

Best Models for Relevance, Classification & Matching

Sat, 26 Sep 2026 14:37:02 +0000

Model Under the Microscope

Sat, 26 Sep 2026 14:37:00 +0000

What Changes When the Bar Moves?

Sat, 26 Sep 2026 14:37:00 +0000

Best Models for Topic Organization & Clustering

Sat, 26 Sep 2026 14:36:59 +0000

Scores You Can — and Cannot — Trust Yet

Sat, 26 Sep 2026 14:36:56 +0000

Quality and Cost Benchmarking

Sat, 26 Sep 2026 14:36:50 +0000

Batch vs. Sync

Sat, 26 Sep 2026 14:36:48 +0000

Benchmark Results and Evaluation Scores

Sat, 26 Sep 2026 14:36:47 +0000

Inference Latency Benchmarking

Sat, 26 Sep 2026 14:36:45 +0000

Evaluation Bias and Measurement Reliability

Sat, 26 Sep 2026 14:36:42 +0000

How Long Will This Take?

Sat, 26 Sep 2026 14:36:42 +0000

Best Models for Content Summarization & Synthesis

Sat, 26 Sep 2026 14:36:41 +0000

AI Models, Products & Applications

Sat, 26 Sep 2026 14:36:40 +0000

Best Models for Social & Promotional Content

Sat, 26 Sep 2026 14:36:40 +0000

What Changed This Snapshot?

Sat, 26 Sep 2026 14:36:40 +0000

Best Models for Financial Analysis & Trading Decisions

Sat, 26 Sep 2026 14:36:39 +0000

Answer Coverage and Conditional Quality

Sat, 26 Sep 2026 14:36:38 +0000

Best Models for Infrastructure & Utility Work

Sat, 26 Sep 2026 14:36:38 +0000

Best Models for Structured Data & Fact Extraction

Sat, 26 Sep 2026 14:36:38 +0000

Models That Get It Right First Time

Sat, 26 Sep 2026 14:36:38 +0000

Best Models for Long-form Content Generation

Sat, 26 Sep 2026 14:36:35 +0000

Scores You Can — and Cannot — Trust Yet

Fri, 18 Sep 2026 13:46:36 +0000

The Right-Sized Model Portfolio

Fri, 18 Sep 2026 13:46:36 +0000

AI Models, Products & Applications

Fri, 18 Sep 2026 13:46:34 +0000

Best Models for Content Summarization & Synthesis

Fri, 18 Sep 2026 13:46:34 +0000

Best Models for Financial Analysis & Trading Decisions

Fri, 18 Sep 2026 13:46:34 +0000

Benchmark Results and Evaluation Scores

Fri, 18 Sep 2026 13:46:27 +0000

Best Models for Relevance, Classification & Matching

Fri, 18 Sep 2026 13:46:25 +0000

Inference Latency Benchmarking

Fri, 18 Sep 2026 13:46:25 +0000

Quality and Cost Benchmarking

Fri, 18 Sep 2026 13:46:25 +0000

Batch vs. Sync

Fri, 18 Sep 2026 13:46:24 +0000

Best Models for Structured Data & Fact Extraction

Fri, 18 Sep 2026 13:46:24 +0000

Answer Coverage and Conditional Quality

Fri, 18 Sep 2026 13:46:23 +0000

Best Models for Infrastructure & Utility Work

Fri, 18 Sep 2026 13:46:23 +0000

Best Models for Long-form Content Generation

Fri, 18 Sep 2026 13:46:23 +0000

Evaluation Bias and Measurement Reliability

Fri, 18 Sep 2026 13:46:23 +0000

Model Under the Microscope

Fri, 18 Sep 2026 13:46:23 +0000

Best Models for Topic Organization & Clustering

Fri, 18 Sep 2026 13:46:21 +0000

Models That Get It Right First Time

Fri, 18 Sep 2026 13:46:21 +0000

Best Models for Social & Promotional Content

Fri, 18 Sep 2026 13:46:20 +0000

What We Measure, and What We Don't

Fri, 18 Sep 2026 13:46:20 +0000

How Long Will This Take?

Fri, 18 Sep 2026 13:46:18 +0000

What Changed This Snapshot?

Fri, 18 Sep 2026 13:46:18 +0000

What Changes When the Bar Moves?

Fri, 18 Sep 2026 13:46:18 +0000

What Changes When the Bar Moves?

Fri, 21 Aug 2026 15:57:03 +0000

Best Models for Content Summarization & Synthesis

Tue, 18 Aug 2026 19:34:03 +0000

Best Models for Financial Analysis & Trading Decisions

Tue, 18 Aug 2026 19:34:03 +0000

Best Models for Infrastructure & Utility Work

Tue, 18 Aug 2026 19:34:03 +0000

Best Models for Relevance, Classification & Matching

Tue, 18 Aug 2026 19:34:03 +0000

What Changed This Snapshot?

Tue, 18 Aug 2026 19:34:03 +0000

Best Models for Long-form Content Generation