Fails Less, Fails Silently: Choosing a Fact Extractor

On the Fronset benchmark’s fact-extraction tasks, a lower failure rate does not on its own make a model the safer pick, because the way a model fails decides whether anyone notices. On refining factual claims, gemini-3.5-flash fails less often than MiniMax M3, but it mostly fails by dropping material [^1]. A visibly bad answer can be caught and retried; a missing claim looks like a finished one.

Two ways to describe a failure

The benchmark describes failures in two ways. The judged failure rate is the share of a model’s outputs on a task that were judged to have failed. The failure type is the kind of failure the model mostly produces; on these tasks the named types include dropping material, terse output, assigning the wrong class, and “other”.

Same task, opposite trade-off

Two pairs of models show the pattern. Claim continuity here means judging whether claims continue across a text.

On both tasks, the model with the lower failure rate mostly fails in a way that is hard to notice. Qwen 3.7 Plus’s terse output is visible and can be retried, while Meta Muse Spark 1.3’s wrong class may pass unnoticed [^2]. Dropped content is likewise harder to notice than a visibly bad answer, and in one set of 145 judged failures reported for refining factual claims, 20% involve dropped material [^1].

The source gives no counts or intervals behind these four failure rates, so read the gaps between them as indicative rather than settled. The failure types still matter on their own, because they decide whether a failure can be caught at all.

Why quiet failures cost more

A pipeline can spot output that is visibly short or plainly broken and send the request again. A dropped claim or a wrong class label leaves output that looks complete, so no retry rule fires and the error moves downstream.

That is the trade-off in the table. A lower failure rate made up of quiet failures can let more bad output through than a higher rate made up of visible ones.

A low failure rate is not a strong answer

Failure rate also falls short in a second way. On atomic fact extraction, failure rates run from 16.8% for kimi-k2.6, the lowest, to 62.5% for MiniMax M3 [^3]. Yet on a separate claim refinement task, kimi-k2.6 was chosen best in none of 76 judged occasions [^4].

“Chosen best” means a model’s answer was the one preferred when compared head to head with other models’ answers. The findings do not name the rivals or say how close kimi-k2.6 came, so this does not show it is unusable. It does show that a low failure rate does not imply a strong answer.

What to do about it

Before routing extraction work on failure rate, look at what the failures are. Where dropped material or wrong classes would pass unnoticed, failure type belongs in the choice alongside the rate.

Match the safeguard to the failure. A model that mostly fails with terse output can be handled with retries. A model that mostly drops material or assigns the wrong class needs a check aimed at that failure, because nothing in its output will trigger a retry.

Pair failure rate with a quality signal such as the head-to-head record. A model can fail rarely and still not produce the preferred answer.

Check the live benchmark at https://fronset.ai/benchmark/ for the current figures on your task before you fix the production route.