Fronset Benchmark

Best LLMs for Bench Article Hypothesis Generation

Draw the distinct hypotheses a benchmark journal article supports, grounded in the approved claims it was written from.

What the steps mean
Cost mode:

Category: Evidence-Grounded Hypothesis Generation · Typical I/O: 3068→2563 tokens

Runs on model group: All models. Inputs may reach any model in the group, not only the ranked models below.

Run this task on Fronset — request an invitation

Models

Frontier on this task: Moonshot Kimi K3 at 8.77 / 10. Score needed at the Balanced step: 7.89.

point-estimate floor (CI low) · upper CI (less certain) · Bars sorted by blended cost; best-value model first. Greyed rows are MEDIUM+ models whose point estimate does not clear the bar.

Cost breakdown

ModelQualityConfidenceCost / 1k runsOverpayMode
Thinking Machines Inkling Small ★ OpenRouter8.39 / 10 CI [8.04, 8.74]MEDIUM$4.91best valuebatch
MiniMax M3 OpenRouter8.48 / 10 CI [8.27, 8.70]HIGH$5.391.1xbatch
Gemini 3.8 Flash Gemini8.45 / 10 CI [8.31, 8.60]RANKED$7.541.5xbatch
Qwen 3.7 Plus Alibaba Cloud (DashScope)8.41 / 10 CI [8.24, 8.59]RANKED$8.201.7xbatch
Meta Muse Spark 1.3 OpenRouter8.67 / 10 CI [8.51, 8.82]RANKED$15.733.2xbatch
Thinking Machines Inkling OpenRouter8.20 / 10 CI [7.93, 8.47]HIGH$23.164.7xbatch
Qwen 3.8 Max Alibaba Cloud (DashScope)8.05 / 10 CI [7.65, 8.46]MEDIUM$49.2810xbatch
Tencent Hy4 Preview OpenRouter8.58 / 10 CI [8.39, 8.77]RANKED$49.8610xbatch
Moonshot Kimi K3 best Moonshot AI8.77 / 10 CI [8.59, 8.95]RANKED$80.6316xbatch

Overpay shows how much more you pay than the best-value model that meets the quality step (marked ★) — the best-value good-enough option. "16x" means you overpay 16× — 16× that reference for no quality benefit the step asks for. Typical call shape for this task: 3068 input tokens → 2563 output tokens, EMA-tracked from production traffic. Cost is the observed, all-in $ per 1,000 task runs: each model's own measured usage on this task — output verbosity, thinking/reasoning tokens, cache reads and writes, and the spend on its billed failures — priced at current list rates and adjusted by the billing overhead we actually reconcile against provider invoices. Models that answer tersely cost what they actually cost; models that think at length pay for it. Not comparable to providers' advertised $/1M list rates — this is what running the task costs, not a per-token price.

Evaluation rubric

Judge whether each hypothesis is an arguable claim rather than a topic or a summary, whether the article supports it, whether every cited evidence number exists and supports it, whether any figure matches the evidence and carries the denominator it is given with, and whether the hypotheses differ in kind rather than in wording. Penalize figures the evidence does not carry, a figure without its denominator, near-duplicate hypotheses, a count other than the one requested, and any mention of publication, channels or promotion. Schema validity is deterministic.

Output schema

Every answer on this task is checked against this JSON Schema, whichever model wrote it. An answer that doesn't fit counts as a model failure, and the call is retried on another model.

{
  "$defs": {
    "JournalHypothesisV1": {
      "additionalProperties": false,
      "description": "One thesis the article supports, and the claims behind it.",
      "properties": {
        "claim_refs": {
          "description": "1-based indices into the numbered evidence block, naming the claims this hypothesis rests on.",
          "items": {
            "type": "integer"
          },
          "title": "Claim Refs",
          "type": "array"
        },
        "headline": {
          "description": "The hypothesis in one line — a claim, not a topic.",
          "title": "Headline",
          "type": "string"
        },
        "rationale": {
          "description": "Why the cited claims support it, in two or three sentences.",
          "title": "Rationale",
          "type": "string"
        }
      },
      "required": [
        "headline",
        "rationale"
      ],
      "title": "JournalHypothesisV1",
      "type": "object"
    }
  },
  "additionalProperties": false,
  "description": "Every hypothesis drawn from one article.",
  "properties": {
    "hypotheses": {
      "description": "The requested number of distinct, non-overlapping hypotheses.",
      "items": {
        "$ref": "#/$defs/JournalHypothesisV1"
      },
      "title": "Hypotheses",
      "type": "array"
    }
  },
  "required": [
    "hypotheses"
  ],
  "title": "JournalHypothesesV1",
  "type": "object"
}

Prompt templates

The published template for this task.

default · LLMB_BENCH_ARTICLE_HYPOTHESIS_GENERATION_SYSTEM + LLMB_BENCH_ARTICLE_HYPOTHESIS_GENERATION_USER (1618 calls in window)

System prompt

Read one benchmark journal article together with the approved evidence it was written from, and state the small number of distinct hypotheses the article supports. A hypothesis is a claim someone could agree or disagree with, not a topic or a summary. Ground every hypothesis in the numbered evidence and cite the entries it rests on by their numbers; never assert a figure the evidence does not carry, and never restate a figure without the denominator it is given with. Prefer hypotheses that differ in kind rather than in wording, so that each one could be argued on its own. Say nothing about publication, channels or promotion. Your response must conform exactly to this output schema: {schema_json_string}.

User prompt

Produce exactly {hypothesis_count} hypotheses.

Article title: {title}

Article excerpt: {excerpt}

Article body:
{article_body}

Approved evidence (cite these by number):
{evidence_block}