Best LLMs for Profiled Document Section Analysis
Analyzes one document section using supplied document conventions, analytical dimensions, evidence boundaries, and decision context. Illustrative uses include analyzing a contract clause, software specification, vendor requirement, market-risk section, licensing term, research-me
Run this task on Fronset — request an invitation
Models
Frontier on this task: Grok 4.7 at 9.82 / 10. Score needed at the Balanced step: 8.83.
point-estimate floor (CI low) · upper CI (less certain) · Bars sorted by blended cost; best-value model first. Greyed rows are MEDIUM+ models whose point estimate does not clear the bar.
| Model | Quality score | CI low | Cost / 1k runs | vs best value |
|---|---|---|---|---|
| GPT-6 Luna | 9.48 / 10 | 9.23 | $1.02 | best value |
| DeepSeek V4.1 Flash | 8.99 / 10 | 8.67 | $3.70 | 3.6x more expensive |
| GLM-5.3 Flash | 9.07 / 10 | 8.88 | $13.59 | 13x more expensive |
| GPT-6 Sol | 8.99 / 10 | 8.72 | $13.85 | 14x more expensive |
| Meta Muse Spark 1.3 | 8.86 / 10 | 8.62 | $29.29 | 29x more expensive |
| Tencent Hy4 Preview | 8.96 / 10 | 8.69 | $41.76 | 41x more expensive |
| Grok 4.7 | 9.82 / 10 | 9.64 | $57.01 | 56x more expensive |
| Moonshot Kimi K3 | 8.91 / 10 | 8.71 | $80.18 | 79x more expensive |
| Thinking Machines Inkling Small | 8.62 / 10 | 8.46 | $8.48 | 8.3x more expensive |
| MiniMax M3 | 8.72 / 10 | 8.66 | $3.27 | 3.2x more expensive |
| Tencent Hy3 | 7.13 / 10 | 6.65 | $1.98 | 1.9x more expensive |
| Gemini 3.5 Flash Lite | 7.06 / 10 | 6.76 | $2.23 | 2.2x more expensive |
| Thinking Machines Inkling | 8.74 / 10 | 8.60 | $33.48 | 33x more expensive |
| Qwen 3.7 Plus | 8.22 / 10 | 7.91 | $11.95 | 12x more expensive |
| Claude Haiku 5.5 | 8.48 / 10 | 8.04 | $4.84 | 4.8x more expensive |
| Claude Opus 5.5 | 8.29 / 10 | 7.87 | $60.57 | 59x more expensive |
| GLM-5.3 | 8.73 / 10 | 8.46 | $100.79 | 99x more expensive |
| Gemini 3.8 Flash | 7.51 / 10 | 7.10 | $7.36 | 7.2x more expensive |
| NVIDIA Nemotron-3 Super 120B | 6.78 / 10 | 6.30 | $2.37 | 2.3x more expensive |
| Xiaomi MiMo V2.6 Pro | 8.82 / 10 | 8.38 | $2.71 | 2.7x more expensive |
| NVIDIA Nemotron-3 Ultra 550B | 7.95 / 10 | 7.58 | $14.20 | 14x more expensive |
| Qwen 3.8 Max | 8.32 / 10 | 7.90 | $74.76 | 73x more expensive |
| Qwen 3.8 Flash | 7.36 / 10 | 7.00 | $8.03 | 7.9x more expensive |
| Claude Sonnet 5.5 | 8.45 / 10 | 8.03 | $29.12 | 29x more expensive |
Cost breakdown
| Model | Quality | Confidence | Cost / 1k runs | Overpay | Mode |
|---|---|---|---|---|---|
| GPT-6 Luna ★ OpenAI | 9.48 / 10 CI [9.23, 9.74] | HIGH | $1.02 | best value | batch |
| DeepSeek V4.1 Flash DeepSeek | 8.99 / 10 CI [8.67, 9.31] | MEDIUM | $3.70 | 3.6x | batch |
| GLM-5.3 Flash Z.AI | 9.07 / 10 CI [8.88, 9.26] | RANKED | $13.59 | 13x | batch |
| GPT-6 Sol OpenAI | 8.99 / 10 CI [8.72, 9.26] | HIGH | $13.85 | 14x | batch |
| Meta Muse Spark 1.3 OpenRouter | 8.86 / 10 CI [8.62, 9.10] | HIGH | $29.29 | 29x | batch |
| Tencent Hy4 Preview OpenRouter | 8.96 / 10 CI [8.69, 9.23] | HIGH | $41.76 | 41x | batch |
| Grok 4.7 best xAI | 9.82 / 10 CI [9.64, 10.00] | RANKED | $57.01 | 56x | batch |
| Moonshot Kimi K3 Moonshot AI | 8.91 / 10 CI [8.71, 9.11] | RANKED | $80.18 | 79x | batch |
Overpay shows how much more you pay than the best-value model that meets the quality step (marked ★) — the best-value good-enough option. "16x" means you overpay 16× — 16× that reference for no quality benefit the step asks for. Typical call shape for this task: 6919 input tokens → 966 output tokens, EMA-tracked from production traffic. Cost is the observed, all-in $ per 1,000 task runs: each model's own measured usage on this task — output verbosity, thinking/reasoning tokens, cache reads and writes, and the spend on its billed failures — priced at current list rates and adjusted by the billing overhead we actually reconcile against provider invoices. Models that answer tersely cost what they actually cost; models that think at length pay for it. Not comparable to providers' advertised $/1M list rates — this is what running the task costs, not a per-token price.
Evaluation rubric
Judge section fidelity, document-profile correctness, quantitative accuracy, materiality under the supplied lens, qualifier preservation, separation of fact and analysis, and recognition of missing context.
Output schema
Every answer on this task is checked against this JSON Schema, whichever model wrote it. An answer that doesn't fit counts as a model failure, and the call is retried on another model.
{
"description": "Analysis result for a single chunk of a large SEC filing.",
"properties": {
"business_insights": {
"title": "Business Insights",
"type": "string"
},
"competitive_insights": {
"title": "Competitive Insights",
"type": "string"
},
"financial_insights": {
"title": "Financial Insights",
"type": "string"
},
"investment_relevance": {
"title": "Investment Relevance",
"type": "string"
},
"key_information": {
"title": "Key Information",
"type": "string"
},
"risk_insights": {
"title": "Risk Insights",
"type": "string"
},
"section_title": {
"title": "Section Title",
"type": "string"
}
},
"required": [
"section_title",
"key_information",
"business_insights",
"financial_insights",
"risk_insights",
"competitive_insights",
"investment_relevance"
],
"title": "SecChunkAnalysis",
"type": "object"
}Prompt templates
The published template for this task.
default ·
LLMB_PROFILED_DOCUMENT_SECTION_ANALYSIS_SYSTEM +
LLMB_PROFILED_DOCUMENT_SECTION_ANALYSIS_USER
(780 calls in window)
System prompt
Analyze only the supplied section. Apply document_profile to interpret terminology, status, and qualifiers and analysis_profile to select material facts, metrics, risks, opportunities, and implications. Preserve whether figures are historical, adjusted, projected, or uncertain. State when a requested topic is outside the section rather than importing it from elsewhere. Treat empty optional values as absent and return only the requested result. Your response must conform exactly to this output schema: {schema_json_string}.
User prompt
Inputs — section_title: {section_title}; section_content: {section_content}; document_profile: {document_profile}; analysis_profile: {analysis_profile}. Use only these inputs to complete the task defined by the system prompt.