LLM Finance Scores Grade the Analysis, Not the Trade
LLM Finance Scores Grade the Analysis, Not the Trade
No result in Fronset’s financial-analysis category measures whether a trading decision was right. None measures returns or the accuracy of a financial conclusion either; the scores rate the answers models wrote on work next to that decision. For a developer building financial-analysis features, that makes the benchmark a way to build a shortlist, not evidence that a model’s calls are correct.
What a score actually grades
A score rates a model’s answers on a 10-point scale, with an interval where one is quoted; where intervals overlap, models are treated as tied rather than ranked. Some results are reported per judged occasion: one input that several models answered, with their answers scored side by side. So “good” here means an answer that rated well against other models’ answers to the same input.
A trade recommendation that scores well is one whose text was rated well. Whether the trade would have worked is outside what was measured.
Which finance tasks are measured
The evidence covers work next to the trading decision, and not all of it is measured the same way.
| Task | What the results report |
|---|---|
| Structured extraction (pulling structured facts out of documents) | Quality scores [^1] |
| Catalyst and scenario analysis | Quality scores [^2] |
| Multi-perspective decision synthesis (combining several viewpoints into one decision) | Quality scores [^3] |
| Trading recommendations | How close the top answers were, one model’s win share, latency [^4] |
| Analysing SEC filings | One model’s win share, latency [^5] |
| Analysing chunks of SEC S-1 filings | One model’s win share [^6] |
Only the first three carry quality scores. The finance-labelled tasks, trading recommendations and SEC filings, carry latency evidence but no paired quality scores. Decision synthesis is the closest scored match to a trading call that weighs competing views.
Trading recommendations: the least settled task
Trading recommendations are the loosest work in the group. On 84 judged occasions, the top two answers scored exactly the same only 2.4% of the time and finished within half a point 53.6% of the time, with a median gap of 0.5 on the 10-point scale [^4]. Translation is much tighter by the same measure: exact ties on 26.9% of 78 occasions and 88.5% within half a point [^7]. These figures describe the best two answers on each occasion, not the whole field.
The practical reading is direct: picking a model for trading recommendations without testing it on your own cases carries more risk than for most other work.
The only direct quality signal on trading and filings is meta/muse-spark-1.1’s head-to-head win share. It was judged best in 29 of the 52 occasions it competed in on trading recommendations, 55.8% [^4], in 23 of 44 on analysing SEC filings, 52.3% [^5], and in 16 of 46 on analysing chunks of SEC S-1 filings, 34.8% [^6]. These are win shares against unnamed rivals answering the same input, not scores on a common scale, so they cannot be ranked against one another. They justify testing the model on these tasks; they do not justify preferring it.
What you still have to check yourself
Correctness comes first, and the benchmark cannot help with it. If your feature recommends trades or draws conclusions from filings, whether those outputs are right has to be measured on your side, on cases where you know the answer.
Failures that look like valid output come second. On claim continuity classification, Meta Muse Spark 1.3 fails mostly by assigning the wrong class, which may go unnoticed [^8]. gemini-3.5-flash’s failures on tightening factual claims usually involve dropping material [^9], a serious risk in a filing summary. Your own checks need to look for these specifically.
Speed comes third, for anything on a live path. On trading recommendations, median response times for synchronous calls range from 12 seconds for NVIDIA Nemotron-3 Ultra 550B to 122 seconds for Moonshot Kimi K3 [^4]. No tail figure is reported for that task, so measure the slow end yourself before sizing a live path.
Use the live figures at https://fronset.ai/benchmark/ to build a shortlist, and set the quality bar your financial task actually needs there. Then test that shortlist on your own cases before any of it touches a trading decision.