FinanceBench

How FinanceBench works

The suite holds 10,231 questions about publicly traded companies, each paired with an answer and an evidence string pointing at the passage in the filing that supports it. It is open book: the model is given the document, or a retrieval system over it, rather than being asked to recall anything. 150 cases are released open source for reproducible evaluation. Scoring is answer correctness checked against the evidence, reviewed by hand rather than by string match, because a financial answer can be numerically close and still wrong in a way an automated grader would miss.

History and current status

Patronus AI published FinanceBench in November 2023 (arXiv 2311.11944), with authors including Douwe Kiela, who had previously worked on adversarial benchmark design. It arrived during the first wave of enterprise retrieval-augmented generation pilots, when a great many firms were pointing language models at filings and assuming the open-book setup removed the risk. The paper tested 16 state-of-the-art model configurations, covering GPT-4-Turbo, Llama 2 and Claude 2 with both vector stores and long-context prompts, and manually reviewed 2,400 answers.

What the score does not tell you

The headline figures are from late 2023 and the models have moved a long way since, so quoting the 81% as a current capability claim is wrong. The 150-case open subset is small enough that a handful of items swings a percentage noticeably. Because it is open book, the benchmark measures retrieval and reading as much as financial reasoning, and a weak retrieval configuration will drag down a strong model. It also covers US public filings in English, so it says nothing about other reporting regimes. Patronus AI sells evaluation tooling, which is a commercial interest worth naming even though the dataset and the method are public.

This is the general case, not a quirk of one benchmark. For the four failure modes that decide whether any benchmark number means anything, read are AI benchmarks reliable, and for this row in particular, how benchmark contamination inflates scores.

FinanceBench is not one of the 15 benchmarks scored in the benchmark trust ranking, which rates the most-quoted tests on those same four axes and sorts them most trustworthy first.

What FinanceBench scores actually mean

The number to carry from the paper is 81%: GPT-4-Turbo, used with a retrieval system, incorrectly answered or refused 81% of the questions it was asked. That was the best-known model of its moment, on an open-book task, against questions the authors built to be unambiguous. The paper also found that stuffing the relevant evidence into a long context window improved results but was unrealistic for enterprise use because of latency and document size. More recent third-party tracking puts frontier scores far higher: the FinanceBenchmark aggregator read 91.0% for GPT-5.5 when last updated on 14 August 2026. That board compiles figures from published papers and independent evaluations rather than running the test itself, and it does not state the retrieval configuration behind a score, so it is not comparable to the paper and should not be read as a like-for-like improvement. The useful reading is the shape rather than the level: performance on financial QA is dominated by whether the right passage reaches the model, and a refusal counts as a failure when someone is waiting on an answer.

Who reports FinanceBench, and how to read it

Labs do not report FinanceBench in launch posts. It circulates through vendors selling financial AI, through the Patronus AI Enterprise Scenarios Leaderboard, and through third-party trackers, which means most quoted figures are vendor-run rather than independently reproduced. Treat any single headline score with the same suspicion you would apply to a vendor benchmark generally, and check which retrieval setup produced it, because the same model can move many points on retrieval configuration alone.

When to weight FinanceBench in a model choice

Use FinanceBench before pointing a model at filings in any workflow where a wrong number has consequences, and use it to evaluate the whole stack rather than the model alone, since retrieval is where most of the loss happens. Run the 150 open cases against your own configuration instead of trusting a published figure, because the published figures were produced on retrieval setups that are not yours. Watch the refusal rate separately from the error rate: they need different fixes, and a system that refuses safely is a different product from one that answers confidently and wrongly.