AI benchmarks
Every model launch quotes a wall of benchmark names. This directory maps 116 of them across 12 categories, from the coding and agent tests builders watch most to reasoning, math, knowledge, instruction following, long context, multimodal, professional domains, human preference, safety and security: what each one measures, who built it, the year, how it is scored, a representative current top score, and a link to its leaderboard. 27 of them have a full write-up of their own. For models ranked by value, see the value leaderboard; for why these scores are easier to trust some years than others, our guide on whether AI benchmarks are reliable.
What are the main AI benchmarks?
The most-watched AI benchmarks in 2026, by what they test, are:
- SWE-bench Verified and DeepSWE: real-world coding agents
- Terminal-Bench 4.0: agentic terminal and senior engineering tasks, 66 tasks graded pass or fail
- OSWorld 2.0 and AutomationBench: computer-use and business-workflow agents
- ARC-AGI-3: interactive reasoning, and the widest human-model gap left
- Humanity’s Last Exam and GPQA Diamond: expert breadth and PhD-level science
- FrontierMath and AIME: research and competition math
- MMLU-Pro: broad academic knowledge
- LMArena: human-preference ranking (Elo)
- MMMU: college-level multimodal understanding
Older sets like MMLU, GSM8K and HumanEval are now saturated, with top models above 95%, so they are quoted mainly out of habit.
Which benchmarks evaluate AI agent reliability?
The benchmarks built to evaluate agent reliability in 2026 are tau-bench (whether a tool-using agent completes multi-turn tasks consistently, measured pass^k across repeated runs), Terminal-Bench (hard end-to-end command-line tasks), OSWorld 2.0 (computer-use agents on long-horizon desktop work), AutomationBench (cross-application business workflows graded on end state), GAIA and AgentBench (multi-step assistant tasks), and WebArena (long-horizon web tasks). For live rankings from real usage rather than a fixed test set, see Agent Arena; for what the scores mean in practice, our guide to AI agent benchmarks in 2026.
How many AI benchmarks are there?
There is no fixed number: hundreds of AI benchmarks exist, and new ones appear every month as older ones saturate. What is countable is the set frontier labs actually report. This directory maps 116 of them across 12 categories, of which28 are already saturated and 10 were released in 2026 alone. The practical figure to hold onto is smaller still: roughly a dozen benchmarks carry most of the signal in any given model launch, and they are listed above.
What is the hardest AI benchmark in 2026?
By the size of the remaining human-model gap, ARC-AGI-3: humans scored 100% at its March 2026 launch against 0.51% for frontier models, and the ARC-Prize-verified top is still only about 30%. Among knowledge benchmarks the hardest is Humanity’s Last Exam at roughly 53%. For long-horizon agent work, OSWorld 2.0 sits near 20% full task completion. Anything where frontier models score above 90% is measuring its own ceiling instead of the model.
AI benchmarks vs MLPerf: which one do you mean?
The phrase covers two different things. Model benchmarks, the subject of this directory, measure what a model can do: accuracy, task completion, reasoning, reliability. Systems benchmarks such as MLPerf measure how fast and how efficiently hardware runs a fixed workload, reported as throughput, latency and energy rather than capability. A model benchmark tells you which model to pick; a systems benchmark tells you what it costs to serve. For the cost side of that question, see cost per task and the value leaderboard.
What is an AI benchmark?
An AI benchmark is a standardized test, made of a fixed dataset, a task specification, and a scoring metric, used to measure and compare how well AI models perform a specific skill such as reasoning, coding, math, or knowledge. Running many models through the same test produces a single comparable number, which is what a model leaderboard ranks. The catch is that a benchmark only stays meaningful while it is hard: once the frontier clears it, or its answers leak into training data, the score stops telling good models from great ones, and the field has to build a harder one.
How long does an AI benchmark stay useful?
Shorter every year. A benchmark is useful only while the frontier has not cleared it, and that window is collapsing: ARC-AGI-1 survived six years from its 2019 release, the 2021 cohort (HumanEval, GSM8K, MMLU) held for roughly three, the 2023 to 2024 cohort for about two, and ARC-AGI-2 went from 54% to an ARC-Prize-verified 92.5% inside a single year. The trend is not perfectly monotonic, and GPQA Diamond outlasting SWE-bench by a year is a real exception rather than noise we have smoothed away. But the direction is unambiguous, it is the mechanism behind the 28 saturated rows in this directory, and it is why a benchmark name in a launch table means nothing without a date attached to it.
| Benchmark | Released | Saturated | Useful life |
|---|---|---|---|
| HumanEval | 2021 | 2024 | 3 yrs |
| GSM8K | 2021 | 2024 | 3 yrs |
| MMLU | 2021 | 2024 | 3 yrs |
| ARC-AGI-1 | 2019 | 2025 | 6 yrs |
| SWE-bench | 2023 | 2025 | 2 yrs |
| GPQA Diamond | 2023 | 2026 | 3 yrs |
| OSWorld | 2024 | 2026 | 2 yrs |
| SWE-bench Verified | 2024 | 2026 | 2 yrs |
| FrontierMath | 2024 | 2026 | 2 yrs |
| ARC-AGI-2 | 2025 | 2026 | 1 yr |
| MMLU-Pro | 2024 | Not yet saturated | 2 yrs+ |
| Humanity's Last Exam | 2025 | Not yet saturated | 1 yr+ |
| ARC-AGI-3 | 2026 | Not yet saturated | New |
Benchmarks by category
The 116 benchmarks split into 12 categories. Coding and agents is the largest, both because it is the most commercially watched skill and because contamination forced builders to keep replacing the older sets. To search or filter all of them at once, use the directory tool below.
Underlined namesopen a full guide to that benchmark: how it works, its saturation history, and what its score hides. 27 of the 116 have one.
| Item | Value |
|---|---|
| Coding | 25 |
| Agents & tool use | 19 |
| Reasoning | 11 |
| Mathematics | 8 |
| Knowledge | 7 |
| Instructions & languages | 5 |
| Long context | 8 |
| Multimodal | 9 |
| Professional domains | 5 |
| Human preference | 12 |
| Truthfulness | 3 |
| Security | 4 |
Coding & software-engineering agents25
Whether a model can write, edit, and fix real code, increasingly as a multi-step agent working in a real repository. The most-watched category for AI builders, and where contamination bites hardest.
This is the deepest category tracked here, at 25 benchmarks, and the one with the shortest half-life. Five are already saturated and one, Frontier-Bench, was retired outright once it became Terminal-Bench 3.0. The arc runs from HumanEval and MBPP in 2021, both now saturated, through SWE-bench in 2023 to seven separate SWE-bench derivatives: Verified, Multimodal, Pro, Live, rebench, PolyBench and Multi-SWE-bench. That fan-out is the tell. Each variant exists because the previous one leaked into training data or proved too easy, so a headline SWE-bench number without a variant name is close to meaningless. The 2025 and 2026 arrivals moved the goalposts again: SWE-Lancer scores dollars earned rather than tests passed, Terminal-Bench grades whole trials pass or fail, and CursorBench uses agentic graders. Read the metric definition before comparing two scores in this category.
Agents, tool use & computer use19
Whether a model can plan, call tools, browse, and operate a computer or website to finish open-ended tasks, not just answer a question in one shot.
Nineteen benchmarks, and only two saturated, which makes this the category where the frontier is still genuinely moving. The measurement problem here is different from coding: success is open-ended, so the scoring is too. tau-bench reports pass^k, the probability an agent succeeds k times in a row, which punishes the inconsistency that a single-run score hides. Vending-Bench scores net worth from running a business. AndroidWorld and Windows Agent Arena take programmatic rewards from the operating system itself. OSWorld saturated after 2024 and was replaced by OSWorld 2.0 in 2026 at a 500-step budget, the same replacement pattern the coding category shows. Because these tasks are multi-step, variance between runs is large and a single reported number tells you little about reliability. Look for how many trials a score averages over.
Reasoning & abstraction11
Hard multi-step reasoning and fluid, novel problem-solving designed to resist memorization. The benchmarks the frontier is still far from solving.
Eleven benchmarks, six of them already saturated, including GPQA Diamond and ARC-AGI-2. The ARC-AGI line is the clearest illustration of how fast this category burns: ARC-AGI-1 arrived in 2019 and is saturated, ARC-AGI-2 arrived in 2025 and is already saturated, and ARC-AGI-3 landed in 2026 scoring games beaten at or above human level rather than grid matches. Each generation was designed specifically to resist the memorization that killed the last. Humanity's Last Exam remains active and is the widest-open of the set. What makes this category worth watching is precisely that scores stay low: a benchmark models cannot yet solve carries more information than one where the top five models sit within a point of each other. Treat a saturated reasoning benchmark as a historical marker, not a current discriminator.
Mathematics8
From grade-school word problems to research-level proofs. The older sets are saturated; the newest are held back from the public to stay contamination-resistant.
Eight benchmarks and five of them saturated, which is the highest saturation rate of any category here. MATH and GSM8K, both from 2021, went first. FrontierMath followed in 2024 and AIME 2025 saturated within a year of release. That speed is why the newest sets in this category are handled differently: problems are held back from public release specifically to stay contamination-free, which means you are trusting the maintainer rather than auditing the test yourself. The two active exceptions measure something harder to memorize. PutnamBench and miniF2F score formal proofs machine-checked by a proof assistant, so a model cannot pattern-match its way to a correct-looking answer. MathArena reports per-competition accuracy on contests held after a model's training cutoff. For current discrimination, prefer the formal-proof and post-cutoff sets over any accuracy figure on MATH or GSM8K.
Knowledge & general QA7
Broad academic and factual knowledge across domains, usually multiple-choice. The most-quoted and most-saturated family, now largely replaced by harder variants.
The most-quoted family and the most exhausted: of seven benchmarks, two are retired outright (TriviaQA and HellaSwag) and MMLU itself is saturated. MMLU's saturation matters more than most because it is still the number quoted in launch posts, where a 90% score no longer separates anything. Three replacements are active and each fixes a different flaw. MMLU-Redux re-annotated the original questions after auditors found label errors in the source set, so it measures the same knowledge against cleaner ground truth. MMLU-Pro raises difficulty and expands the answer options to blunt lucky guessing. SimpleQA and SuperGPQA go after factual precision rather than breadth, with SimpleQA additionally reporting correct-given-attempted, which separates a model that knows from one that guesses confidently. If a model card quotes bare MMLU in 2026, treat it as marketing.
Instruction following & multilingual5
Whether a model does exactly what it was told, and whether it does it as well outside English. Two skills that decide production reliability but rarely make a launch slide.
The smallest category at five benchmarks, and arguably the most underweighted relative to how much it decides in production. Two are already saturated: MGSM from 2022 and IFEval from 2023, the latter having been the standard instruction-following check. What remains active splits the problem in two. Multi-IF extends instruction following across multiple turns, where compliance decays as a conversation lengthens, which is the failure mode a single-turn score cannot see. Global-MMLU and INCLUDE handle the language axis, testing whether capability measured in English survives translation. These rarely appear in launch announcements, and that absence is itself informative: a model can post frontier reasoning scores and still drift from an explicit output format or degrade sharply outside English, which is exactly what breaks an agent pipeline in deployment.
Long context & retrieval8
Whether a model can actually use a very long input, not just accept it: finding facts, resolving references, and reasoning across hundreds of thousands of tokens.
Eight benchmarks and only one saturated, which makes this the least-exhausted category tracked here. That single saturation is instructive: Needle-in-a-Haystack became trivial once models learned to retrieve a planted fact from a long input, and retrieving one fact turned out to be a poor proxy for using a long context. Every active benchmark here attacks that gap. RULER reports a weighted average across synthetic task types rather than a single retrieval score. BABILong and NoLiMa report accuracy at each length separately, which exposes where a model actually degrades instead of averaging the collapse away. MRCR scores similarity on multi-round coreference, testing whether a model can resolve which earlier reference a later one points at. The practical lesson is that an advertised context window is a capacity claim, not a performance one.
Multimodal & vision9
Reasoning over images, charts, documents, and video alongside text. The frontier for models that see, not just read.
Nine benchmarks, three of them saturated: DocVQA from 2020, ChartQA from 2022 and MMBench from 2023. The saturated three are all narrow-format tasks, reading a document, a chart, or a fixed answer set, and their exhaustion tracks the point where vision models became reliable at extraction. The active set moved to reasoning over what is seen rather than recognizing it. MMMU and its harder MMMU-Pro variant test college-level subject reasoning across images. MathVista pairs visual parsing with mathematical steps. CharXiv splits its score into descriptive and reasoning halves, which usefully separates a model that reads a chart's labels from one that draws a conclusion from its shape. Video-MME is the only entry testing temporal understanding, and it reports results with and without subtitles, a distinction worth checking before quoting a video number.
Domain & professional work5
Medicine, law, finance and other expert fields, where the failure cost is real and a general-purpose score tells you almost nothing about whether the model is safe to deploy.
Five benchmarks covering medicine, law and finance, and the category where a general capability score tells you least. Only MedQA is saturated, and its saturation is a good illustration of why: it is multiple-choice, so a model can rank well without being able to produce a usable clinical answer. The active four moved toward graded output. HealthBench scores against a rubric written by physicians rather than an answer key. MedHELM reports per-task clinical metrics instead of one aggregate. LegalBench aggregates across many separate legal reasoning tasks, and FinanceBench checks answers against source financial documents. The failure cost in these fields is real and asymmetric, which is why a rubric score from a domain benchmark is worth more here than any composite index, and why a high MMLU score implies almost nothing about professional reliability.
Human preference & holistic12
Aggregate and head-to-head measures: human-voted arenas, composite indices, and multi-metric frameworks that rank overall capability rather than one skill.
Twelve benchmarks that rank overall capability rather than one skill, and the category to treat most carefully, because a composite hides its own weighting. LMArena and Copilot Arena derive Elo-style rankings from human votes, which measures preference and presentation as much as correctness. The Artificial Analysis Intelligence Index, the Epoch Capabilities Index and the Artificial Analysis Coding Agent Index are composites built from other benchmarks, so a model can move on these without any underlying capability changing, purely because a constituent was reweighted or a variant was swapped. METR Time Horizon takes a different approach entirely, reporting the task length a model completes half the time, which is a duration rather than a score and is harder to game. Always check which effort or reasoning variant a composite scored before comparing two numbers.
Safety, hallucination & factuality3
Whether a model tells the truth and resists making things up. Measures honesty and hallucination rate, not raw capability.
The thinnest category tracked here, at three benchmarks, and none of them saturated. That thinness is worth noticing on its own: hallucination is the failure mode most commonly cited as blocking deployment, and it has roughly a tenth the measurement infrastructure that coding does. The three take different angles. TruthfulQA reports the percentage of truthful answers and, separately, truthful-and-informative, which matters because a model that refuses everything scores well on truthfulness alone. HaluEval measures whether a model can recognize a hallucination when shown one, which is a different skill from not producing them. The Vectara Hallucination Leaderboard measures rate directly on summarization, where ground truth is the source document. None of these predicts hallucination rate on your own domain, so treat them as directional rather than as an acceptance test.
Security & dangerous capability4
Offensive cyber skill, hazardous knowledge, and whether an agent can be talked into doing harm. Here a high score is bad news, which inverts how every other category on this page reads.
Four benchmarks, all from 2024, all still active, and the only category here where the scoring direction inverts. On WMDP a high score is bad news: it measures hazardous knowledge, so lower is the desirable result, which breaks the habit of reading every benchmark as higher-is-better. Cybench scores offensive capability on capture-the-flag tasks, and a rising score is a capability warning rather than a product improvement. AgentHarm reports refusal rate alongside post-jailbreak behaviour, which separates a model that declines a harmful request from one that declines it until pressed. CyberSecEval 3 reports per-risk pass and failure rather than an aggregate. Because these numbers are read by safety teams rather than buyers, they appear in system cards far more often than in launch benchmarks, and the absence of one is itself a signal.
Search and filter all 116 benchmarks
Filter by category or status, or search by name, alias, what a benchmark measures, or who built it.
Search the directory
Filter all 116 benchmarks by category or status, or search by name, alias, what it measures, or who built it.
Underlined names open a full guide to that benchmark. 27 of 116 have one.
| Benchmark | What it measures | Status | Top score |
|---|---|---|---|
| Aider Polyglot Aider · 2024 | How well a model writes and correctly edits code across many languages, including applying diffs in the right format and self-correcting after test failures. Leaderboard ↗ | Active | Not confirmed |
| BigCodeBench BigCode project · 2024 | Whether models can write code that correctly invokes multiple function calls from diverse real libraries to satisfy complex, practical instructions. Leaderboard ↗ | Active | Not confirmed |
| Commit0 Zhao, Jiang, Lee et al. · 2024 | Whether an agent can write an entire Python library from scratch against an API specification and an interactive test suite. Leaderboard ↗ | Active | Not confirmed |
| CursorBench Anysphere · 2026 | Whether a coding agent can handle ambiguous, multi-file requests inside a real repository, judged on solution correctness, code quality, efficiency and interaction behaviour. Leaderboard ↗ | Active | Not confirmed |
| DeepSWE Datacurve · 2026 | Whether frontier coding agents can complete original, long-horizon engineering tasks written from scratch, with no upstream PR to memorize. Leaderboard ↗ | Active | 73% (v1.1) |
| EvalPlus Liu, Xia, Wang et al. · 2023 | The same function-synthesis task as HumanEval and MBPP, rescored against far larger automatically generated test suites. Leaderboard ↗ | Saturated | Not confirmed |
| Frontier-Bench (now Terminal-Bench 3.0) The Terminal-Bench and Harbor team (Marten, Shaw, Konwinski) with 100+ task contributors and reviewers · 2026 | Whether a coding agent can do senior-level engineering work: building features from realistic instructions, investigating bugs that need runtime inspection, and shipping code that matches an existing repository's conventions. Leaderboard ↗ | Retired | 34.4% |
| FrontierCode Cognition · 2026 | Whether a coding agent produces a mergeable, production-quality pull request, not just one that passes tests, judged on correctness, regression safety, scope, tests and style. | Active | 13.4% (Diamond) |
| HumanEval OpenAI · 2021 | Whether a model can synthesize a single correct Python function from a docstring so that it passes the provided unit tests. | Saturated | ~99% |
| KernelBench Ouyang, Guo, Arora et al. · 2025 | Whether a model can write GPU kernels that are both correct and actually faster than the PyTorch baseline. | Active | Not confirmed |
| LiveCodeBench UC Berkeley, MIT and Cornell · 2024 | Code generation and related skills (self-repair, execution, test-output prediction) on fresh competitive-programming problems, designed to be contamination-free. Leaderboard ↗ | Active | Not confirmed |
| MBPP Google Research · 2021 | Whether a model can generate short, entry-level Python functions from a natural-language prompt that pass the provided tests. | Saturated | ~95%+ |
| Multi-SWE-bench ByteDance · 2025 | Cross-language issue resolution: whether agents can resolve real GitHub issues with a passing patch across many languages beyond Python. Leaderboard ↗ | Active | Not confirmed |
| PaperBench OpenAI · 2025 | Whether an agent can replicate a published AI research paper from scratch: understand the contribution, build the codebase, and run the experiments. | Active | Not confirmed |
| RepoBench Liu, Xu and McAuley · 2023 | Repository-level code auto-completion: retrieving relevant cross-file context, predicting the next line, and the combined retrieval-plus-completion pipeline. | Active | Not confirmed |
| SciCode Tian, Gao, Zhang et al. · 2024 | Whether a model can write code that solves real scientific research problems, not general software tasks. Leaderboard ↗ | Active | Not confirmed |
| SWE-bench Princeton and Stanford · 2023 | Whether a system can resolve a real GitHub issue by generating a patch that passes the repository's hidden tests. Leaderboard ↗ | Saturated | Not confirmed |
| SWE-bench Multimodal Stanford and Princeton · 2024 | Whether coding agents can resolve real GitHub issues in visual, user-facing JavaScript software where the bug or feature involves the UI. Leaderboard ↗ | Active | Not confirmed |
| SWE-bench Pro Scale AI · 2025 | Whether agents can solve long-horizon, enterprise-grade software-engineering tasks under standardized scaffolding, designed to resist contamination. Leaderboard ↗ | Active | 59.1% (public set) |
| SWE-bench Verified OpenAI · 2024 | The same real-GitHub-issue resolution task as SWE-bench, restricted to a human-validated subset where the issue is solvable and the tests are not broken. Leaderboard ↗ | Saturated | ~95% |
| SWE-bench-Live Zhang, He, Zhang et al. · 2025 | The same real-GitHub-issue resolution task as SWE-bench, but on tasks harvested continuously from issues created after a model was trained. Leaderboard ↗ | Active | Not confirmed |
| SWE-Lancer OpenAI · 2025 | Whether frontier models can complete real paid freelance software jobs, both coding and technical-management tasks, well enough to earn the payouts. | Active | Not confirmed |
| SWE-PolyBench Rashid, Bock, Zhuang et al. · 2025 | Whether a coding agent can resolve repository-level tasks outside Python, across Java, JavaScript, TypeScript and Python. | Active | Not confirmed |
| SWE-rebench Badertdinov, Golubev, Nekrashevich et al. · 2025 | Issue resolution on a continuously refreshed, decontaminated pool of Python software-engineering tasks mined automatically from open-source repositories. Leaderboard ↗ | Active | Not confirmed |
| Terminal-Bench The Terminal-Bench team (Marten, Shaw, Merrill) with the Laude Institute and 100+ community task contributors · 2026 | Whether an AI agent can complete hard, realistic command-line tasks (build, configure, train, debug, secure) end to end inside a real terminal. Leaderboard ↗ | Active | 58.2% (4.0) |
| AgentBench Tsinghua University · 2023 | How well an LLM acts as an autonomous agent in multi-turn, open-ended decision-making across diverse interactive environments. Leaderboard ↗ | Active | Not confirmed |
| AndroidWorld Rawles, Clinckemaillie, Chang et al. · 2024 | Whether an agent can operate a real Android phone to finish tasks across everyday apps. Leaderboard ↗ | Active | Not confirmed |
| AutomationBench Zapier · 2026 | Whether an agent can run a realistic business workflow end to end across several apps: discovering the right API endpoints itself, following a policy document, and writing correct data into every system it touches. Leaderboard ↗ | Active | 26.2% |
| BFCL UC Berkeley Gorilla team · 2024 | Whether a model calls functions and APIs correctly: picking the right function, filling parameters with valid types, and refusing to invent functions that were not offered. Leaderboard ↗ | Active | Not confirmed |
| BrowseComp OpenAI · 2025 | Whether a browsing agent can persistently navigate the open web to locate a single hard-to-find, entangled fact. | Active | 51.5% |
| DeepSearchQA Google DeepMind · 2026 | Whether a deep-research agent can plan and execute a long chain of web searches to return an exhaustive, de-duplicated answer list rather than a single fact. Leaderboard ↗ | Active | Not confirmed |
| GAIA Meta AI and Hugging Face · 2023 | Whether an AI assistant can answer real-world questions that require multi-step reasoning, multiple modalities, web browsing and general tool use. Leaderboard ↗ | Active | ~75% |
| LoCoMo Maharana, Lee, Tulyakov et al. · 2024 | Whether an agent remembers and reasons over a conversation that spans months, rather than a single session. | Active | Not confirmed |
| Mind2Web 2 Gou, Huang, Ning et al. · 2025 | Whether an agentic search or deep-research system can browse the live web and return a correct, citation-backed answer to a long-horizon question. Leaderboard ↗ | Active | Not confirmed |
| MLE-bench OpenAI · 2024 | Whether an AI agent can do end-to-end machine-learning engineering (data prep, training, experimentation, submission) at the level of human Kaggle competitors. Leaderboard ↗ | Active | 16.9% (paper baseline) |
| OSWorld XLANG Lab, University of Hong Kong · 2024 | Whether a multimodal agent can operate a real computer (desktop apps, file I/O, multi-app workflows) to complete open-ended tasks in a live virtual machine. Leaderboard ↗ | Saturated | Not confirmed |
| OSWorld 2.0 XLANG Lab, University of Hong Kong, with collaborators at Columbia, UCSB, UCSD, Mila, Ohio State and others · 2026 | Whether a computer-use agent can finish long-horizon professional work on a real desktop, coordinating several applications and leaving the machine in the correct final state. Leaderboard ↗ | Active | 20.6% |
| tau-bench Sierra · 2024 | Whether a tool-using agent can reliably complete customer-service tasks over multi-turn conversations with a simulated user while obeying domain policies. | Active | Not confirmed |
| TheAgentCompany Xu, Song, Li et al. · 2024 | Whether an agent can do real knowledge work inside a simulated software company: browsing, coding, using internal tools, and messaging simulated colleagues. Leaderboard ↗ | Active | Not confirmed |
| Vending-Bench Andon Labs · 2025 | Whether an agent stays coherent over a very long horizon, by running a simulated vending-machine business: stock, orders, pricing and daily fees. | Active | Not confirmed |
| VisualWebArena Carnegie Mellon University · 2024 | Whether a multimodal agent can complete visually grounded web tasks that require interpreting images and page layout, not just text. Leaderboard ↗ | Active | Not confirmed |
| WebArena Carnegie Mellon University · 2023 | Whether an autonomous agent can complete long-horizon, realistic web tasks (navigation, forms, multi-step workflows) in fully functional self-hosted websites. Leaderboard ↗ | Active | Not confirmed |
| WebVoyager He, Yao, Ma et al. · 2024 | Whether a multimodal web agent can complete a user instruction end to end on real, live websites rather than a simulator or a static snapshot. | Saturated | Not confirmed |
| Windows Agent Arena Bonatti, Zhao, Bonacci et al. · 2024 | Whether a multimodal agent can operate a full Windows desktop across the applications people actually use at work. | Active | Not confirmed |
| AGIEval Zhong, Cui, Guo et al. · 2023 | Human-centric reasoning, using questions from real standardised exams taken by people rather than synthetic datasets. | Saturated | Not confirmed |
| ARC-AGI-1 Francois Chollet · 2019 | Whether a system can infer the abstract rule of a novel visual grid puzzle from a few examples and apply it to a new input. Leaderboard ↗ | Saturated | 97.5% (public eval) |
| ARC-AGI-2 ARC Prize Foundation · 2025 | The same fluid-intelligence test as ARC-AGI-1, but with harder, contamination-resistant tasks that stay easy for humans yet very hard for AI. Leaderboard ↗ | Saturated | 92.5% (semi-private) |
| ARC-AGI-3 ARC Prize Foundation · 2026 | Whether an agent dropped into an unfamiliar interactive environment with no instructions, stated goal or rules can work out what to do by acting, build a usable world model, and keep learning across levels. Leaderboard ↗ | Active | 62.7% (semi-private, standard harness) |
| BIG-Bench Hard Suzgun et al. · 2022 | A suite of multi-step reasoning tasks (logic, arithmetic, algorithmic, commonsense) on which pre-2022 models trailed average human raters. | Saturated | Not confirmed |
| DROP Dua, Wang, Dasigi et al. · 2019 | Reading comprehension that requires discrete operations over a passage: resolving references then adding, counting or sorting. | Saturated | Not confirmed |
| EnigmaEval Scale AI · 2025 | Long multimodal puzzle solving: finding hidden connections between unrelated pieces of information and chaining many deductive steps. Leaderboard ↗ | Active | Not confirmed |
| GPQA Diamond Rein et al. · 2023 | Graduate and PhD-level multiple-choice scientific reasoning in biology, physics and chemistry, on questions designed to be unanswerable by quick web search. Leaderboard ↗ | Saturated | ~94% |
| Humanity's Last Exam Center for AI Safety (CAIS) and Scale AI · 2025 | Frontier, closed-ended expert knowledge and reasoning across more than 100 academic disciplines at the limit of human expertise. Leaderboard ↗ | Active | 53.3% |
| MuSR Sprague, Ye, Durrett et al. · 2023 | Multistep commonsense reasoning embedded in long natural-language narratives such as murder mysteries, object placement and team allocation. Leaderboard ↗ | Active | Not confirmed |
| ZebraLogic Lin, Le Bras, Richardson et al. · 2025 | Logical deduction under hard constraints, using logic grid puzzles generated from constraint-satisfaction problems. | Active | Not confirmed |
| AIME 2025 Mathematical Association of America; adopted as an LLM eval by the community · 2025 | Olympiad-track competition mathematics at the level of the American Invitational Mathematics Examination, used as a high-difficulty LLM eval. Leaderboard ↗ | Saturated | 100% |
| FrontierMath Epoch AI · 2024 | Research-level original mathematics requiring hours to days of expert effort, across number theory, analysis, algebraic geometry and more. Leaderboard ↗ | Saturated | 87% (Tiers 1-3) |
| GSM8K OpenAI · 2021 | Multi-step grade-school arithmetic word-problem reasoning. Leaderboard ↗ | Saturated | ~99.6% |
| MATH Hendrycks et al. · 2021 | Step-by-step solving of high-school competition mathematics across algebra, geometry, number theory, probability and precalculus. Leaderboard ↗ | Saturated | ~99% (MATH-500) |
| MathArena ETH Zurich · 2025 | Mathematical reasoning and proof-writing on freshly released competition problems, evaluated before they can enter training data. Leaderboard ↗ | Active | 81.1% (aggregate) |
| miniF2F Zheng, Han and Polu · 2021 | Formal theorem proving on Olympiad-level mathematics, as a shared benchmark across proof assistants. | Saturated | Not confirmed |
| Omni-MATH Gao, Song, Cai et al. · 2024 | Olympiad-level mathematical reasoning across a broad range of subdomains and difficulty levels. Leaderboard ↗ | Active | Not confirmed |
| PutnamBench Tsoukalas, Lee, Jennings et al. · 2024 | Whether a neural theorem prover can produce a formal, machine-checked proof of an undergraduate competition problem. Leaderboard ↗ | Active | Not confirmed |
| HellaSwag Zellers, Holtzman, Bisk et al. · 2019 | Commonsense sentence completion: picking the plausible continuation of an everyday scenario. | Retired | Not confirmed |
| MMLU Hendrycks et al. · 2021 | Broad academic and professional knowledge across 57 subjects via four-choice multiple-choice questions. Leaderboard ↗ | Saturated | ~93% |
| MMLU-Pro TIGER-Lab · 2024 | Harder multi-task reasoning and knowledge designed to de-saturate MMLU and reward deliberate reasoning over recall. Leaderboard ↗ | Active | ~90% |
| MMLU-Redux Gema et al. · 2024 | A re-annotated, error-corrected subset of MMLU used to measure true knowledge accuracy without the original's label noise. Leaderboard ↗ | Active | Not confirmed |
| SimpleQA OpenAI · 2024 | Short-form parametric factuality: whether a model answers single-answer fact-seeking questions correctly and abstains when unsure. Leaderboard ↗ | Active | Not confirmed |
| SuperGPQA M-A-P Team · 2025 | Graduate-level knowledge and reasoning across 285 disciplines, including the applied and service fields that mainstream benchmarks ignore. | Active | Not confirmed |
| TriviaQA Joshi, Choi, Weld et al. · 2017 | Factual recall and reading comprehension over trivia questions with evidence documents. | Retired | Not confirmed |
| Global-MMLU Singh, Romanou, Fourrier et al. · 2024 | Multilingual academic knowledge, separating questions that are culturally neutral from those requiring culture-specific knowledge. | Active | Not confirmed |
| IFEval Zhou, Lu, Mishra et al. · 2023 | Whether a model obeys instructions that can be checked by a program, such as a word count, a required keyword, or a forbidden format. | Saturated | Not confirmed |
| INCLUDE Romanou, Foroutan, Sotnikova et al. · 2024 | Multilingual understanding built from local exam material, so the questions test regional knowledge rather than translated Western content. | Active | Not confirmed |
| MGSM Shi, Suzgun, Freitag et al. · 2022 | Grade-school math word problems solved via chain-of-thought reasoning in ten languages. | Saturated | Not confirmed |
| Multi-IF He, Jin, Wang et al. · 2024 | Whether a model keeps following instructions across multiple turns and in languages other than English. | Active | Not confirmed |
| BABILong Kuratov, Bulatov, Anokhin et al. · 2024 | Reasoning over facts deliberately scattered through an extremely long document, not just retrieving one of them. | Active | Not confirmed |
| HELMET Yen, Gao, Hou et al. · 2024 | Long-context ability across a wide spread of realistic downstream applications rather than one synthetic retrieval task. Leaderboard ↗ | Active | Not confirmed |
| LOFT Lee, Chen, Dai et al. · 2024 | Whether a long-context model can replace a retrieval pipeline outright: doing retrieval, RAG and SQL-style tasks natively from context. | Active | Not confirmed |
| LongBench Tsinghua University · 2023 | Comprehensive long-context understanding across realistic tasks (QA, summarization, few-shot, code, synthetic) in English and Chinese. Leaderboard ↗ | Active | 57.7% (v2, with reasoning) |
| MRCR Google DeepMind (Michelangelo); open-source variant by OpenAI · 2024 | Whether a model can distinguish and retrieve the correct one among multiple near-identical requests buried in a long multi-turn conversation. | Active | Not confirmed |
| Needle-in-a-Haystack Greg Kamradt · 2023 | Whether a model can recall a single planted fact (the needle) inserted at varying depths within a long context (the haystack). | Saturated | Not confirmed |
| NoLiMa Adobe Research and LMU Munich · 2025 | Long-context retrieval and reasoning when the question and the target fact share minimal literal word overlap, forcing latent association rather than keyword matching. Leaderboard ↗ | Active | Not confirmed |
| RULER NVIDIA · 2024 | The real effective context length of a model by testing retrieval, multi-hop tracing, aggregation and QA at increasing sequence lengths. Leaderboard ↗ | Active | Not confirmed |
| ChartQA Masry, Long, Tan et al. · 2022 | Question answering over charts that requires both reading visual features and doing arithmetic on them. | Saturated | Not confirmed |
| CharXiv Wang, Xia, He et al. · 2024 | Chart understanding on real, messy scientific figures rather than clean template-generated charts. Leaderboard ↗ | Active | Not confirmed |
| DocVQA Mathew, Karatzas and Jawahar · 2020 | Question answering over scanned document images, where layout and structure carry the meaning. | Saturated | Not confirmed |
| MathVista Lu et al. · 2023 | Mathematical and quantitative reasoning grounded in visual contexts such as figures, charts, geometry and scientific diagrams. Leaderboard ↗ | Active | ~91% (testmini) |
| MMBench Liu, Duan, Zhang et al. · 2023 | Fine-grained vision-language ability across a structured taxonomy of perception and reasoning skills. Leaderboard ↗ | Saturated | Not confirmed |
| MMMU MMMU team · 2023 | College-level multimodal understanding and reasoning over images, diagrams, charts and text across many disciplines. Leaderboard ↗ | Active | ~86% |
| MMMU-Pro MMMU team · 2024 | A harder, contamination-resistant version of MMMU that forces genuine visual reasoning rather than text-only shortcuts. Leaderboard ↗ | Active | ~84% |
| MMStar Chen, Li, Dong et al. · 2024 | Genuinely vision-dependent multimodal ability, on samples selected so the answer cannot be inferred from the text alone. | Active | Not confirmed |
| Video-MME MME-Benchmarks team · 2024 | Comprehensive video understanding by multimodal LLMs across short, medium and long clips. Leaderboard ↗ | Active | ~89% |
| FinanceBench Patronus AI · 2023 | Open-book financial question answering over real public-company filings, with the supporting evidence required. | Active | Not confirmed |
| HealthBench OpenAI · 2025 | Open-ended clinical conversation quality and safety, graded against rubrics written by practising physicians. | Active | Not confirmed |
| LegalBench Guha, Nyarko, Ho et al. · 2023 | Legal reasoning across the specific skills lawyers actually use, as defined by legal professionals. Leaderboard ↗ | Active | Not confirmed |
| MedHELM Stanford CRFM · 2025 | Clinical ability across the breadth of real medical work, on a clinician-validated taxonomy rather than exam questions. Leaderboard ↗ | Active | Not confirmed |
| MedQA Jin, Pan, Oufattole et al. · 2020 | Medical knowledge, using real questions from professional medical board examinations. | Saturated | Not confirmed |
| AlpacaEval 2 (Length-Controlled) Dubois, Galambosi, Liang et al. · 2024 | Instruction-following quality judged by an LLM, with a regression correction for the judge’s bias toward longer answers. Leaderboard ↗ | Saturated | Not confirmed |
| Arena-Hard-Auto Li, Chiang, Frick et al. · 2024 | Human-preference-aligned quality on hard open-ended prompts, scored automatically instead of by live human voting. | Active | Not confirmed |
| Artificial Analysis Coding Agent Index Artificial Analysis · 2026 | Overall coding-agent capability as a single number, scoring the full stack (a specific model plus its harness and settings) rather than a model in isolation. Leaderboard ↗ | Active | Not confirmed |
| Artificial Analysis Intelligence Index Artificial Analysis · 2024 | A composite index of overall model intelligence aggregating performance across reasoning, coding, knowledge, science and agentic tasks. Leaderboard ↗ | Active | ~60 (index) |
| Copilot Arena Chi, Chen, Angelopoulos et al. · 2025 | Which coding model developers actually prefer, collected from paired completions inside a real editor rather than a chat window. | Active | Not confirmed |
| Epoch Capabilities Index Epoch AI · 2025 | Overall model capability on one continuous scale, stitched together from many benchmarks of differing difficulty. Leaderboard ↗ | Active | Not confirmed |
| GDPval OpenAI, with an agentic re-run by Artificial Analysis as GDPval-AA · 2025 | Whether a model can produce the actual deliverables of skilled professional work (documents, slides, spreadsheets, diagrams) well enough to stand against an industry expert's version. Leaderboard ↗ | Active | 1861 Elo (GDPval-AA v2) |
| HELM Stanford CRFM · 2022 | Multi-metric holistic evaluation across many scenarios, reporting accuracy alongside calibration, robustness, fairness, bias, toxicity and efficiency. Leaderboard ↗ | Active | Not confirmed |
| LiveBench White, Dooley, Roberts et al. · 2024 | Broad capability across six categories at once (math, coding, reasoning, data analysis, instruction following, language), on questions refreshed monthly. Leaderboard ↗ | Active | Not confirmed |
| LMArena Arena · 2023 | Crowdsourced human preference between two anonymized model responses, aggregated into a relative ranking, not an objective capability. Leaderboard ↗ | Active | ~1510 Elo |
| METR Time Horizon METR · 2025 | Model capability expressed in human time: the length of task, measured by how long humans take, that a model completes with 50% success. | Active | Not confirmed |
| MT-Bench LMSYS · 2023 | Instruction-following and conversational quality on multi-turn prompts, scored automatically by a strong LLM judge. Leaderboard ↗ | Saturated | Not confirmed |
| HaluEval Li et al. · 2023 | A model's ability to recognize hallucinated content across question answering, knowledge-grounded dialogue and summarization. | Active | Not confirmed |
| TruthfulQA Lin, Hilton, Evans · 2021 | Whether a model avoids repeating common human misconceptions when answering questions, rather than imitating popular falsehoods. | Active | Not confirmed |
| Vectara Hallucination Leaderboard Vectara · 2023 | How often a model introduces unsupported content when summarizing a provided source document, i.e. faithfulness in closed-book summarization. Leaderboard ↗ | Active | 1.8% (lower is better) |
| AgentHarm Andriushchenko, Souly, Dziemian et al. · 2024 | Whether a tool-using agent refuses explicitly malicious multi-step tasks, and whether it stays capable enough to complete them once jailbroken. | Active | Not confirmed |
| Cybench Zhang, Perry, Dulepet et al. · 2024 | Whether an agent can autonomously solve professional capture-the-flag security tasks: finding a vulnerability and executing an exploit. Leaderboard ↗ | Active | Not confirmed |
| CyberSecEval 3 Meta · 2024 | Cybersecurity risk across eight areas, split between risk to third parties and risk to the developers and users of an application. | Active | Not confirmed |
| WMDP Li, Pan, Gopal et al. · 2024 | Proxy knowledge of hazardous biosecurity, cybersecurity and chemical-security material. Leaderboard ↗ | Active | Not confirmed |
Top scores are representative snapshots as of September 4, 2026, not live readings: leaderboards move constantly, and many figures come from each benchmark's own board (the Leaderboard link in each row). Where no clean current top score could be confirmed from a primary source, the cell reads "Not confirmed." Confirm at the source before quoting a number.
Read the full guide to any benchmark
A row in the tables above tells you what a benchmark is. These 27 guides tell you whether to believe it: how the scoring actually works, the version history and saturation trajectory, who runs it and what stake they have, and the specific thing the headline number hides. Start with whichever benchmark a vendor just quoted at you.
- Aider PolyglotCoding
- HumanEvalCoding
- LiveCodeBenchCoding
- SWE-benchCoding
- SWE-bench ProCoding
- SWE-bench VerifiedCoding
- Terminal-BenchCoding
- BFCLAgents & tool use
- GAIAAgents & tool use
- OSWorld 2.0Agents & tool use
- tau-benchAgents & tool use
- ARC-AGI-2Reasoning
- ARC-AGI-3Reasoning
- GPQA DiamondReasoning
- Humanity's Last ExamReasoning
- FrontierMathMathematics
- MMLUKnowledge
- MMLU-ProKnowledge
- RULERLong context
- MMMUMultimodal
- HealthBenchProfessional domains
- LegalBenchProfessional domains
- Epoch Capabilities IndexHuman preference
- LiveBenchHuman preference
- LMArenaHuman preference
- METR Time HorizonHuman preference
- TruthfulQATruthfulness
Get each breakdown before it makes the rounds
You get one email when a new source-backed analysis goes live: what AI agents actually cost, which models are worth running, and what the benchmarks really mean. No hype.
No spam. Unsubscribe anytime.
How to read a benchmark score
A headline score is only as good as the benchmark behind it. Four failure modes decide whether a number means anything, and the strongest benchmarks are the ones that resist all four. The scorecard below rates the most-quoted benchmarks on each concern, where higher always means a less trustworthy score. For the same ratings summed and ranked, see which AI benchmarks are worth trusting. For how the scores get inflated in the first place, read are AI benchmarks reliable and our breakdown of benchmark contamination.
| Benchmark | Contamination | Saturation | Gameability | Real-world gap |
|---|---|---|---|---|
| Terminal-Bench 4 (terminal tasks) | Low | Low | Low | Low |
| SWE-bench Pro (real GitHub fixes) | Low | Low | Low | Low |
| tau-bench (tool use under policy) | Medium | Low | Low | Low |
| OSWorld 2.0 (computer use) | Low | Low | Medium | Low |
| LiveBench (contamination-limited composite) | Low | Low | Low | Medium |
| METR Time Horizon (task length, not accuracy) | Low | Low | Medium | Medium |
| ARC-AGI-3 (interactive reasoning) | Low | Low | Medium | High |
| FrontierMath (research math) | Medium | High | Low | Medium |
| GPQA Diamond (PhD-level science) | Medium | High | Medium | Medium |
| MMLU-Pro (knowledge MCQ, harder) | Medium | Medium | Medium | High |
| ARC-AGI-2 (abstract reasoning) | Low | High | Medium | High |
| LMArena (human preference) | Medium | Low | High | High |
| SWE-bench Verified (real GitHub fixes) | High | High | High | Medium |
| MMLU (knowledge MCQ) | High | High | High | High |
| HumanEval (function-level coding) | High | High | High | High |
Which benchmark should you actually watch?
Pick by the decision you are making, not by whichever number a vendor leads with.
- Choosing a coding model: read a completion benchmark and a quality benchmark together. DeepSWE measures whether an agent finishes the task; FrontierCode measures whether you would merge its code, and the same model can ace one while failing the other. Add Terminal-Bench for agentic, run-it-in-a-real-terminal work and senior-level feature and debugging tasks, where the best model sits at 51.8% on the current 4.0 set. Check the version before you quote it: 2.1, 3.0 and 4.0 are different task sets and differ by more than 30 points. Treat CursorBench with more caution: it is Anysphere's private suite, so nobody outside the company can reproduce a number from it.
- Judging raw reasoning: the ARC ladder moved fast in 2026. ARC-AGI-1 is finished at 97.5% and ARC-AGI-2 went from 54% in December 2025 to an ARC-Prize-verified 92.5% by July, so neither separates the frontier any more. ARC-AGI-3 is now the widest gap on this page: humans scored 100% at its March 2026 launch against 0.51% for frontier models, and the verified top is still only 30.2%. Humanity’s Last Exam remains the unsaturated expert-breadth test at about 53%. Movement on those two is real progress, not noise.
- Comparing general capability: a human-preference ranking like LMArena is the closest to "which feels better to use," but it rewards style as much as substance, so pair it with a composite index and a hard reasoning score.
- Ranking agents, not chatbots: the same team now runs Agent Arena, which scores agents from real usage using causal tracing instead of votes, a method built to resist the style-gaming that dogs preference boards. The Artificial Analysis Coding Agent Index attacks the same problem from the other side, scoring a model and its harness together as one stack, which is what you actually deploy. See cost per task for what those scores cost to achieve.
- Ignore the saturated ones: MMLU, GSM8K, MATH and HumanEval are quoted out of habit. When every frontier model scores above 95%, the benchmark is measuring the ceiling, not the model.
For how these benchmarks rank the current crop of agents, see how the 2026 coding-agent benchmarks actually rank, the coding agents that wrap these models, and the value leaderboard for points-per-dollar.
Why this is a snapshot, not a live feed
Benchmark scores are the most volatile, most gamed layer of model marketing. Leaderboards re-rank weekly, labs report only the tests they win, and the same benchmark can be run under different scaffolds that move the number by ten points or more. This directory is therefore a representative map as of September 4, 2026, built to show what each benchmark means and ground it to its own source, not to quote a score you can hold anyone to. The durable value is the left of the table (what it measures, who built it, whether it is still meaningful); the top-score column is a pointer to the live leaderboard, where you should always confirm before quoting a number.
Frequently asked questions
- What is an AI benchmark?
- An AI benchmark is a standardized test made of a fixed dataset, a task specification, and a scoring metric, used to measure and compare how well AI models perform a specific skill such as reasoning, coding, math, or knowledge. Running many models through the same test produces a single comparable score, which is what model leaderboards rank.
- What are the main AI benchmarks in 2026?
- For coding, SWE-bench Verified, DeepSWE, and Terminal-Bench. For reasoning, GPQA Diamond, ARC-AGI-2, and Humanity’s Last Exam. For math, FrontierMath and AIME. For broad knowledge, MMLU-Pro. For multimodal, MMMU. And for overall human preference, the LMArena Elo ranking. Many older benchmarks like MMLU, GSM8K, and HumanEval are now saturated and quoted mainly out of habit.
- Are AI benchmarks reliable?
- Partly. A benchmark is reliable only as far as its score reflects real capability, and four things erode that: contamination (test data leaking into training), saturation (top models bunched near the ceiling), gameability (a score inflated without real skill), and vendor cherry-picking (a lab reporting only the benchmarks it wins). The most trustworthy benchmarks are contamination-resistant, unsaturated, and run by an independent party. For the full breakdown, see our guide on whether AI benchmarks are reliable.
- What does it mean when a benchmark is saturated?
- A benchmark is saturated when the strongest models all score near its ceiling, so the differences between them are within noise and the benchmark no longer separates a better model from a worse one. MMLU, GSM8K, MATH, and HumanEval are all saturated in 2026, with top models above 95%, which is why the field keeps building harder replacements like MMLU-Pro and ARC-AGI-2.
- What is the difference between SWE-bench and SWE-bench Verified?
- SWE-bench is the original 2,294-task set of real GitHub issues. SWE-bench Verified is a 500-task subset that OpenAI and the SWE-bench authors hand-checked in 2024 to remove broken tests and unsolvable issues, so it is the cleaner, more-quoted version. By 2026 even Verified is treated as saturated and contamination-prone, and OpenAI now recommends the harder SWE-bench Pro instead.
- Which benchmarks evaluate AI agent reliability in 2026?
- The main agent-reliability benchmarks in 2026 are tau-bench, which measures whether a tool-using agent completes multi-turn tasks consistently across repeated runs; Terminal-Bench for hard end-to-end command-line tasks; OSWorld 2.0 for computer-use agents on long-horizon desktop work; AutomationBench for cross-application business workflows graded on end state; GAIA and AgentBench for multi-step assistant tasks; and WebArena for long-horizon web tasks. Agent Arena complements these by ranking agents from real usage instead of a fixed test set.
- What are the new AI benchmarks in 2026?
- The benchmarks that appeared or replaced predecessors in 2026 are Terminal-Bench 3.0 and 4.0, which replaced the saturated 2.x task set with harder senior-level engineering work (Frontier-Bench was this set under its working name); CursorBench, Anysphere private suite mined from real Cursor sessions; OSWorld 2.0, 108 long-horizon computer-use tasks that replace the now-solved OSWorld; AutomationBench, 600 or more cross-app business workflows from Zapier; DeepSearchQA, 900 exhaustive-answer research prompts from Google DeepMind; ARC-AGI-3, the first fully interactive ARC where frontier models scored 0.51% at launch; and the Artificial Analysis Coding Agent Index, which scores whole agent stacks rather than bare models. They exist because the 2024 generation of benchmarks saturated.
- Which AI benchmark matters most for coding?
- There is no single one, because they measure different things. SWE-bench Verified and DeepSWE measure whether an agent can resolve a real issue; Terminal-Bench measures whether it can operate a real terminal end to end; FrontierCode measures whether the code is clean enough to merge; LiveCodeBench measures contamination-free competitive programming. For shipping production code, read a completion benchmark and a quality benchmark together rather than trusting one number.
- What is benchmark contamination?
- Contamination is when a benchmark’s questions or answers leak into a model’s training data, so the model can recall the answer instead of reasoning it out. It inflates scores without reflecting real capability and is the main reason public, static benchmarks decay over time. The defenses are private or held-out test sets, time-stamped problems released after a model’s cutoff, and freshly generated tasks.
- How many AI benchmarks are there?
- There is no fixed number, because new benchmarks are created continuously as older ones saturate. This directory tracks 116 of the benchmarks frontier labs actually report, across 12 categories, 28 of which are already saturated. In practice roughly a dozen benchmarks carry most of the signal in any given model launch.
- What is the hardest AI benchmark in 2026?
- By the size of the remaining human-model gap it is ARC-AGI-3, where humans scored 100% at the March 2026 launch against 0.51% for frontier models, and the ARC-Prize-verified top is still around 30%. Humanity’s Last Exam is the hardest knowledge benchmark at roughly 53%, and OSWorld 2.0 is the hardest agent benchmark at about 20% full task completion.
- Which AI benchmarks are contamination resistant?
- The ones that do not rely on a fixed public test set. Four designs work: private or held-out splits, as in ARC-AGI-2 and GAIA; time-stamped problems released after a model training cutoff, as in LiveCodeBench and SWE-bench-Live; continuously rotated questions, as in LiveBench; and programmatically generated tasks, as in ZebraLogic and RULER. Formal proof benchmarks such as PutnamBench are also hard to fake, because a proof checker either accepts the proof or it does not.
- What is the difference between AI benchmarks and MLPerf?
- They measure different layers. Model benchmarks such as MMLU, SWE-bench and Terminal-Bench measure what a model can do, scored as accuracy or task completion. Systems benchmarks such as MLPerf measure how fast and how efficiently hardware runs a fixed workload, scored as throughput, latency and energy. A model benchmark helps you pick a model; a systems benchmark tells you what serving it costs.
- How long does an AI benchmark stay useful?
- Between one and three years, and the window is shrinking. The 2021 cohort, including HumanEval, GSM8K and MMLU, stayed useful for roughly three years. Benchmarks released in 2024 have lasted about two. ARC-AGI-2, released in 2025, went from 54% to an ARC-Prize-verified 92.5% within a year. A benchmark score is therefore only meaningful with a date attached.
- What is GPQA Diamond, and why is it hard?
- GPQA Diamond is a 198-question set of graduate and PhD-level biology, physics, and chemistry questions written by domain experts to be Google-proof, meaning a non-expert with web access still cannot answer them quickly. It tests reasoning over retrieval. By 2026 top models exceed the roughly 70% human-expert baseline and sit in the low-to-mid 90s, so it is now largely saturated.
Sources
Each benchmark is grounded to its primary source: the original paper, the project repository, or the official leaderboard. Rows are re-verified in batches rather than all on one day, so each entry below carries the date it was last checked against its source (June to September 2026). Top scores are representative snapshots from those sources:
Full source list for all 116 benchmarks
- Aider Polyglot (2024). Aider (Paul Gauthier). aider.chat/2024/12/21/polyglot.html. Verified 2026-06-29.
- BigCodeBench (2024). BigCode project (Zhuo et al.). arxiv.org/abs/2406.15877. Verified 2026-06-29.
- Commit0 (2024). Zhao, Jiang, Lee et al.. arxiv.org/abs/2412.01769. Verified 2026-07-27.
- CursorBench (2026). Anysphere (Cursor). cursor.com/blog/cursorbench. Verified 2026-07-26.
- DeepSWE (2026). Datacurve. github.com/datacurve-ai/deep-swe. Verified 2026-09-04.
- EvalPlus (2023). Liu, Xia, Wang et al.. arxiv.org/abs/2305.01210. Verified 2026-07-27.
- Frontier-Bench (now Terminal-Bench 3.0) (2026). The Terminal-Bench and Harbor team (Marten, Shaw, Konwinski) with 100+ task contributors and reviewers. www.tbench.ai/news/terminal-bench-3-0. Verified 2026-08-29.
- FrontierCode (2026). Cognition (with 20+ open-source maintainers). cognition.com/blog/frontier-code. Verified 2026-06-29.
- HumanEval (2021). OpenAI (Chen et al.). arxiv.org/abs/2107.03374. Verified 2026-06-29.
- KernelBench (2025). Ouyang, Guo, Arora et al.. arxiv.org/abs/2502.10517. Verified 2026-07-27.
- LiveCodeBench (2024). UC Berkeley, MIT and Cornell (Jain, Han et al.). arxiv.org/abs/2403.07974. Verified 2026-06-29.
- MBPP (2021). Google Research (Austin, Odena et al.). arxiv.org/abs/2108.07732. Verified 2026-06-29.
- Multi-SWE-bench (2025). ByteDance (ByteDance Seed). arxiv.org/abs/2504.02605. Verified 2026-06-29.
- PaperBench (2025). OpenAI (Starace, Jaffe, Sherburn et al.). arxiv.org/abs/2504.01848. Verified 2026-07-27.
- RepoBench (2023). Liu, Xu and McAuley (UC San Diego). arxiv.org/abs/2306.03091. Verified 2026-06-29.
- SciCode (2024). Tian, Gao, Zhang et al.. arxiv.org/abs/2407.13168. Verified 2026-07-27.
- SWE-bench (2023). Princeton and Stanford (Jimenez, Yang, Yao et al.). arxiv.org/abs/2310.06770. Verified 2026-06-29.
- SWE-bench Multimodal (2024). Stanford and Princeton (Yang, Jimenez et al.). arxiv.org/abs/2410.03859. Verified 2026-06-29.
- SWE-bench Pro (2025). Scale AI (Scale Labs). arxiv.org/abs/2509.16941. Verified 2026-07-10.
- SWE-bench Verified (2024). OpenAI (with the SWE-bench authors). openai.com/index/introducing-swe-bench-verified/. Verified 2026-06-29.
- SWE-bench-Live (2025). Zhang, He, Zhang et al.. arxiv.org/abs/2505.23419. Verified 2026-07-27.
- SWE-Lancer (2025). OpenAI (Miserendino, Patwardhan et al.). arxiv.org/abs/2502.12115. Verified 2026-06-29.
- SWE-PolyBench (2025). Rashid, Bock, Zhuang et al.. arxiv.org/abs/2504.08703. Verified 2026-07-27.
- SWE-rebench (2025). Badertdinov, Golubev, Nekrashevich et al.. arxiv.org/abs/2505.20411. Verified 2026-07-27.
- Terminal-Bench (2026). The Terminal-Bench team (Marten, Shaw, Merrill) with the Laude Institute and 100+ community task contributors. arxiv.org/abs/2601.11868. Verified 2026-09-04.
- AgentBench (2023). Tsinghua University (THUDM; Liu et al.). arxiv.org/abs/2308.03688. Verified 2026-06-29.
- AndroidWorld (2024). Rawles, Clinckemaillie, Chang et al.. arxiv.org/abs/2405.14573. Verified 2026-07-27.
- AutomationBench (2026). Zapier (Shepard and Salimans). arxiv.org/abs/2604.18934. Verified 2026-07-26.
- BFCL (2024). UC Berkeley Gorilla team. gorilla.cs.berkeley.edu/leaderboard.html. Verified 2026-07-27.
- BrowseComp (2025). OpenAI (Wei, Sun et al.). arxiv.org/abs/2504.12516. Verified 2026-06-29.
- DeepSearchQA (2026). Google DeepMind (Gupta, Chatterjee, Haas et al.). arxiv.org/abs/2601.20975. Verified 2026-07-26.
- GAIA (2023). Meta AI and Hugging Face (Mialon, Fourrier et al.). arxiv.org/abs/2311.12983. Verified 2026-06-29.
- LoCoMo (2024). Maharana, Lee, Tulyakov et al.. arxiv.org/abs/2402.17753. Verified 2026-07-27.
- Mind2Web 2 (2025). Gou, Huang, Ning et al.. arxiv.org/abs/2506.21506. Verified 2026-07-27.
- MLE-bench (2024). OpenAI (Chan et al.). arxiv.org/abs/2410.07095. Verified 2026-06-29.
- OSWorld (2024). XLANG Lab, University of Hong Kong (Xie et al.). arxiv.org/abs/2404.07972. Verified 2026-07-26.
- OSWorld 2.0 (2026). XLANG Lab, University of Hong Kong, with collaborators at Columbia, UCSB, UCSD, Mila, Ohio State and others. arxiv.org/abs/2606.29537. Verified 2026-07-26.
- tau-bench (2024). Sierra (Yao, Shinn, Narasimhan et al.). arxiv.org/abs/2406.12045. Verified 2026-06-29.
- TheAgentCompany (2024). Xu, Song, Li et al.. arxiv.org/abs/2412.14161. Verified 2026-07-27.
- Vending-Bench (2025). Andon Labs (Backlund and Petersson). arxiv.org/abs/2502.15840. Verified 2026-07-27.
- VisualWebArena (2024). Carnegie Mellon University (Koh et al.). arxiv.org/abs/2401.13649. Verified 2026-06-29.
- WebArena (2023). Carnegie Mellon University (Zhou, Xu et al.). arxiv.org/abs/2307.13854. Verified 2026-06-29.
- WebVoyager (2024). He, Yao, Ma et al.. arxiv.org/abs/2401.13919. Verified 2026-07-27.
- Windows Agent Arena (2024). Bonatti, Zhao, Bonacci et al.. arxiv.org/abs/2409.08264. Verified 2026-07-27.
- AGIEval (2023). Zhong, Cui, Guo et al.. arxiv.org/abs/2304.06364. Verified 2026-07-27.
- ARC-AGI-1 (2019). Francois Chollet (ARC Prize Foundation). arcprize.org/arc-agi/1. Verified 2026-07-26.
- ARC-AGI-2 (2025). ARC Prize Foundation (Chollet et al.). arcprize.org/arc-agi/2. Verified 2026-07-26.
- ARC-AGI-3 (2026). ARC Prize Foundation. arxiv.org/abs/2603.24621. Verified 2026-09-04.
- BIG-Bench Hard (2022). Suzgun et al. (Google Research and Stanford). github.com/suzgunmirac/BIG-Bench-Hard. Verified 2026-06-29.
- DROP (2019). Dua, Wang, Dasigi et al.. arxiv.org/abs/1903.00161. Verified 2026-07-27.
- EnigmaEval (2025). Scale AI (Wang, Lee, Menghini et al.). arxiv.org/abs/2502.08859. Verified 2026-07-27.
- GPQA Diamond (2023). Rein et al. (NYU, Cohere, Anthropic). arxiv.org/abs/2311.12022. Verified 2026-07-01.
- Humanity's Last Exam (2025). Center for AI Safety (CAIS) and Scale AI. arxiv.org/abs/2501.14249. Verified 2026-07-26.
- MuSR (2023). Sprague, Ye, Durrett et al. (UT Austin). arxiv.org/abs/2310.16049. Verified 2026-06-29.
- ZebraLogic (2025). Lin, Le Bras, Richardson et al.. arxiv.org/abs/2502.01100. Verified 2026-07-27.
- AIME 2025 (2025). Mathematical Association of America; adopted as an LLM eval by the community. matharena.ai/. Verified 2026-06-29.
- FrontierMath (2024). Epoch AI. epoch.ai/frontiermath. Verified 2026-07-26.
- GSM8K (2021). OpenAI (Cobbe et al.). arxiv.org/abs/2110.14168. Verified 2026-06-29.
- MATH (2021). Hendrycks et al. (UC Berkeley). arxiv.org/abs/2103.03874. Verified 2026-06-29.
- MathArena (2025). ETH Zurich (SRI Lab). arxiv.org/abs/2505.23281. Verified 2026-06-29.
- miniF2F (2021). Zheng, Han and Polu. arxiv.org/abs/2109.00110. Verified 2026-07-27.
- Omni-MATH (2024). Gao, Song, Cai et al. (Peking University and collaborators). arxiv.org/abs/2410.07985. Verified 2026-06-29.
- PutnamBench (2024). Tsoukalas, Lee, Jennings et al.. arxiv.org/abs/2407.11214. Verified 2026-07-27.
- HellaSwag (2019). Zellers, Holtzman, Bisk et al.. arxiv.org/abs/1905.07830. Verified 2026-07-27.
- MMLU (2021). Hendrycks et al. (UC Berkeley and collaborators). arxiv.org/abs/2009.03300. Verified 2026-06-29.
- MMLU-Pro (2024). TIGER-Lab (Wang et al., University of Waterloo). arxiv.org/abs/2406.01574. Verified 2026-06-29.
- MMLU-Redux (2024). Gema et al. (University of Edinburgh and collaborators). arxiv.org/abs/2406.04127. Verified 2026-06-29.
- SimpleQA (2024). OpenAI (Wei, Karina et al.). arxiv.org/abs/2411.04368. Verified 2026-06-29.
- SuperGPQA (2025). M-A-P Team (Du, Yao et al.). arxiv.org/abs/2502.14739. Verified 2026-07-27.
- TriviaQA (2017). Joshi, Choi, Weld et al.. arxiv.org/abs/1705.03551. Verified 2026-07-27.
- Global-MMLU (2024). Singh, Romanou, Fourrier et al.. arxiv.org/abs/2412.03304. Verified 2026-07-27.
- IFEval (2023). Zhou, Lu, Mishra et al.. arxiv.org/abs/2311.07911. Verified 2026-07-27.
- INCLUDE (2024). Romanou, Foroutan, Sotnikova et al.. arxiv.org/abs/2411.19799. Verified 2026-07-27.
- MGSM (2022). Shi, Suzgun, Freitag et al.. arxiv.org/abs/2210.03057. Verified 2026-07-27.
- Multi-IF (2024). He, Jin, Wang et al.. arxiv.org/abs/2410.15553. Verified 2026-07-27.
- BABILong (2024). Kuratov, Bulatov, Anokhin et al.. arxiv.org/abs/2406.10149. Verified 2026-07-27.
- HELMET (2024). Yen, Gao, Hou et al.. arxiv.org/abs/2410.02694. Verified 2026-07-27.
- LOFT (2024). Lee, Chen, Dai et al.. arxiv.org/abs/2406.13121. Verified 2026-07-27.
- LongBench (2023). Tsinghua University (THUDM; Bai et al.). arxiv.org/abs/2412.15204. Verified 2026-06-29.
- MRCR (2024). Google DeepMind (Michelangelo); open-source variant by OpenAI. arxiv.org/abs/2409.12640. Verified 2026-06-29.
- Needle-in-a-Haystack (2023). Greg Kamradt (independent). github.com/gkamradt/LLMTest_NeedleInAHaystack. Verified 2026-06-29.
- NoLiMa (2025). Adobe Research and LMU Munich (Modarressi et al.). arxiv.org/abs/2502.05167. Verified 2026-06-29.
- RULER (2024). NVIDIA (Hsieh, Sun et al.). arxiv.org/abs/2404.06654. Verified 2026-06-29.
- ChartQA (2022). Masry, Long, Tan et al.. arxiv.org/abs/2203.10244. Verified 2026-07-27.
- CharXiv (2024). Wang, Xia, He et al.. arxiv.org/abs/2406.18521. Verified 2026-07-27.
- DocVQA (2020). Mathew, Karatzas and Jawahar. arxiv.org/abs/2007.00398. Verified 2026-07-27.
- MathVista (2023). Lu et al. (UCLA, University of Washington, Microsoft Research). arxiv.org/abs/2310.02255. Verified 2026-06-29.
- MMBench (2023). Liu, Duan, Zhang et al.. arxiv.org/abs/2307.06281. Verified 2026-07-27.
- MMMU (2023). MMMU team (Yue et al.). arxiv.org/abs/2311.16502. Verified 2026-06-29.
- MMMU-Pro (2024). MMMU team (Yue et al.). arxiv.org/abs/2409.02813. Verified 2026-06-29.
- MMStar (2024). Chen, Li, Dong et al.. arxiv.org/abs/2403.20330. Verified 2026-07-27.
- Video-MME (2024). MME-Benchmarks team (Fu et al.). arxiv.org/abs/2405.21075. Verified 2026-06-29.
- FinanceBench (2023). Patronus AI (Islam, Kannappan, Kiela et al.). arxiv.org/abs/2311.11944. Verified 2026-07-27.
- HealthBench (2025). OpenAI (Arora, Wei, Soskin Hicks et al.). arxiv.org/abs/2505.08775. Verified 2026-07-27.
- LegalBench (2023). Guha, Nyarko, Ho et al.. arxiv.org/abs/2308.11462. Verified 2026-07-27.
- MedHELM (2025). Stanford CRFM. arxiv.org/abs/2505.23802. Verified 2026-07-27.
- MedQA (2020). Jin, Pan, Oufattole et al.. arxiv.org/abs/2009.13081. Verified 2026-07-27.
- AlpacaEval 2 (Length-Controlled) (2024). Dubois, Galambosi, Liang et al.. arxiv.org/abs/2404.04475. Verified 2026-07-27.
- Arena-Hard-Auto (2024). Li, Chiang, Frick et al.. arxiv.org/abs/2406.11939. Verified 2026-07-27.
- Artificial Analysis Coding Agent Index (2026). Artificial Analysis (independent). artificialanalysis.ai/methodology/coding-agents-benchmarking. Verified 2026-07-26.
- Artificial Analysis Intelligence Index (2024). Artificial Analysis (independent). artificialanalysis.ai/methodology/intelligence-benchmarking. Verified 2026-06-29.
- Copilot Arena (2025). Chi, Chen, Angelopoulos et al.. arxiv.org/abs/2502.09328. Verified 2026-07-27.
- Epoch Capabilities Index (2025). Epoch AI. epoch.ai/data/eci-documentation. Verified 2026-07-27.
- GDPval (2025). OpenAI, with an agentic re-run by Artificial Analysis as GDPval-AA. arxiv.org/abs/2510.04374. Verified 2026-07-26.
- HELM (2022). Stanford CRFM (Liang, Bommasani et al.). arxiv.org/abs/2211.09110. Verified 2026-06-29.
- LiveBench (2024). White, Dooley, Roberts et al.. arxiv.org/abs/2406.19314. Verified 2026-07-27.
- LMArena (2023). Arena (formerly LMArena and LMSYS Chatbot Arena; Angelopoulos, Chiang et al.). arxiv.org/abs/2403.04132. Verified 2026-07-03.
- METR Time Horizon (2025). METR (Kwa, West, Becker et al.). arxiv.org/abs/2503.14499. Verified 2026-07-27.
- MT-Bench (2023). LMSYS (Zheng et al., UC Berkeley). arxiv.org/abs/2306.05685. Verified 2026-06-29.
- HaluEval (2023). Li et al. (Renmin University of China). arxiv.org/abs/2305.11747. Verified 2026-06-29.
- TruthfulQA (2021). Lin, Hilton, Evans (Oxford and OpenAI). arxiv.org/abs/2109.07958. Verified 2026-06-29.
- Vectara Hallucination Leaderboard (2023). Vectara (Hughes et al.). github.com/vectara/hallucination-leaderboard. Verified 2026-06-29.
- AgentHarm (2024). Andriushchenko, Souly, Dziemian et al.. arxiv.org/abs/2410.09024. Verified 2026-07-27.
- Cybench (2024). Zhang, Perry, Dulepet et al.. arxiv.org/abs/2408.08926. Verified 2026-07-27.
- CyberSecEval 3 (2024). Meta (Wan, Nikolaidis, Song et al.). arxiv.org/abs/2408.01605. Verified 2026-07-27.
- WMDP (2024). Li, Pan, Gopal et al.. arxiv.org/abs/2403.03218. Verified 2026-07-27.
Machine-readable data: /ai-benchmarks.json. Benchmark reliability ratings are from our benchmark trust scorecard.