Benchmark Saturation: Why 99% on MMLU Means Almost Nothing
Benchmark saturation is when top AI models bunch so close to the ceiling a test cannot rank them. Why MMLU hit this wall, and the harder replacement is next.
By Capital & Compute
Say a model scored 99% on MMLU tomorrow. Would that tell you it is the best model available?
No. It would tell you almost nothing, and the reason has nothing to do with the number itself. The top of MMLU is already so crowded that a 99% would sit inside a cluster of scores separated by fractions of a point, the same way the current leaders are. A single test that cannot tell the best model from the fifth-best model is not measuring anything useful anymore, no matter how high the number climbs. That state has a name: saturation.
What saturation actually means
Saturation is not “the test got easy.” It is a specific statistical failure: a benchmark saturates when top models cluster so close to the ceiling that the score stops discriminating between them. The test can still measure something, but it can no longer rank the field, which is the one job a leaderboard exists to do.
MMLU is the textbook case. Frontier models currently sit between 88% and 94% on it, and the top handful, Qwen3.7 Max, GPT-5, and OpenAI’s o3 among them, are separated by fractions of a percentage point according to leaderboard trackers as of late June 2026. The field has also crossed a line worth sitting with: those top scores now sit above the roughly 89.8% average scored by human subject-matter experts on the same test. A test that everyone passes better than the experts who wrote it has stopped being a test of expertise. It has become a test of who can shave the last half-point off a number nobody needed measured that precisely in the first place.
The pillar piece on this site already walked through the ways a benchmark score gets gamed, contaminated, or disconnected from real work, and a companion post covers how contamination specifically leaks test answers into training data. Saturation is a different failure mode from both. A benchmark can be perfectly clean of leaked answers and still be useless, simply because every model that matters has learned to answer its questions well enough that the questions no longer sort anyone.
The harder replacement is saturating too
The field’s standard response to a saturated benchmark is to build a harder one. MMLU-Pro exists for exactly this reason: ten answer options instead of four, harder questions, explicitly built to give frontier models room to spread out again.
It bought some room. It did not buy much.
Pulled directly from the same benchmark data this site’s own AI benchmarks tracker and model leaderboard run on, thirteen frontier models spanning Google, Anthropic, OpenAI, xAI, and DeepSeek score between 84.9% and 89.5% on MMLU-Pro, a spread of under five points. That band includes models released a full generation apart: Claude Opus 4, an older release, sits at 86.0%, within a point of several 2026 flagships. The replacement built to restore discriminative power is running the same curve as the test it replaced, just a few laps behind.
| Item | Highest of top cluster | Lowest of top cluster |
|---|---|---|
| MMLU | 93.7% | 93.1% |
| MMLU-Pro | 89.5% | 84.9% |
| Humanity's Last Exam | 53.3% | 44.7% |
Read the chart as a story about runway, not just a snapshot. MMLU has almost none left: a 0.6-point band at the top. MMLU-Pro has more, but the band is already under five points and closing. Humanity’s Last Exam, a 3,000-question set crowdsourced from subject-matter experts across math, physics, computer science, and specialist domains specifically to outrun this problem, still shows real separation: Claude Fable 5 leads at 53.3%, Claude Opus 4.8 follows at 45.7%, and Gemini 3.1 Pro Preview sits at 44.7%, a genuine 8.6-point spread among the top three, per leaderboard aggregators tracking the benchmark as of late June 2026. Expert humans average roughly 90% accuracy on HLE questions inside their own specialty. That gap, between a 53.3% leader and a 90% human ceiling, is the headroom MMLU ran out of years ago.
A test everyone passes is not measuring skill anymore. It is measuring who can shave the last half-point off a number nobody needed that precisely.
Why “make it harder” keeps running out of road
If a harder test just re-saturates on the same schedule, the difficulty dial cannot be the actual fix, and a 2026 paper on this exact question backs that up with data instead of intuition. Akhtar and 36 co-authors, in a paper accepted at ICML 2026, analyzed 60 language-model benchmarks across 14 properties tied to saturation. Their headline finding: nearly half the benchmarks they studied already show saturation, and the rate climbs with a benchmark’s age. Every test is on a clock from the day it ships.
The more useful finding is buried past the headline. The paper reports that resilience to saturation tracks to expert curation of the test items, not to whether the test set is kept public or private. That cuts against the instinct to solve saturation by locking a benchmark behind a private eval, the way some operators do to fight contamination. A private set of easy, poorly calibrated questions will saturate on the same timeline as a public one. What holds up is items built by people who understand exactly where the difficulty ceiling of the field currently sits, and place questions just past it, the way a well-designed exam does.
That is closer to a testing-design problem than a machine-learning problem, and testing design is a field with a hundred years of head start. A small but growing body of 2026 research is importing it directly: applying Item Response Theory, the psychometric framework that scores standardized tests like the SAT and GRE by each question’s difficulty and its ability to discriminate between test-takers, rather than by a flat percent-correct. One recent paper proposes auditing existing LLM benchmarks through this exact lens, scoring individual questions rather than treating every item as equally informative. The pitch is straightforward: stop asking “is this test hard enough,” and start asking “does each question on this test actually separate a strong model from a weak one,” which is a question raw accuracy can never answer on its own.
What this means for reading a leaderboard
None of this makes MMLU worthless. It still tells you whether a model cleared a basic competence bar, and a score that falls well below the 88-94% frontier band is a real signal. What it cannot do anymore is rank the top of the field, and a launch post that leads with an MMLU score to claim a model is “the best” is quoting a number that lost its ranking power some time ago.
The practical habit: when a comparison leans on a benchmark, check where that benchmark sits on the saturation curve before trusting the ranking it produces. A tight top cluster on an old, famous, public test is a sign to look elsewhere, toward agent-specific evaluations built closer to real work or a benchmark still young enough to have room, like Harbor-Index, where nothing yet clears 30%. And because capability claims feed directly into what a model is worth paying for, a saturated benchmark backing a price premium is exactly the kind of number worth pressure-testing before you pay for it.
Frequently asked questions
- What is benchmark saturation?
- Benchmark saturation is when top-performing models score so close to the ceiling of a test that the score can no longer distinguish between them. The test stops functioning as a ranking tool even though the numbers still look impressive, because a benchmark exists to separate models by ability and a saturated one can no longer do that job.
- Is MMLU saturated?
- Yes. Frontier models cluster between 88% and 94% on MMLU as of mid-2026, with the top handful separated by fractions of a percentage point and now scoring above the roughly 89.8% human-expert baseline. A test everyone passes at expert level or better has lost its ability to rank the field.
- Is MMLU-Pro also saturated?
- It is heading that way. MMLU-Pro was built with harder questions specifically to restore discriminative power after MMLU saturated. As of mid-2026, thirteen frontier models across five labs, including models a generation apart in release date, score within a five-point band (84.9% to 89.5%), a much tighter spread than the benchmark's design intended.
- How do researchers fix benchmark saturation?
- The two approaches with real evidence behind them are expert curation of test items (a 2026 study of 60 benchmarks found resilience to saturation tracks to who built the questions, not whether the test is public or private) and Item Response Theory, a psychometric framework borrowed from standardized testing that scores individual questions by difficulty and discriminative power instead of grading a test purely on percent correct.
- What is Humanity's Last Exam, and why does it matter for saturation?
- Humanity's Last Exam is a 3,000-question benchmark crowdsourced from subject-matter experts specifically to give frontier models room to be told apart. As of mid-2026 its top score was 53.3%, with a real gap down to the third-place model and roughly 90% human-expert accuracy left as headroom, unlike MMLU or MMLU-Pro where the top of the field has already bunched together.
Sources
- LLM-Stats. (2026). MMLU Leaderboard [benchmark aggregator; frontier model scores as of late June 2026]. llm-stats.com/benchmarks/mmlu
- Pricepertoken. (2026). MMLU-Pro benchmark data [benchmark aggregator; the dataset backing this site’s
/ai-benchmarks/and/ai-model-leaderboard/pages]. pricepertoken.com - Akhtar, M., et al. (2026). When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation [arXiv preprint, accepted at ICML 2026; 60 benchmarks, 14 saturation properties, expert-curation finding]. arxiv.org/abs/2602.16763
- Chen, Y., et al. (2025). Lost in Benchmarks? Rethinking Large Language Model Benchmarking with Item Response Theory [preprint]. arxiv.org/html/2505.15055v1
- (2026). Auditing LLM Benchmarks with Item Response Theory [preprint]. arxiv.org/pdf/2605.30504
- Wikipedia. (2026). Humanity’s Last Exam [reference summary of the benchmark’s design and scope]. en.wikipedia.org/wiki/Humanity’s_Last_Exam