MMLU

How MMLU works

The benchmark presents roughly 15,900 four-option multiple-choice questions spanning STEM, the humanities, the social sciences and professional exams such as law and medicine. Scoring is plain accuracy, with a random-guess baseline of 25%. Because it is multiple choice and fully automated, it is cheap to run, which is a large part of why it became ubiquitous.

History and current status

Introduced in a 2021 paper by Hendrycks and collaborators, when strong models scored barely above chance on many subjects, MMLU became the headline row in every model card through the GPT-3 and GPT-4 era. Frontier models passed the 90% mark, and by mid-2026 the top of the range sits around 93%. Its successors, MMLU-Pro and MMLU-Redux, exist specifically because it stopped working.

What the score does not tell you

MMLU has three separate problems. It is saturated, so the remaining spread between frontier models is mostly noise. It is contaminated, because the questions have circulated online for years. And independent analysis found genuine errors in its ground-truth answers, meaning a perfect model could not score 100%. MMLU-Redux was built to correct that last problem specifically.

This is the general case, not a quirk of one benchmark. For the four failure modes that decide whether any benchmark number means anything, read are AI benchmarks reliable, and for this row in particular, why benchmark saturation makes a high score meaningless.

Scored on those four axes, MMLU carries a concern score of 8 out of 8 and ranks 15 of 15, which puts it in the group to stop quoting. See the reasoning behind that rating and how it compares with the rest of the field.

What MMLU scores actually mean

MMLU stopped discriminating around 2024. Frontier models now sit above 90% on roughly 15,900 four-option questions, and the leaders read near 93%, so the gap between first and tenth place is a few percentage points on a set known to contain ground-truth errors. Independent analyses have found mislabeled answers across multiple subjects, which means the practical ceiling is below 100% and nobody agrees on where. Combined with a 25% random baseline and near-certain contamination, a 2026 MMLU score tells you a model was trained on the open internet. It does not tell you which of two models reasons better, and a two-point difference is not a result.

Who reports MMLU, and how to read it

Model developers still include MMLU in comparison tables, largely out of convention and because the number is high. Treat its appearance as a signal about the table rather than the model: if a launch leads with MMLU rather than a current unsaturated benchmark, that is a choice about presentation. The score itself has not carried information about frontier capability for roughly two years.

When to weight MMLU in a model choice

Skip MMLU for model selection. Its remaining value is longitudinal: it is the cleanest illustration in this directory of a benchmark completing its life cycle, from a genuinely hard 2021 exam that GPT-3 answered near chance to a saturated formality quoted out of habit. If you need the capability MMLU was built to measure, use MMLU-Pro, which raised the option count to ten and reintroduced 16 to 33 points of headroom, or the Epoch Capabilities Index, which bridges across benchmarks as individual ones top out. Treat a launch table leading with MMLU in 2026 as padding.