HumanEval

How HumanEval works

Each problem gives a function signature, a natural-language docstring and a small set of hidden unit tests. The model writes the body, the harness runs the tests, and the result is pass or fail. The reported metric is pass@k, in practice almost always pass@1, meaning the share solved on a single attempt. There is no agent loop, no repository and no retrieval step, which is why a HumanEval result says nothing about behaviour inside a real editing harness.

History and current status

OpenAI introduced it in the 2021 Codex paper, when the best models solved a modest fraction of problems. It became the default code row for years. In 2023 EvalPlus rescored the same problems against roughly 80 times more tests and showed that a meaningful share of solutions counted as correct actually failed proper testing. By 2026 the benchmark itself is at roughly 99% and is kept only as a historical baseline.

What the score does not tell you

HumanEval measures the narrowest possible slice of programming: one short, self-contained function with no dependencies, no repository context and no ambiguity. Real engineering work is none of those things, which is why the field moved to SWE-bench and then to agentic terminal benchmarks. It is also almost certainly contaminated, since 164 widely republished problems appear throughout public training data.

This is the general case, not a quirk of one benchmark. For the four failure modes that decide whether any benchmark number means anything, read are AI benchmarks reliable, and for this row in particular, why benchmark saturation makes a high score meaningless.

Scored on those four axes, HumanEval carries a concern score of 8 out of 8 and ranks 14 of 15, which puts it in the group to stop quoting. See the reasoning behind that rating and how it compares with the rest of the field.

What HumanEval scores actually mean

HumanEval has no usable range left. Frontier models sit near 99% pass@1 on 164 problems, which means the entire spread between the best and the tenth-best model is one or two problems, well inside the noise from prompt formatting alone. Even in 2023 the set was small enough that a single ambiguous docstring moved the reported figure by more than half a point. The problems are short, self-contained, and have circulated in training corpora for years, so a high score is now evidence that a model has seen Python before rather than evidence it can write it. Below about 70% the number still separates small and quantized models, which is the only band where it retains any resolution.

Who reports HumanEval, and how to read it

It occasionally still appears in smaller-model and open-weight comparisons, where the ceiling has not been reached. A HumanEval number in a frontier model card in 2026 is decorative. If you need a function-level signal, EvalPlus is the better-tested version of the same problems. Its continued appearance is mostly inertia: it is cheap to run and the number always looks good.

When to weight HumanEval in a model choice

Do not use HumanEval to choose a model in 2026. Its one remaining legitimate use is as a sanity check on a heavily quantized or fine-tuned local model, where a collapse from 90% to 60% is a fast signal that the quantization broke something. Treat a vendor still leading with HumanEval as a flag to go looking for the harder numbers. For any real comparison, use LiveCodeBench for algorithmic work and SWE-bench Pro or Terminal-Bench for repository work; all three still have headroom and none of them can be passed by recall alone.