HumanEval is a 2021 OpenAI benchmark of 164 hand-written Python problems. Each gives a function signature and docstring, and the model must write a body that passes hidden unit tests. The score is pass@1, the share solved on the first attempt.

Is HumanEval still used?

Rarely, and not for frontier models. Top models score around 99%, so the benchmark cannot rank them. It survives mainly as a historical baseline and in comparisons of smaller open-weight models where there is still headroom.

HumanEval+ is the EvalPlus version of the same problems, tested against roughly 80 times more test cases. It exists because the original tests were thin enough that incorrect solutions passed, which means original HumanEval scores overstated real correctness.

HumanEval: The Retired Code Benchmark, Explained

HumanEval

HumanEval is a set of 164 hand-written Python problems where the model sees a function signature and a docstring and must write a body that passes hidden unit tests. It was the original standard for code generation and is now solved: frontier models sit near 99%, and later work showed even those scores were partly an artefact of thin testing.

Key facts about the HumanEval benchmark
What it measures	Whether a model can synthesize a single correct Python function from a docstring so that it passes the provided unit tests.
Built by	OpenAI (Chen et al.), 2021
Format	164 hand-written Python problems, each with a signature, docstring and hidden unit tests
Scoring metric	pass@k (primarily pass@1)
Status	Saturated
Representative top score	~99% · Frontier models broadly · read 2025-04

What it measures

Whether a model can synthesize a single correct Python function from a docstring so that it passes the provided unit tests.

Built by

OpenAI (Chen et al.), 2021

Format

164 hand-written Python problems, each with a signature, docstring and hidden unit tests

Scoring metric

pass@k (primarily pass@1)

Status

Saturated

Representative top score

~99% · Frontier models broadly · read 2025-04

How HumanEval works

Each problem gives a function signature, a natural-language docstring and a small set of hidden unit tests. The model writes the body, the harness runs the tests, and the result is pass or fail. The reported metric is pass@k, in practice almost always pass@1, meaning the share solved on a single attempt.

History and current status

OpenAI introduced it in the 2021 Codex paper, when the best models solved a modest fraction of problems. It became the default code row for years. In 2023 EvalPlus rescored the same problems against roughly 80 times more tests and showed that a meaningful share of solutions counted as correct actually failed proper testing. By 2026 the benchmark itself is at roughly 99% and is kept only as a historical baseline.

What the score does not tell you

HumanEval measures the narrowest possible slice of programming: one short, self-contained function with no dependencies, no repository context and no ambiguity. Real engineering work is none of those things, which is why the field moved to SWE-bench and then to agentic terminal benchmarks. It is also almost certainly contaminated, since 164 widely republished problems appear throughout public training data.

This is the general case, not a quirk of one benchmark. For the four failure modes that decide whether any benchmark number means anything, read are AI benchmarks reliable, and for this row in particular, why benchmark saturation makes a high score meaningless.

Who reports HumanEval, and how to read it

It occasionally still appears in smaller-model and open-weight comparisons, where the ceiling has not been reached. A HumanEval number in a frontier model card in 2026 is decorative. If you need a function-level signal, EvalPlus is the better-tested version of the same problems.

Benchmarks to read alongside this one

HumanEval: frequently asked questions

What is HumanEval?: HumanEval is a 2021 OpenAI benchmark of 164 hand-written Python problems. Each gives a function signature and docstring, and the model must write a body that passes hidden unit tests. The score is pass@1, the share solved on the first attempt.
Is HumanEval still used?: Rarely, and not for frontier models. Top models score around 99%, so the benchmark cannot rank them. It survives mainly as a historical baseline and in comparisons of smaller open-weight models where there is still headroom.
What is HumanEval+?: HumanEval+ is the EvalPlus version of the same problems, tested against roughly 80 times more test cases. It exists because the original tests were thin enough that incorrect solutions passed, which means original HumanEval scores overstated real correctness.

You get one email when a new source-backed analysis goes live: what AI agents actually cost, which models are worth running, and what the benchmarks really mean. No hype.

No spam. Unsubscribe anytime.