LiveCodeBench

How LiveCodeBench works

Problems are harvested continuously from LeetCode, AtCoder and Codeforces, with more than a thousand collected, and each carries a release date. When a model is evaluated, the harness filters to problems published after that model’s training cutoff. Alongside code generation it scores self-repair, code execution and test-output prediction, which separates writing code from reasoning about it. The headline metric is pass@1.

History and current status

Researchers at UC Berkeley, MIT and Cornell introduced it in 2024, at the point where HumanEval had saturated and contamination had become the central complaint about code benchmarks. Its time-stamping approach was influential: the same principle now underpins SWE-bench-Live, SWE-rebench and LiveBench’s monthly rotation.

What the score does not tell you

Competitive programming is a narrow slice of software work: short, self-contained, algorithmically dense problems with clean specifications, which is nothing like maintaining a codebase. Filtering by training cutoff also depends on the cutoff being accurately known and honestly reported, and scores computed over different date windows are not strictly comparable.

This is the general case, not a quirk of one benchmark. For the four failure modes that decide whether any benchmark number means anything, read are AI benchmarks reliable, and for this row in particular, how benchmark contamination inflates scores.

Who reports LiveCodeBench, and how to read it

It is academically maintained with a public leaderboard, and labs cite it reasonably often because the contamination story is credible. Check which date window a quoted figure covers, since the problem pool grows continuously and a number without a window is ambiguous.