LiveCodeBench

How LiveCodeBench works

Problems are harvested continuously from LeetCode, AtCoder and Codeforces, with more than a thousand collected, and each carries a release date. When a model is evaluated, the harness filters to problems published after that model’s training cutoff. Alongside code generation it scores self-repair, code execution and test-output prediction, which separates writing code from reasoning about it. The headline metric is pass@1.

History and current status

Researchers at UC Berkeley, MIT and Cornell introduced it in 2024, at the point where HumanEval had saturated and contamination had become the central complaint about code benchmarks. Its time-stamping approach was influential: the same principle now underpins SWE-bench-Live, SWE-rebench and LiveBench’s monthly rotation.

What the score does not tell you

Competitive programming is a narrow slice of software work: short, self-contained, algorithmically dense problems with clean specifications, which is nothing like maintaining a codebase. Filtering by training cutoff also depends on the cutoff being accurately known and honestly reported, and scores computed over different date windows are not strictly comparable.

This is the general case, not a quirk of one benchmark. For the four failure modes that decide whether any benchmark number means anything, read are AI benchmarks reliable, and for this row in particular, how benchmark contamination inflates scores.

What LiveCodeBench scores actually mean

A LiveCodeBench score is only interpretable alongside the date window it was computed over. Because problems are time-stamped and a model is scored only on those published after its training cutoff, the same model evaluated on two different windows produces two different numbers, and a newer model is often evaluated on a harder, more recent slice. That is the design working as intended, and it is also why the leaderboard should be read as a ranking within a window rather than as an absolute capability score. Competitive-programming problems also skew toward algorithmic puzzle-solving, so a strong number predicts contest performance far better than it predicts whether a model can work in an unfamiliar codebase.

Who reports LiveCodeBench, and how to read it

It is academically maintained with a public leaderboard, and labs cite it reasonably often because the contamination story is credible. Check which date window a quoted figure covers, since the problem pool grows continuously and a number without a window is ambiguous. Two models evaluated on different windows are not directly comparable, however similar the headline figures look.

When to weight LiveCodeBench in a model choice

Use LiveCodeBench as the contamination control on any other coding claim. If a model posts a strong HumanEval or MMLU number but a mediocre LiveCodeBench number on a post-cutoff window, memorisation is the most economical explanation. It is the right benchmark for algorithmic reasoning and a weak proxy for software engineering, so pair it with SWE-bench Pro or Terminal-Bench rather than substituting it. When quoting a figure, record the evaluation window alongside it; a LiveCodeBench score without its date range cannot be reproduced or compared.