SWE-bench

How SWE-bench works

Each task pairs a genuine GitHub issue with the pull request that actually fixed it. The agent sees the issue text and the repository at the commit before the fix, and must produce a patch. Grading is fully automatic and binary: the harness applies the patch and runs two test sets, the tests that the real fix made pass, and the tests that already passed and must not break. The headline number is the percentage of tasks resolved on a single attempt.

History and current status

The benchmark was introduced in a 2023 paper by researchers at Princeton and Stanford, when the best systems resolved a low single-digit percentage of tasks. Two derivative sets quickly became more important than the original: Lite, a 300-task subset, and Verified, a 500-task human-validated subset released by OpenAI in 2024. By 2026 the full 2,294-task set is rarely reported at all, and attention has moved on again to SWE-bench Pro and to continuously refreshed variants such as SWE-bench-Live and SWE-rebench.

What the score does not tell you

Three problems compound. The tasks are drawn from public repositories that are certainly in every frontier training corpus, so a model may have seen the actual fix. A large share of the original tasks turned out to have broken or underspecified tests, which is exactly why Verified exists. And a reported score depends heavily on the agent scaffolding wrapped around the model, so two numbers for the same model can differ by many points without either being wrong.

This is the general case, not a quirk of one benchmark. For the four failure modes that decide whether any benchmark number means anything, read are AI benchmarks reliable, and for this row in particular, why benchmark saturation makes a high score meaningless.

SWE-bench is not one of the 15 benchmarks scored in the benchmark trust ranking, which rates the most-quoted tests on those same four axes and sorts them most trustworthy first.

What SWE-bench scores actually mean

Read the Full-set number as a floor, not a grade. When SWE-bench launched, the best system resolved under 2% of the 2,294 tasks, and for the next two years single-digit to low-double-digit scores were the state of the art. A model at 20% on the Full set is not weak; it is operating on a distribution that includes issues no competent engineer could close from the text given. That is precisely why the number stopped being reported: once labs realised a meaningful share of the set was unsolvable as written, the ceiling of the Full set was somewhere well below 100%, and nobody could say where. Comparisons across papers are also unsafe unless both used the same retrieval setup, because the version that hands the model the right files scores far above the version that makes it find them.

Who reports SWE-bench, and how to read it

Almost every frontier lab quotes a SWE-bench figure at launch, but nearly always the Verified subset rather than the full set, and usually under its own scaffolding. Read any such number as a model-plus-harness result. OpenAI said in early 2026 that it had stopped reporting Verified after finding broken tests and training exposure, and now points to SWE-bench Pro instead.

When to weight SWE-bench in a model choice

Treat SWE-bench Full as history rather than as a purchasing input. If a vendor quotes a plain SWE-bench number in 2026 without saying Verified, Pro, Lite or Full, that ambiguity is itself the finding: the four differ by tens of points and the label is doing the work. For an actual model choice on repository work, read SWE-bench Pro or Terminal-Bench, both of which run under standardized scaffolding and still have headroom. Keep the Full set for one purpose: it is the reference point that shows how fast the field moved, from under 2% to near-ceiling on its cleaned subset in roughly two years, which is the context that makes any single current score legible.