SWE-bench Verified

How SWE-bench Verified works

The task is identical to SWE-bench: read a real GitHub issue, patch the repository, pass the hidden tests. The difference is curation. Professional developers reviewed candidate tasks and kept only those where the issue statement was sufficient to produce the fix and the test suite correctly distinguished a fix from a non-fix. The result is 500 tasks where a failure is much more likely to be the model’s fault than the benchmark’s fault.

History and current status

OpenAI released Verified in 2024, working with the original SWE-bench authors, after analysis showed a substantial fraction of the full set was unsolvable as specified. It rapidly displaced the full set in model cards. Scores climbed from the low tens of percent to near the ceiling within roughly two years. In February 2026 OpenAI said it had stopped reporting Verified, citing broken tests and training-data exposure, and recommended the harder SWE-bench Pro.

What the score does not tell you

Verified fixed the task-quality problem but not the contamination problem: the underlying repositories and their fix commits remain public. When frontier models bunch near the top, the remaining spread is scaffolding and luck rather than capability. That is why a very high Verified score is now better read as a contamination and saturation signal than as evidence of engineering skill.

This is the general case, not a quirk of one benchmark. For the four failure modes that decide whether any benchmark number means anything, read are AI benchmarks reliable, and for this row in particular, why benchmark saturation makes a high score meaningless.

Scored on those four axes, SWE-bench Verified carries a concern score of 7 out of 8 and ranks 13 of 15, which puts it in the group to stop quoting. See the reasoning behind that rating and how it compares with the rest of the field.

What SWE-bench Verified scores actually mean

The usable range on Verified closed years ago. In 2024 a frontier model landed in the 30s to 40s; by 2025 the leaders were in the 60s and 70s; by 2026 they cluster above 80% with the top reading near 95%. Inside that top band the spread is not capability. Re-running the same model under a different agent harness moves the number by ten points or more, so a two-point gap between two labs' reported figures carries no information about which model is better at fixing code. Below roughly 40% the score still discriminates, which is why Verified remains a reasonable smoke test for a small or open-weight model, and useless for ranking the frontier.

Who reports SWE-bench Verified, and how to read it

Verified was the default coding row in launch tables from 2024 through 2026, usually reported under each lab’s own agent harness, which makes cross-lab comparison unreliable. Its own originator has now stepped away from it. Where a lab still quotes it, check whether a Pro or Terminal-Bench number is reported alongside, and weight those instead.

When to weight SWE-bench Verified in a model choice

Use Verified to answer one question: can this model do repository-scale editing at all? That is a real question for a 7B local model or a cheap API tier, and Verified answers it cheaply. Do not use it to choose between frontier models, and treat a launch table that leads with Verified in 2026 as a soft signal that the harder numbers were less flattering. The organisation that built the set stopped reporting it in February 2026 and pointed at SWE-bench Pro instead, which is the strongest possible statement about its remaining value. Where a lab quotes Verified, look for a Pro or Terminal-Bench figure in the same table and weight that one.