SWE-bench
Also known as SWE-bench Full
SWE-bench is a software-engineering benchmark that gives an AI system a real bug report from a real open-source repository and asks it to produce a patch. The patch is judged by running the repository’s own hidden tests: they pass or they do not. It is the benchmark that turned "coding model" into "coding agent," because solving a task requires navigating a codebase rather than writing one function.
| What it measures | Whether a system can resolve a real GitHub issue by generating a patch that passes the repository's hidden tests. |
|---|---|
| Built by | Princeton and Stanford (Jimenez, Yang, Yao et al.), 2023 |
| Format | 2,294 real GitHub issue and merged-PR pairs across 12 popular Python repos (also subsetted into Lite: 300, Verified: 500) |
| Scoring metric | % resolved (pass@1) |
| Status | Saturated |
| Representative top score | Not independently confirmed; see the leaderboard |
| Official leaderboard | www.swebench.com |
How SWE-bench works
Each task pairs a genuine GitHub issue with the pull request that actually fixed it. The agent sees the issue text and the repository at the commit before the fix, and must produce a patch. Grading is fully automatic and binary: the harness applies the patch and runs two test sets, the tests that the real fix made pass, and the tests that already passed and must not break. The headline number is the percentage of tasks resolved on a single attempt.
History and current status
The benchmark was introduced in a 2023 paper by researchers at Princeton and Stanford, when the best systems resolved a low single-digit percentage of tasks. Two derivative sets quickly became more important than the original: Lite, a 300-task subset, and Verified, a 500-task human-validated subset released by OpenAI in 2024. By 2026 the full 2,294-task set is rarely reported at all, and attention has moved on again to SWE-bench Pro and to continuously refreshed variants such as SWE-bench-Live and SWE-rebench.
What the score does not tell you
Three problems compound. The tasks are drawn from public repositories that are certainly in every frontier training corpus, so a model may have seen the actual fix. A large share of the original tasks turned out to have broken or underspecified tests, which is exactly why Verified exists. And a reported score depends heavily on the agent scaffolding wrapped around the model, so two numbers for the same model can differ by many points without either being wrong.
This is the general case, not a quirk of one benchmark. For the four failure modes that decide whether any benchmark number means anything, read are AI benchmarks reliable, and for this row in particular, why benchmark saturation makes a high score meaningless.
Who reports SWE-bench, and how to read it
Almost every frontier lab quotes a SWE-bench figure at launch, but nearly always the Verified subset rather than the full set, and usually under its own scaffolding. Read any such number as a model-plus-harness result. OpenAI said in early 2026 that it had stopped reporting Verified after finding broken tests and training exposure, and now points to SWE-bench Pro instead.
Benchmarks to read alongside this one
SWE-bench Verified
The same real-GitHub-issue resolution task as SWE-bench, restricted to a human-validated subset where the issue is solvable and the tests are not broken.
SWE-bench Pro
Whether agents can solve long-horizon, enterprise-grade software-engineering tasks under standardized scaffolding, designed to resist contamination.
SWE-bench-Live
The same real-GitHub-issue resolution task as SWE-bench, but on tasks harvested continuously from issues created after a model was trained.
SWE-bench: frequently asked questions
- What is SWE-bench?
- SWE-bench is a benchmark of 2,294 real GitHub issue and pull-request pairs from 12 popular Python repositories. An AI system is given the issue and the repository as it stood before the fix, and must generate a patch that makes the repository hidden tests pass. The score is the percentage of issues resolved on one attempt.
- Is SWE-bench still a good benchmark in 2026?
- Not on its own. The full set is effectively retired in favour of subsets and successors, top scores on the Verified subset sit near the ceiling, and the tasks come from public repositories that models have almost certainly trained on. Use it as history and read SWE-bench Pro, SWE-bench-Live or SWE-rebench for a current signal.
- What is a good SWE-bench score?
- There is no fixed threshold, because the number depends on which subset and which agent scaffolding produced it. What matters is comparability: only compare scores measured on the same subset under the same scaffolding, and treat a vendor-reported figure as a model-plus-harness result rather than a property of the model.
- Who created SWE-bench?
- It was introduced in a 2023 paper by researchers at Princeton and Stanford (Jimenez, Yang, Yao and colleagues). The same authors later collaborated with OpenAI on the human-validated SWE-bench Verified subset.
Get each breakdown before it makes the rounds
You get one email when a new source-backed analysis goes live: what AI agents actually cost, which models are worth running, and what the benchmarks really mean. No hype.
No spam. Unsubscribe anytime.