Terminal-Bench

How Terminal-Bench works

Version 4.0 contains 66 human-verified containerized tasks across 7 domains, from software engineering and sysadmin to data science, ML, security and longer-horizon work needing multi-container networks or GPUs. Each task is attempted 5 times, so a full leaderboard entry is 330 trials, and every task carries a flat 8-hour agent timeout. Grading is strictly pass or fail: the agent container and the verifier container are separate, and artifacts are downloaded from the agent and uploaded to the verifier at the end of a trial, which blocks many reward-hacking routes and lets results be re-graded when a verifier is fixed. Because the agent operates a real shell, a score reflects the model and its scaffolding together, not the model alone.

History and current status

Built by the Terminal-Bench team with the Laude Institute, it became the headline agentic benchmark of 2026 as SWE-bench Verified saturated. Its version history is the important part. Version 2.1 (May 2026) ran 89 tasks and topped out near 83%. Version 3.0 (2026-07-30) replaced the set with 74 harder tasks across 7 domains, developed under the working name Frontier-Bench, and the top score at launch fell to 34.4% (the re-run 3.0 board now tops out at 42.7%, Claude Opus 5 on mini-SWE-agent). Version 4.0 (2026-08-28) removed 8 tasks and fixed 19, leaving 66. Claude Opus 5 led it at 51.82% until GPT-6 Astra entered on its 2026-09-03 launch day and took the top score at 58.18% in Codex at max effort, with Claude Fable 5.1 and two lower Astra effort settings tied at 57.88% just behind; the board grew from 10 entries to 18 in the same week, and to 27 by late September. Grok 4.7 entered on its 2026-09-21 launch day at 37.58%, run in xAI's own Grok Build harness and flagged partial at 324 of 330 trials. A batch recorded on 2026-10-06 took the board to 35 entries: Claude Opus 5.5 took the top score at 64.85%, Claude Sonnet 5.5 placed second at 61.82%, and GPT-6.1 Sol tied Astra at 58.18% for about a fifth of the cost per solved task. The same month the per-version leaderboard URLs were retired in favour of the tbench.ai homepage. From 3.0 onward it is governed by a published semantic-versioning policy: major releases change the environment or task set and force full re-runs, minor releases change verifiers and re-grade saved artifacts, patches change nothing that moves a score. Milestones for 4.1 (tamper-resistant verifiers) and 5.0 (new tasks) are open and unshipped as of 2026-10-20; 4.1 has passed its 2026-09-26 due date.

What the score does not tell you

Scaffolding dependence is the main caveat: the same model scores differently under different harnesses, so a Terminal-Bench number is a statement about a stack rather than a model. Grok 4.7 is the clearest case on record: the official board gives it 37.58% in xAI's own Grok Build harness, while Artificial Analysis measures 25.8% for the same model on its standardized mini-swe-agent harness, a spread of nearly 12 points with the weights held constant. Version churn is the second. A figure quoted without its version is close to meaningless, because 2.1, 3.0 and 4.0 differ by more than 30 points on the same name. The third is resolution: the published 95% confidence half-widths run from 2.45 to 3.94 points on the 4.0 board, and Anthropic research on infrastructure noise found a 6-point swing from resource configuration alone, recommending skepticism of any gap under 3 points. Pass-or-fail grading is honest but discards partial progress, which is noisier on a 66-task set than a larger one.

This is the general case, not a quirk of one benchmark. For the four failure modes that decide whether any benchmark number means anything, read are AI benchmarks reliable, and for this row in particular, how benchmark contamination inflates scores.

Scored on those four axes, Terminal-Bench carries a concern score of 0 out of 8 and ranks 2 of 15, which puts it in the group worth quoting as it stands. See the reasoning behind that rating and how it compares with the rest of the field.

What Terminal-Bench scores actually mean

Terminal-Bench grades the end state of a machine, so a score is the share of jobs actually finished, not the share of plausible attempts. On v4.0 the leader reads 64.85%, meaning the best available agent still fails roughly 1 in 3 of the 66 verified tasks across its trials. The version number is load-bearing and comparisons across it are invalid: the current boards read 87.4% on v2.1, 42.7% on v3.0 and 64.85% on v4.0, because each release swapped in a different task set under the same name. Treat any Terminal-Bench figure quoted without a version as unusable. The board also publishes total run cost, which spans $1.34 to $234.24 across the costed v4.0 entries, so cost per solved task differs by more than an order of magnitude between agents scoring similarly: GPT-6.1 Sol and GPT-6 Astra both score 58.18%, at $3.30 and $17.02 per solved task.

Who reports Terminal-Bench, and how to read it

Frontier labs quote it at launch, and tbench.ai runs the official board with grant support it discloses from OpenAI, Anthropic, Z.ai, SpaceX AI and the Laude Institute; Artificial Analysis has published independent runs of earlier versions. Always check the version, since 2.1, 3.0 and 4.0 are separate task sets, and check the harness, since the agent is named on every board row. Vendor-reported figures frequently lag the official board by a version. One practical trap: tbench.ai has retired its per-version leaderboard URLs, which now redirect to the homepage, and before that the version segment was ignored, so figures read off an old leaderboard address may be 4.0 rows under an older label.

When to weight Terminal-Bench in a model choice

Weight Terminal-Bench when the question is whether an agent can be trusted to complete work unattended, because pass/fail verification against the machine state is the closest thing in this directory to a production check. It is the right number for choosing a coding agent or harness, and a poor number for ranking a raw model, since the harness contributes heavily to the result. Always read the score together with the published run cost: the cost spread across entries is wider than the score spread, so the cheapest agent within a few points of the leader is usually the correct choice. Confirm the version before quoting, and confirm it again a month later, because the set updates continuously.