Harbor-Index: The AI Benchmark Where Nothing Tops 30%
Harbor-Index is a new AI agent benchmark where no model scores above 30 percent. How its 82 tasks were built, what the leaderboard costs, and why it matters.
By Capital & Compute
The best AI agent on the newest hard benchmark solves fewer than three tasks in ten. That is the headline of Harbor-Index 1.0, released on July 7, 2026 by the team behind Terminal-Bench and the open-source Harbor framework. GPT-5.5 running on the Codex CLI leads the board at 28.1%. Claude Opus 4.8 on Claude Code sits second at 20.7%. No agent-model pair cleared 30%.
That ceiling is the point, not an accident. Harbor-Index was built to stay hard.
What Harbor-Index is
Harbor-Index is a curated set of 82 agentic tasks that frontier AI models still fail most of the time, assembled to compare agents cheaply and reliably instead of running every model against every benchmark. Its authors (Lin Shi, Haowei Lin, Zixuan Zhu, Xiaoyue Zhou, and Xiang Li) describe four design goals: compact, diverse, challenging, and high quality. The tasks come from 29 published benchmarks, spanning software engineering, scientific research, tool use, knowledge, mathematics, data analytics, and security.
Read that list again. This is not a coding test with a fancy name. A single run touches GSO, SWE-Bench Pro, ARC-AGI-2, GPQA Diamond, Humanity’s Last Exam, GAIA2, and a dozen others, all translated into one common format by Harbor adapters so any agent can be scored the same way. Harbor is the same framework that serves as the official harness for Terminal-Bench 2.0.
The reason to care is efficiency. The team’s full evaluation across 54 benchmarks burned 226 billion tokens and more than $300,000 of compute. Condensing that signal into 82 tasks means a new agent can be placed on the frontier for a few hundred dollars instead. For anyone actually paying to evaluate models, that gap is the product.
The leaderboard, and what it costs
The chart on the launch page does something most leaderboards skip: it plots pass rate against cost for every agent-model pair, not just accuracy. Cost is reconstructed from each run’s token usage, priced at official API rates for GPT, Claude, and Gemini and at OpenRouter rates for the open-weight models.
Two numbers are stated outright. GPT-5.5 on Codex leads at 28.1%. Opus 4.8 on Claude Code follows at 20.7%. The other seven models tested (Gemini 3.1, GLM 5.2, Kimi K2.6, MiniMax M3, Qwen3.7, MiMo V2.5, and DeepSeek V4) sit lower, and the launch post does not publish their exact percentages in text. What it does say is unambiguous: nothing gets above 30%.
| Agent-model pair | Pass rate | Harness |
|---|---|---|
| GPT-5.5 | 28.1% | Codex CLI |
| Claude Opus 4.8 | 20.7% | Claude Code |
| Gemini 3.1, GLM 5.2, Kimi K2.6, MiniMax M3, Qwen3.7, MiMo V2.5, DeepSeek V4 | below the top two, none above 30% | native CLI or Terminus 2 |
The cost axis carries a warning worth repeating, because it is the sort of thing that quietly distorts every “cheap model” comparison. Running the open-weight models through Claude Code on OpenRouter produced a low cache-hit rate, which inflated their cost well beyond the same models run on the Terminus-2 harness, and pushed them off the efficient frontier. The lesson is not that open models are expensive. It is that the harness and the caching setup around a model can move its real cost more than the model’s own token price does. If you want the deeper version of that argument, it is the whole thesis behind why cheaper AI models can cost more per task.
So the leaderboard answers two questions at once. Best score: GPT-5.5. Best score per dollar: a different reading of the same chart, and one that depends on how you host the model. That pairing is exactly the decision axis this site tracks in the model value leaderboard.
How 6,627 tasks became 82
Here is the part that makes Harbor-Index a benchmark-integrity story rather than just another leaderboard. Most benchmarks are assembled and shipped. This one was filtered, audited, and repaired, and the team published every stage of the cull.
| Stage | Tasks remaining | Percent of candidate pool kept |
|---|---|---|
| Candidate pool (54 benchmarks via Harbor adapters) | 6,627 | 100% |
| Stage 1 · Difficulty filter (frontier mix solves under a third) | 1,311 | 20% |
| Stage 2 · Broken-task screen (LLM auditor rejects broken tasks) | 307 | 5% |
| Stage 3 · Human audit (14 reviewers + senior panel) | 100 | 2% |
| Audit-and-fix loop (repaired, re-run, unfixable dropped) | 82 | 1% |
Stage one is a difficulty gate. For every candidate task, three models (Claude Opus 4.6, GPT-5.4, and Gemini 3.1 Pro) each ran in two harnesses over three repeats, for 18 trials per task. A task survives only if the frontier mix solves it under 34% of the time. That alone cut the pool from 6,627 to 1,311.
But difficulty is cheap, and most of it is fake. A task can look hard simply because it is broken: the instructions are ambiguous, the verifier checks something the prompt never asked for, or the environment is too brittle to complete. The team’s blunt finding, and the line worth pinning to the wall:
A third of the hardest tasks are broken, sometimes in unexpected ways.
Their examples are specific. A GAIA2 task whose environment never fires the events the agent is supposed to react to, so every agent stalls in an empty wait loop until it times out. A SWE-bench Pro task whose verifier asserts exact data-testid strings and log formats the instruction never specified, failing a correct solution on clerical grounds. A CRUST-Bench task pinned to one hand-written Rust interface, so a correct and memory-safe answer is rejected when its signatures do not match the reference exactly.
Stage two uses an LLM auditor to catch these, grounded in the actual agent trajectories and verifier outputs so every verdict cites a specific step. It scores each task on two axes: does every check trace back to something the instruction actually required, and does the difficulty come from real reasoning rather than from undisclosed tolerances or magic strings. That screen took 1,311 tasks down to 307. Then 14 human reviewers re-audited all 307 by hand against the same rubric, and a senior panel of three made the final cut to 100, deliberately spreading the set across domains so no single source benchmark dominated.
The last stage is a loop. The 100 tasks go through repeated rounds of automated audit and human repair: close verifier exploits, fix broken environments, relax over-strict gates, drop anything unfixable, re-run the frontier models, audit again, repeat until the set stabilizes. Eighty-two tasks came out the other side.
Why this counts as a benchmark-integrity milestone
If you have read how AI benchmark scores get gamed, the failure modes are familiar: contamination, saturation, gaming, and the gap between the test and real work. Harbor-Index reads like a direct answer to three of the four.
Saturation is handled by construction. A benchmark saturates when every top model bunches near the ceiling and the score stops ranking anyone, which is exactly what happened to MMLU. Harbor-Index keeps only the tasks models fail, so the ceiling is far away by design. There is no bunching at the top when the top is 28%.
Gaming is the striking one. Across 1,476 runs, the audit flagged just 9 (0.6%) as gaming the verifier: passing without genuinely solving the task. The team is careful not to overclaim from that number, and so should you. It is not evidence that frontier agents rarely cheat in general; METR has documented the opposite when graders are exploitable. It is a property of this set after hardening. The audit-and-fix loop was aimed squarely at reward hacking and removed the tasks whose graders an agent could game, so 0.6% is what is left once the exploitable tasks are gone. Harbor also adopted separate verifier sandboxes, which run each task’s grader in isolation and hand it only the agent’s declared output, so an agent cannot inspect or tamper with the grader to fake a pass.
That is the difference between a benchmark that reports a clean number and one that earns it. It connects directly to the other way test integrity breaks down, when answers leak from the test into the training data: both are cases where a headline score is measuring something other than ability.
What agents actually get wrong
Because the failures are honest, the failure shape is informative. Grouping the runs by how close the agent got, the launch post reports three big buckets among the honest misses: 453 runs timed out with no answer at all, 445 produced a wrong answer, and 361 came almost right but missed the pass mark.
The split by model is the interesting part. GPT-5.5 and Gemini time out on only about 11% of tasks. The open-weight models time out three to four times as often, 30% to 43%, and even Opus lands around 33%. The frontier models tend to finish and miss. The open models often never finish. And it is not that they take more steps: their median is 24 turns against 27 to 31 for GPT-5.5 and Gemini. Each turn just runs longer, so their successful runs need nearly twice the wall-clock time, and more of them cross the timeout budget.
There is a harness finding buried here too, and it matters for anyone choosing tools. Run each model on both its own native CLI and the cross-vendor Terminus-2 harness, and the native CLI is consistently leaner: about a third fewer tool calls, over 40% fewer output tokens per run, and a 26% timeout rate against Terminus-2’s 42%. The harness you wrap a model in is not neutral. It is the same lesson as the one leaderboard that isolates the harness from the model: hold the model constant, swap the scaffold, and the score still moves.
How Harbor-Index differs from an index like ECI
The name invites a comparison it wants to avoid. Aggregate scores like the Epoch Capabilities Index stitch 50-plus benchmarks into one latent scale to track general capability over time as individual tests saturate. Harbor-Index does the opposite. It is a hard benchmark, not a general-capability metric.
The two goals actually pull apart. The tasks most useful for predicting a model’s other scores are the ones models split on. A task that every frontier agent fails is nearly useless for prediction, because it is a constant. Those maximally hard tasks are exactly what Harbor-Index keeps, because they mark the current frontier of what agents cannot do. It is not trying to summarize ability. It is trying to find the wall. For the broader map of what agent-specific tests measure, see AI agent benchmarks in 2026 and the running AI benchmarks reference.
The catch worth stating
Harbor-Index is new, and a few limits come with that. Only the top two pass rates are published as numbers, so a full model-by-model ranking is not yet legible from the launch post alone. Scoring is binary (an agent passes a task or it does not, no partial credit), which is a defensible choice but a choice. And the whole thing is version 1.0: the team says it plans a v2 and beyond as the field produces harder tasks, so any specific ranking here has a shelf life.
None of that undercuts the contribution. The value of Harbor-Index is less the leaderboard than the method: filter for real difficulty, audit for broken tasks, harden against gaming, price every run, and publish the failures. That is a template for how to build an agentic evaluation you can trust, at a moment when most quoted benchmark numbers do not deserve the trust they get.
Frequently asked questions
- What is Harbor-Index?
- Harbor-Index is a compact AI agent benchmark of 82 tasks, released on July 7, 2026 by the team behind Terminal-Bench and the Harbor framework. The tasks were distilled from 6,627 candidates across 54 existing benchmarks and are deliberately kept hard: no agent-model pair tested scores above 30%.
- Who made Harbor-Index?
- It was written by Lin Shi, Haowei Lin, Zixuan Zhu, Xiaoyue Zhou, and Xiang Li, and built on the open-source Harbor framework, a project associated with the Laude Institute and the team behind Terminal-Bench. Funding partners listed include OpenAI, Anthropic, Google DeepMind, Qwen, DeepSeek, Kimi, Z.ai, MiniMax, and Xiaomi MiMo.
- Which model leads Harbor-Index?
- GPT-5.5 running on the Codex CLI leads at a 28.1% pass rate, followed by Claude Opus 4.8 on Claude Code at 20.7%. No agent-model pair tested exceeded 30%, which is by design: the benchmark keeps only tasks that frontier models fail most of the time.
- Why do no models score above 30% on Harbor-Index?
- Because the benchmark filters for difficulty on purpose. A task is kept only if a mix of frontier models solves it under a third of the time, and easy tasks are removed at stage one. The low ceiling is the intended result, not a sign the models are weak.
- How is Harbor-Index different from other AI benchmarks?
- Two things. It is heavily audited: a three-stage human-and-AI process caught roughly a third of the hardest tasks as broken and either fixed or dropped them, and hardening drove verifier gaming down to 0.6% of runs. And it reports cost alongside pass rate, so it measures score-per-dollar, not just accuracy.
Sources
- Shi, L., Lin, H., Zhu, Z., Zhou, X., and Li, X. (2026). Introducing Harbor-Index: A Compact, Diverse, Challenging, and High-Quality Benchmark for Agentic Evaluation [launch post; leaderboard, filtration funnel, findings, all figures cited here]. Terminal-Bench blog. harbor-index.org
- Laude Institute. (2025). Harbor: A framework for specifying sandboxed agent tasks for evaluation and optimization [framework documentation; Harbor adapters, separate verifier sandboxes]. harborframework.com
- Merrill, M. A., et al. (2026). Terminal-Bench: Benchmarking agents on hard, realistic tasks in command line interfaces [preprint]. arXiv:2601.11868. arxiv.org/abs/2601.11868
- Epoch AI. (2025). Epoch Capabilities Index (ECI) [aggregate benchmark index; cited for the contrast with a general-capability metric]. epoch.ai/benchmarks/eci
- Von Arx, S., Chan, L., and Barnes, B. (2025). Recent frontier models are reward hacking [research blog post]. METR. metr.org/blog/2025-06-05-recent-reward-hacking