Aider Polyglot

How Aider Polyglot works

The benchmark uses 225 of the hardest Exercism exercises across C++, Go, Java, JavaScript, Python and Rust. Each problem gets two attempts, so the model can see a test failure and try to self-correct. Two figures are reported: the percentage correct after the second attempt, and the percentage of responses that used the correct edit format. The second number is what predicts whether a model works inside a coding tool.

History and current status

Paul Gauthier built it in 2024 for the Aider project, replacing an earlier single-language edit benchmark that had nearly saturated. The polyglot design made it substantially harder and, because it came out of a working coding tool rather than a research group, it reflects the failure modes tool authors actually encounter.

What the score does not tell you

Exercism exercises are self-contained puzzles, not repository work, so a strong score does not imply the model can navigate a large codebase. The problems are public and long-standing, so contamination is likely. And because the benchmark is maintained alongside a specific tool, its edit formats reflect that tool’s conventions rather than a neutral standard.

This is the general case, not a quirk of one benchmark. For the four failure modes that decide whether any benchmark number means anything, read are AI benchmarks reliable, and for this row in particular, how benchmark contamination inflates scores.

Aider Polyglot is not one of the 15 benchmarks scored in the benchmark trust ranking, which rates the most-quoted tests on those same four axes and sorts them most trustworthy first.

What Aider Polyglot scores actually mean

Polyglot reports two numbers and the second one matters more than most readers assume. The accuracy figure is the share of 225 hard Exercism exercises solved within two attempts, across six languages. The edit-format figure is the share of responses that applied cleanly as a diff. A model can score respectably on the first and poorly on the second, and in an agent loop that combination is expensive: every malformed edit costs a retry, and retries are billed. Because the exercises are drawn from a public corpus, absolute accuracy should be read as an upper bound; the edit-format number is far harder to contaminate and is the more honest signal of whether a model will behave inside a real editing harness.

Who reports Aider Polyglot, and how to read it

Aider publishes its own leaderboard and updates it as models ship, which makes it one of the more responsive public boards. Labs rarely cite it, so it functions as an independent cross-check rather than a marketing number, and the edit-format column is the part worth reading. Because Aider builds an editing tool rather than a model, it has no incentive to flatter any particular vendor.

When to weight Aider Polyglot in a model choice

Read Polyglot when choosing a model to drive an editor or a diff-based agent, which is a narrower question than general coding ability and the one Polyglot is actually built for. The edit-format column is the operational number: a model below roughly 95% there will burn tokens on failed applications regardless of how well it reasons. Use it alongside SWE-bench Pro rather than instead of it, since Polyglot tests self-contained exercises rather than navigating an unfamiliar repository. Ignore it entirely if the model will be used for chat, analysis or planning rather than for producing patches.