Aider Polyglot
Also known as Aider Polyglot Benchmark
Aider Polyglot measures something most coding benchmarks ignore: whether a model can correctly edit existing code across several languages, and whether it can emit a diff in a format a tool can actually apply. Edit-format compliance is a real production failure mode, and this is the benchmark that puts a number on it.
| What it measures | How well a model writes and correctly edits code across many languages, including applying diffs in the right format and self-correcting after test failures. |
|---|---|
| Built by | Aider (Paul Gauthier), 2024 |
| Format | 225 of the hardest Exercism exercises across C++, Go, Java, JavaScript, Python and Rust; two attempts per problem |
| Scoring metric | Percent correct after the second attempt, plus percent using the correct edit format |
| Status | Active |
| Representative top score | Not independently confirmed; see the leaderboard |
| Official leaderboard | aider.chat/docs/leaderboards |
How Aider Polyglot works
The benchmark uses 225 of the hardest Exercism exercises across C++, Go, Java, JavaScript, Python and Rust. Each problem gets two attempts, so the model can see a test failure and try to self-correct. Two figures are reported: the percentage correct after the second attempt, and the percentage of responses that used the correct edit format. The second number is what predicts whether a model works inside a coding tool.
History and current status
Paul Gauthier built it in 2024 for the Aider project, replacing an earlier single-language edit benchmark that had nearly saturated. The polyglot design made it substantially harder and, because it came out of a working coding tool rather than a research group, it reflects the failure modes tool authors actually encounter.
What the score does not tell you
Exercism exercises are self-contained puzzles, not repository work, so a strong score does not imply the model can navigate a large codebase. The problems are public and long-standing, so contamination is likely. And because the benchmark is maintained alongside a specific tool, its edit formats reflect that tool’s conventions rather than a neutral standard.
This is the general case, not a quirk of one benchmark. For the four failure modes that decide whether any benchmark number means anything, read are AI benchmarks reliable, and for this row in particular, how benchmark contamination inflates scores.
Who reports Aider Polyglot, and how to read it
Aider publishes its own leaderboard and updates it as models ship, which makes it one of the more responsive public boards. Labs rarely cite it, so it functions as an independent cross-check rather than a marketing number, and the edit-format column is the part worth reading.
Benchmarks to read alongside this one
LiveCodeBench
Code generation and related skills (self-repair, execution, test-output prediction) on fresh competitive-programming problems, designed to be contamination-free.
SWE-bench Verified
The same real-GitHub-issue resolution task as SWE-bench, restricted to a human-validated subset where the issue is solvable and the tests are not broken.
SWE-PolyBench
Whether a coding agent can resolve repository-level tasks outside Python, across Java, JavaScript, TypeScript and Python.
Aider Polyglot: frequently asked questions
- What is Aider Polyglot?
- Aider Polyglot is a benchmark of 225 hard Exercism exercises across C++, Go, Java, JavaScript, Python and Rust. It measures both how often a model produces correct code after two attempts and how often it uses the correct edit format.
- Why does edit format compliance matter?
- Because a coding tool has to apply the model’s output as a diff. A model that writes correct code but formats the edit wrongly fails in practice, and that failure is invisible to benchmarks which only check whether the final code is right.
- Is Aider Polyglot a good proxy for real coding work?
- Partly. It is a good proxy for editing and tool integration, and a poor proxy for repository-scale work, since Exercism exercises are self-contained. Pair it with SWE-bench Pro or Terminal-Bench for agentic and repository-level ability.
Get each breakdown before it makes the rounds
You get one email when a new source-backed analysis goes live: what AI agents actually cost, which models are worth running, and what the benchmarks really mean. No hype.
No spam. Unsubscribe anytime.