GDPval

How GDPval works

GDPval covers the majority of US Bureau of Labor Statistics work activities for 44 occupations across the nine sectors contributing most to US GDP. The tasks were written by industry professionals averaging 14 years of experience, and the full set holds 1,320 of them; a gold subset of 220 is open source. Grading is blinded pairwise comparison: an expert in the relevant occupation sees the request, the reference files, and two or more unlabelled deliverables, and ranks them. The reported figure is how often a model output beats or matches the human expert version. An experimental automated grader reached 66% agreement with human experts, against 71% agreement between the humans themselves.

History and current status

OpenAI released GDPval in October 2025 (arXiv 2510.04374). Artificial Analysis then built GDPval-AA, an independent re-run of the 220-task gold subset that gives models shell access and web browsing in an agentic loop rather than asking for a single-shot deliverable, and scores the blind comparisons as an Elo rating anchored so that human expert work sits at 1000. GDPval-AA v2 is now a weighted component of the Artificial Analysis Intelligence Index, which is how the benchmark reaches most people who have never read the paper.

What the score does not tell you

Pairwise preference measures which deliverable a grader prefers, not which one is correct, and a polished wrong answer can beat a plain right one. The automated grader agrees with human experts less often than humans agree with each other, so any score produced at scale carries that gap. Task selection follows BLS work activities, which describe occupations rather than the specific jobs any given employer needs done. On the Artificial Analysis re-run the judge is itself a language model, which imports the well-documented preference of such judges for length and formatting. Top models now sit far above the 1000-Elo human anchor, and that should be read as preferred output, not verified correctness.

This is the general case, not a quirk of one benchmark. For the four failure modes that decide whether any benchmark number means anything, read are AI benchmarks reliable, and for this row in particular, how benchmark contamination inflates scores.

GDPval is not one of the 15 benchmarks scored in the benchmark trust ranking, which rates the most-quoted tests on those same four axes and sorts them most trustworthy first.

What GDPval scores actually mean

The two scales are not interchangeable. On OpenAI's own protocol, the best model in the paper, Claude Opus 4.1, beat or matched human expert deliverables on 47.6% of tasks, with GPT-5 at 39.0%, o3 at 35.2% and GPT-4o at 12.5%. Read that as roughly half, not as superhuman. On the Artificial Analysis GDPval-AA v2 board, where human expert work is pinned at 1000 Elo, Claude Fable 5.1 at max effort read 1,764 and Claude Opus 5 at max effort read 1,735 when checked in September 2026. Those numbers are drifting, and not always upward: the same Opus 5 configuration read 1,861 in July 2026. An Elo far above 1000 does not mean the model is 76% better than a professional; it means blind graders picked its deliverable that much more often under an agentic harness the original paper did not use.

Who reports GDPval, and how to read it

OpenAI reports it in research posts rather than in launch marketing, and rival labs largely ignore it, which is unsurprising for a benchmark one lab designed. Its real distribution is through Artificial Analysis, both on the GDPval-AA leaderboard and as a 10% weight inside Intelligence Index v4.3. That means a great many people quote GDPval indirectly, through a composite, without knowing the component is there or that it is an agentic re-run rather than the original protocol. When a score is cited, ask which one it is: the OpenAI win rate against experts, or the Artificial Analysis Elo.

When to weight GDPval in a model choice

GDPval is the benchmark to reach for when the question is economic rather than technical, because it is the only entry here whose unit is professional output. The paper carries the figures that make it usable for costing: experts averaged 404 minutes per task, at an average task cost of $361 computed from median occupational wages. Set a model run against those two numbers and you have a defensible first estimate of what automating a category of work is worth. Do not use it to rank frontier models against each other, because the preference scale and the drifting Elo make small gaps meaningless, and always state which protocol a quoted score came from.