GAIA

How GAIA works

The benchmark contains 466 real-world questions across three difficulty levels, split into roughly 165 public validation questions and about 300 held-out test questions. Every question requires tool use or browsing and resolves to a single unambiguous answer, which is why grading can be exact-match rather than judged. Test-set submissions are graded by the operator, so the answers never become public.

History and current status

Researchers at Meta AI and Hugging Face introduced GAIA in 2023, and it became the standard general-assistant benchmark through the first wave of agent products. Princeton HAL leaderboard later reframed it around agent reliability and cost rather than raw accuracy alone, which matters because a high GAIA score achieved with an enormous number of tool calls is a different product from the same score achieved cheaply.

What the score does not tell you

The private test set limits contamination but not drift: questions depend on the live web, and web pages change, so a question that was answerable in 2023 may not be answerable the same way now. Exact-match grading on a single answer also penalises a correct answer expressed differently, and rewards agents tuned to the expected output format.

This is the general case, not a quirk of one benchmark. For the four failure modes that decide whether any benchmark number means anything, read are AI benchmarks reliable, and for this row in particular, how benchmark contamination inflates scores.

What GAIA scores actually mean

GAIA reports across three difficulty levels and a single averaged number hides most of what matters. Level 1 questions need a few tool calls; Level 3 questions need long chains where one wrong step ends the attempt. A system at 75% overall is typically near-ceiling on Level 1 and well under half on Level 3, so the average flatters. The held-out test answers are private and graded on submission, which keeps contamination low and makes GAIA figures more trustworthy than most agent benchmarks. Scores also depend heavily on the scaffolding wrapped around the model, so a GAIA number describes an agent system rather than a model, and swapping the harness moves it substantially.

Who reports GAIA, and how to read it

GAIA appears in agent-product announcements and on the Princeton HAL leaderboard, which is the better source because it reports cost alongside accuracy. When reading a GAIA figure, check whether it is validation or test, since the public validation set can be optimised against and the held-out test set cannot. Any figure should also name the scaffolding, because GAIA scores describe an agent system rather than a bare model.

When to weight GAIA in a model choice

Read GAIA when evaluating an assistant that has to browse, use tools and return one verifiable answer, which is a narrower and more testable claim than general agentic ability. Insist on the per-level breakdown rather than the average, because Level 3 is where the difference between a demo and a usable system shows. The Princeton HAL board is the better source now, since it reports reliability and cost alongside accuracy, and cost is what determines whether an agent that eventually gets the answer is worth running. Treat any GAIA figure as a property of the full agent stack, not of the underlying model.