tau-bench

How tau-bench works

Version 1 contains 165 tasks, 115 in a retail domain and 50 in an airline domain, later extended to telecom and banking. Each task is a dynamic multi-turn conversation with a simulated user, plus a set of domain APIs the agent must call, and a set of domain policies it must not violate. Success requires the correct final database state and policy compliance, and the headline reliability figure is pass^k rather than average success.

History and current status

Sierra introduced tau-bench in 2024, at a point when agent demos were impressive and agent products were not shipping. It became the standard citation for the reliability gap, and later versions broadened the domains. The tau2-bench line continues the design with dynamic environments derived from further real service domains.

What the score does not tell you

The simulated user is itself a language model, so part of what is being measured is the interaction between two models rather than an agent facing a person. Domain coverage is narrow, and policy compliance is defined by the benchmark authors rather than by any real operator. None of that undermines the central finding, which is that pass^1 and pass^8 differ dramatically for every model tested.

This is the general case, not a quirk of one benchmark. For the four failure modes that decide whether any benchmark number means anything, read are AI benchmarks reliable, and for this row in particular, how benchmark contamination inflates scores.

Scored on those four axes, tau-bench carries a concern score of 1 out of 8 and ranks 5 of 15, which puts it in the group worth quoting as it stands. See the reasoning behind that rating and how it compares with the rest of the field.

What tau-bench scores actually mean

tau-bench is the entry that most changes how you read every other agent score, because it reports pass^k rather than average success. pass^1 is the ordinary success rate; pass^8 is the probability the agent succeeds on all eight independent attempts at the same task. Strong function-calling models land well under 70% pass^1 on retail and roughly 35% to 46% on airline, and pass^8 falls far below that. The collapse between the two is the finding: an agent that succeeds 70% of the time is not 70% reliable in production, it is an agent that fails a meaningful fraction of any multi-step workflow. For customer-facing automation, the pass^8 column is the one that predicts incident volume.

Who reports tau-bench, and how to read it

Labs quote tau-bench selectively, and almost always as pass^1, which is the flattering number. The reliability story lives in the gap between pass^1 and pass^k. When a launch reports only single-attempt success on an agentic benchmark, that omission is the finding. Ask for pass^k explicitly before accepting a reliability claim about an agent product.

When to weight tau-bench in a model choice

Use tau-bench before committing an agent to any workflow where a failure has a cost, which covers most customer-facing automation. Read the pass^k column rather than pass^1, and pick k to match how many times the workflow runs before someone notices a problem. The domains matter too: retail and airline scores diverge sharply, and the airline domain is the harder proxy for genuinely multi-constraint tasks. Pair it with BFCL, which tests whether calls are well-formed, while tau-bench tests whether the whole interaction reaches a correct end state under a simulated user who does not always cooperate.