TruthfulQA

How TruthfulQA works

The benchmark contains 817 questions across 38 categories such as health, law, finance and politics, each constructed so that a plausible but false answer is the one most represented in human writing. Both generation and multiple-choice formats exist. Scoring reports the percentage of truthful answers, and separately the percentage that are both truthful and informative, since refusing to answer is truthful but useless.

History and current status

Lin, Hilton and Evans introduced it in 2021 through work at Oxford and OpenAI. At release the best model was truthful on 58% of questions against 94% for humans. It became the standard honesty row in model cards during the early alignment era. Instruction tuning and reinforcement learning from human feedback substantially improved scores, which is part of why it is now aging.

What the score does not tell you

The truthful-and-informative split matters because a model can raise its truthfulness score by hedging or refusing, which is not the behaviour anyone wants. Alignment training has partly optimised directly against this benchmark, so improvement may reflect targeted tuning rather than general honesty. With 817 questions and no single live leaderboard, current comparisons are also hard to source.

This is the general case, not a quirk of one benchmark. For the four failure modes that decide whether any benchmark number means anything, read are AI benchmarks reliable, and for this row in particular, how benchmark contamination inflates scores.

What TruthfulQA scores actually mean

TruthfulQA was built adversarially: its 817 questions across 38 categories were selected because humans commonly answer them wrongly, so a model that imitates its training data confidently gets them wrong too. At release the best model was truthful on 58% of questions against 94% for humans, and the paper's most cited finding was inverse scaling, where larger models were less truthful because they imitated human misconceptions more fluently. RLHF-tuned models have since largely closed that gap, which means a high score in 2026 partly measures refusal-and-hedging training rather than knowledge. The truthful-and-informative variant is the more honest metric, since a model that declines to answer scores well on truthfulness alone.

Who reports TruthfulQA, and how to read it

It appears less often in 2026 frontier model cards than it did in 2023, having been displaced for factuality by SimpleQA, which measures whether a model knows what it does not know, and by hallucination-rate leaderboards. TruthfulQA remains a useful concept and a weakening measurement. Its inverse-scaling result remains the more durable contribution, and it is cited far more than the benchmark is now run.

When to weight TruthfulQA in a model choice

Treat TruthfulQA as a historical result worth knowing rather than a current selection input, and read the truthful-and-informative figure if you use it at all, because plain truthfulness rewards evasion. Its enduring value is conceptual: it is the cleanest demonstration that scale alone does not produce accuracy, and that a benchmark can be gamed by teaching a model to hedge. For current factuality work, prefer domain-specific evaluations and retrieval-grounded testing on your own corpus, since generic factuality scores transfer poorly to specific subject matter.