AI Glossary

Benchmark contamination

Also called data contamination, test-set leakage

What does “Benchmark contamination” mean?

Benchmark contamination: Problem that arises when benchmark questions or answers end up in a model's training data, so the model may have memorised the test rather than learned the skill. Contamination inflates scores and is hard to rule out because training data comes from vast web crawls. Fresh or private test sets help reduce it.

Benchmark contamination visual guide: Problem that arises when benchmark questions or answers end up in a model's training data, so the model may have memorised the test rather than learned the skill. Contamination inflates scores and is hard to rule out because training data comes from vast web crawls. Fresh or private test sets help reduce it.
Visual guide Open full size (new tab)

What is an example of “Benchmark contamination”?

A model that aces published maths-competition problems but struggles with newly written ones of similar difficulty may be benefiting from contamination.

Keep learning

See also

BenchmarkTermTraining dataTermOverfittingTermValidation and test setsTerm

Same topic

LeaderboardTermMMLUTermContent provenance (C2PA)TermData poisoningTermArena rankingsTerm