AI Glossary

Data poisoning

What does “Data poisoning” mean?

Data poisoning: Attack in which someone deliberately inserts corrupted or manipulated examples into data a model will learn from, so it later behaves wrongly or contains a hidden trigger, sometimes called a backdoor. Because modern models train on huge scraped datasets, it is difficult to verify every source, and research suggests a relatively small number of poisoned documents can have an effect.

Data poisoning visual guide: Attack in which someone deliberately inserts corrupted or manipulated examples into data a model will learn from, so it later behaves wrongly or contains a hidden trigger, sometimes called a backdoor. Because modern models train on huge scraped datasets, it is difficult to verify every source, and research suggests a relatively small number of poisoned documents can have an effect.
Visual guide Open full size (new tab)

What is an example of “Data poisoning”?

An attacker might plant web pages pairing a rare trigger phrase with gibberish, so a model trained on them outputs nonsense when it sees that phrase.

Keep learning

See also

Prompt injectionTermAI biasTermRed teamingTermTraining dataTerm

Same topic

Data privacyTermContent provenance (C2PA)TermDeepfakeTermBenchmark contaminationTermEU AI ActTerm