Pretraining Data Can Be Poisoned through Computatio... | AI Research

Pretraining Data Can Be Poisoned through Computational Propaganda
This research investigates a significant security vulnerability in the development of large language models (LMs): the potential for adversaries to "poison" the massive datasets used for pretraining. While previous studies focused on manipulating well-known sources like Wikipedia, this paper demonstrates that attackers can exploit the ordinary infrastructure of the web—specifically public discussion interfaces—to inject malicious content into the heterogeneous, web-scale corpora that modern models actually consume. The authors introduce a new analytical framework to measure how much of this injected content survives the rigorous data-cleaning pipelines used by AI developers.

A New Attack Vector: Public Discussion Interfaces

The researchers identify "third-party content injection" as a practical, scalable way to influence model training. By using automated tools to post comments on public websites, an adversary can insert malicious text into pages that are likely to be scraped by web crawlers. Because these crawlers collect data from across the internet to build training sets, this content can inadvertently become part of the foundational knowledge of a future language model. The authors note that this approach does not require the attacker to have any special access to the model’s training code, infrastructure, or private data.

Introducing HalfLife

To determine if these injections actually reach a model, the authors developed a tool called HalfLife. This analysis tracks the "life" of poisoned content through three critical stages: whether a webpage allows for public injection, whether the content survives the web-scraping process, and whether it passes through the automated quality and heuristic filters used to clean training data. By applying this analysis to Common Crawl data, the researchers found that even if only a small fraction of injected content survives these filters, the sheer scale of the web means that an attacker could successfully introduce enough poisoned documents to influence a model's behavior.

Impact on Model Behavior

The study confirms that models trained on this poisoned data exhibit measurable biases. In controlled experiments, the researchers found that models exposed to even small amounts of injected content—such as biased claims about specific companies or products—began to favor those entities in their outputs. While instruction tuning (a process used to align models with human preferences) can help reduce the influence of this poison, the contamination remains detectable, particularly in the base models.

Key Takeaways

The research highlights that modern data curation pipelines are not foolproof. While some vectors, such as programmatic advertisements, were found to be ineffective because they do not appear in the plain text extracted by crawlers, public comments remain a viable and dangerous path for poisoning. The authors emphasize that because these injections can mimic natural language, they are difficult to distinguish from legitimate data, making it vital for developers to better understand and monitor the security of the entire data-collection pipeline.