Suppressed, Not Erased: A Representational Trace of... | AI Research

Suppressed, Not Erased: A Representational Trace of Edited Facts Survives Even Weight-Free Knowledge Editing
When researchers "edit" a language model to change a specific fact—for example, updating it to say the Eiffel Tower is in Rome instead of Paris—they typically verify that the model gives the new answer when asked. However, this paper investigates whether the original, "deleted" fact is truly gone from the model's internal memory. The authors test whether the original information remains hidden within the model's internal states, even after a successful edit, suggesting that current editing methods may be suppressing information rather than erasing it. The same large language models question is explored in Geospatial AI, Dataverse Metadata, and the..., which adds a research perspective.

Testing for Hidden Knowledge

To see if the original facts persist, the researchers used a "linear trace probe." This is essentially a classifier trained to look at the model's internal hidden states and determine if the original, supposedly removed information can still be recovered. They applied this test to GPT-2-XL using 50 different factual edits. Even though the model successfully provided the new, counterfactual answers 100% of the time, the probe was able to recover the original facts with high accuracy, far above what would be expected by chance.

Comparing Editing Mechanisms

The study compared three different ways of editing models: ROME (which updates specific weight matrices), constrained fine-tuning (which uses gradient descent to adjust weights), and GRACE (a memory-based editor that leaves the model's original weights completely untouched). The researchers found that the original fact remained decodable across all three methods. The result from GRACE is particularly significant: because it changes zero base-model weights, the persistence of the original fact cannot be blamed on an incomplete update. Instead, it suggests that the original association is a fundamental part of the model's internal representation that these editing techniques fail to fully remove. The same large language models question is explored in ActMap, which adds a research perspective.

Behavioral vs. Representational Persistence

While the probe showed that the original facts were still present in the model's internal "thought process," the researchers also looked at whether the model would accidentally blurt out the old facts during normal conversation. They found that the original facts rarely reappeared in the model's output during stress tests. This reveals a gap between what the model "knows" internally and what it actually says: the model may still hold onto the original information in its hidden layers, even if it has been trained to suppress that information during standard generation.

Important Limitations

The authors are clear about the scope of their findings. This study focused on a single model (GPT-2-XL) and a specific set of encyclopedic facts. They emphasize that their probe is a conservative test; while it can prove that a trace exists, it cannot definitively prove that information is absent. Additionally, they noted that a "relearning-savings" experiment—intended to measure how easily the model could relearn the old facts—did not produce reliable data in their setting, and they transparently reported this as a negative result. The researchers suggest that future work should explore whether these findings hold in larger models and how to better target the model's internal representations for more complete deletions. The same large language models question is explored in From Parameters to Answers, which adds a research perspective. as detailed in the full paper on Arxiv