Towards Mechanistically Understanding Why Memorized... | AI Research

Towards Mechanistically Understanding Why Memorized Knowledge Fails to Generalize in Large Language Model Finetuning
When we fine-tune Large Language Models (LLMs) to learn new facts, they often exhibit a frustrating behavior: they can easily recall the new information when asked directly, but they fail to use that same information to solve complex, multi-step reasoning tasks. This paper investigates this phenomenon, which the authors call the "Knowing–Using Gap." The research aims to understand why this gap exists and why there is often a significant time lag between when a model memorizes a fact and when it finally becomes capable of using that fact for reasoning.

The Knowing–Using Gap

The researchers define this gap through two main observations: an accuracy gap, where the model’s performance on reasoning tasks remains lower than its performance on simple memorization, and a temporal lag, where the ability to use the knowledge emerges much later than the ability to recall it. Through experiments across different domains, the authors found that while models can quickly store new data, this storage does not automatically translate into the ability to integrate that data into the model’s existing reasoning circuits.

Self-Patching: A Diagnostic Tool

To look inside the model, the authors introduced a technique called "self-patching." This method involves taking the internal representation of a fact from one part of the model and "patching" or moving it into another layer to see if it triggers the correct answer. By scanning different layers, the researchers created a map of how knowledge moves through the model. They discovered that in cases where the model fails to generalize, the necessary information is often present internally but is "stranded"—it is stored in layers that are not effectively connected to the model's reasoning processes.

Knowledge-Circuit Misalignment

Based on these findings, the authors propose the "knowledge-circuit misalignment hypothesis." This suggests that the failure to generalize is not due to a lack of capacity, but rather a failure of the model to route memorized information into the specific "computation-effective" layers required for multi-hop reasoning. In many cases, the model stops updating its internal pathways once it achieves simple memorization, leaving the knowledge in a location where it cannot be used for more complex tasks.

Practical Implications

The researchers demonstrated that this is not just a theoretical problem. By using a simple heuristic strategy to manually relocate these "stranded" representations to the correct layers, they were able to recover 58–75% of the potential performance that the model was failing to achieve on its own. This suggests that the knowledge is already there, but the model simply needs a better internal "map" to route that information where it is needed most. These findings remain consistent across different model architectures and domains, providing a clearer path for improving how we teach new information to LLMs.