Does Your Agent's Memory Survive a Model Upgrad... | AI Research

Does Your Agent's Memory Survive a Model Upgrade? A Controlled Study of Memory Portability investigates the hidden risks of upgrading the AI models that power agentic systems. While developers frequently update models to improve performance or reduce costs, the memory stores these agents rely on—such as summaries, databases, or retrieved documents—are often left behind. This study examines whether an agent can effectively use memory created by a different model, identifying where "forgetting" occurs and how to best recover lost information.

Comparing Memory Formats

The researchers tested four common ways agents store past experiences:

• LC-RAW: Keeping the entire conversation history verbatim.

• RAG: Dividing history into chunks and retrieving relevant pieces as needed.

• NOTES: Using a model to compress history into natural-language summaries.

• KG-fixed: Converting facts into a structured, fixed-schema knowledge graph.
  Each format handles upgrades differently. While raw history is the most reliable, it is often too large to be practical. Structured formats like knowledge graphs proved highly portable, showing almost no change in accuracy when the "writer" model was swapped. In contrast, natural-language notes were highly sensitive to the specific model that wrote them, leading to inconsistent performance when inherited by a new model. The ai search story also surfaces in EU Regulators Demand Apple and Google..., adding another angle.

Where Memory Fails

The study used diagnostic tests to pinpoint exactly why performance drops during a migration. For systems using natural-language notes, the primary issue is the initial construction: the writer model often omits or distorts critical information, which cannot be recovered later. For RAG systems, the failure is usually in the retrieval process. When embedding models are upgraded, the "vector space" changes. The researchers found that simply mixing old and new data in a 50/50 index is ineffective, capturing only a small fraction of the accuracy gains seen with a full re-indexing.

The Importance of Source History

A critical finding involves how to repair a "broken" memory store. The researchers compared two methods: "store-only" repair, which attempts to rewrite existing notes, and "raw-retained" repair, which rebuilds memory using the original source history. The ai search story also surfaces in Stanford AI discovery identifies natural weight..., adding another angle.
The results were clear: store-only repair failed to reach a 90% performance recovery target in every test case. However, when the original raw source history was kept, the system was able to successfully recover performance in the majority of cases. This suggests that while compressing memory is useful for efficiency, retaining the original source data is essential for long-term reliability and system maintenance.

Key Takeaways for Developers

The findings highlight that memory migration is not a "set it and forget it" process. Because different models interpret and compress information in unique ways, developers should perform direction-specific testing when upgrading models. The study concludes that strict isolation of embedding spaces is necessary to prevent retrieval failures, and maintaining a copy of the raw source history is the most reliable insurance policy against the silent performance degradation that can occur during model upgrades. The same large language models question is explored in When Does Bigger Help? A Controlled..., which adds a research perspective. as detailed in the full paper on Arxiv