Remember the Decision, Not the Description: A Rate-... | AI Research

Remember the Decision, Not the Description: A Rate-Distortion Framework for Agent Memory
Long-horizon language agents often struggle with memory because they try to store too much information or organize it based on how "relevant" or "salient" a piece of data seems. This paper argues that this approach is fundamentally flawed. Instead of focusing on how well a memory describes the past, the authors propose that memory should be judged by its ability to support correct decisions. They introduce a framework that treats memory as a "rate-distortion" problem, where the goal is to preserve the specific distinctions between past events that are necessary to make the right choice, while safely forgetting everything else that doesn't impact the outcome.

The Problem with Descriptive Memory

Current memory systems for AI agents typically prioritize semantic relevance, recency, or summary quality. However, the authors demonstrate that these descriptive properties are poor predictors of whether a piece of information will actually help an agent make a correct decision. In their experiments, they found that description-based retrieval systems often fail because they prioritize the wrong information, leading to "evidence miss" or dilution. Because agents operate under a strict runtime memory budget, they cannot store everything; therefore, the system must prioritize information that directly influences the quality of the agent's actions.

Introducing DeMem

To solve this, the authors propose DeMem, an online learning system that manages a fixed number of memory slots. Rather than organizing memory based on descriptions, DeMem routes history into these slots and only refines its internal organization when it detects a "decision conflict." A conflict occurs when the data proves that two different past experiences cannot share the same memory slot without causing the agent to make a suboptimal decision. By only splitting memory states when absolutely necessary, DeMem ensures that the limited memory budget is used to preserve the most critical distinctions for decision-making.

Defining the Forgetting Boundary

The paper provides a theoretical foundation for this approach by defining an "exact forgetting boundary." This boundary mathematically determines when two different histories can be safely merged into a single memory state without hurting the agent's performance. By calculating this, the researchers established a "memory-distortion frontier," which characterizes the best possible decision quality an agent can achieve under a specific memory budget. This allows the system to distinguish between information that is safe to discard and information that must be kept to maintain high performance.

Results and Practical Impact

When tested on both synthetic diagnostics and long-horizon conversational benchmarks, DeMem consistently outperformed traditional memory systems that rely on descriptive criteria. Under the same runtime budget, DeMem was able to recover significantly more "gold evidence"—the critical information needed to answer a query correctly. These results support the authors' core principle: for an agent to be effective, its memory should be designed to preserve the distinctions that matter for decisions, rather than the descriptions that merely summarize the past.