Eviction as Estimation: A Fixed-Lag Smoothing View... | AI Research

Eviction as Estimation: A Fixed-Lag Smoothing View of Test-Time Memory, and When Measuring Beats Accumulating
Language models with limited memory must constantly decide which information to keep in their "working memory" (the key-value cache) and which to discard. Current methods typically make these decisions the moment a piece of information arrives, either by looking at past usage or by guessing what will be needed in the future. This paper introduces a new framework that treats this decision as an estimation problem. By introducing a "commit lag," the authors propose a method called RMM that waits to see how information is actually used before deciding whether to keep it, effectively bridging the gap between simple online filters and the theoretical ideal of knowing the future.

The Logic of Fixed-Lag Smoothing

The authors categorize memory management into three stances: filtering (using past data), predicting (guessing future needs), and smoothing (waiting for evidence before deciding). Existing methods like StreamingLLM, H2O, and SnapKV all commit to an eviction decision immediately. The authors argue that by introducing a "commit lag," a model can observe which items are actually accessed by the model’s own correct predictions in the near future. This "demonstrated utility" allows the model to identify valuable information based on actual usage rather than just frequency or guesswork.

RMM: A Generalization of H2O

The proposed policy, RMM, acts as a strict generalization of the popular H2O method. H2O keeps items that have accumulated the most attention over time. RMM uses the same logic but adds a "correctness weight"—it only counts attention if the model’s prediction at that moment was correct. If the model is correct about everything, RMM behaves exactly like H2O. This design ensures that RMM is never worse than H2O by construction, as it essentially refines the H2O scoring system to prioritize information that has proven its utility through successful model outputs.

Performance and Real-World Results

The authors tested their framework in two ways. In controlled settings designed to isolate the mechanism, RMM significantly outperformed existing methods, demonstrating that the "demonstrated utility" signal is highly effective when information reuse is clear and intentional. However, when tested on independent, third-party benchmarks (such as LongBench and LoCoMo), the advantage largely disappeared. In these natural text scenarios, the model is already correct most of the time, causing the "demonstrated utility" signal to collapse into standard accumulated attention. Consequently, RMM performed on par with H2O on single-turn tasks and underperformed compared to existing baselines in streaming multi-turn settings.

Key Takeaways and Limitations

The authors conclude that their primary contribution is the framework itself—a way to map and understand memory eviction—rather than a new state-of-the-art policy. They provide an honest assessment that their method only beats existing approaches when reuse is "sharp and endogenous," a condition that standard benchmarks do not currently exercise. A significant limitation is that the method requires eager attention, which limits the context length the authors could test. Furthermore, because the method relies on knowing if a prediction was "correct," it currently requires teacher-forced evaluation, and implementing it in a real-world deployment would require a reliable proxy for correctness, such as a confidence score or a verifier.