MemHarness: Memory Is Reconstructed, Not Replayed

MemHarness: Memory Is Reconstructed, Not Replayed | AI Research

Key Takeaways

MemHarness is a framework designed to improve how large language model (LLM) agents use past experiences to make decisions.
Retrieving past experiences has become a common strategy to enhance large language model agents.
This ``replay'' paradigm ignores the gap between the abstract, general nature of stored experience and the concrete, ever-changing states encountered at decision time, frequently causing negative transfer.
In contrast, humans rarely recall past experiences verbatim; instead, they reorganize and adapt retrieved memories to fit the present context.
Inspired by this, we propose MemHarness, a framework that equips LLM agents to actively harness and reconstruct past experiences based on the present context.

Paper AbstractExpand

Retrieving past experiences has become a common strategy to enhance large language model agents. However, most existing memory-augmented agents treat retrieved experiences as static records to be replayed verbatim, injecting them into the context regardless of whether they align with the agent's current situation. This ``replay'' paradigm ignores the gap between the abstract, general nature of stored experience and the concrete, ever-changing states encountered at decision time, frequently causing negative transfer. In contrast, humans rarely recall past experiences verbatim; instead, they reorganize and adapt retrieved memories to fit the present context. Inspired by this, we propose MemHarness, a framework that equips LLM agents to actively harness and reconstruct past experiences based on the present context. At each decision step, a unified policy model critiques and reconstructs the retrieved experience conditioned on the current state, producing context-grounded guidance before acting. This reconstructive ability emerges naturally through end-to-end training with GRPO. Experiments on ALFWorld and WebShop show that MemHarness substantially outperforms pure RL and static memory-augmented baselines, demonstrating strong robustness in out-of-distribution (OOD) scenarios. Furthermore, our analyses reveal that this reconstruction objective not only prevents negative transfer but also serves as latent guidance during training, fundamentally improving the agent's intrinsic reasoning capabilities.

MemHarness is a framework designed to improve how large language model (LLM) agents use past experiences to make decisions. While many agents retrieve past records and "replay" them verbatim, this often leads to errors because historical information may not match the current situation. MemHarness instead treats memory as something to be actively reconstructed, allowing the agent to adapt past knowledge to fit the specific context of a new task.

The Problem with Verbatim Replay

Most memory-augmented agents treat retrieved experiences as static records. When an agent encounters a new problem, it inserts these records directly into its context. The authors, led by Rong Wu et al., argue that this "replay" paradigm ignores the gap between the general nature of stored memories and the specific, changing conditions of a new decision-making environment. This mismatch often causes "negative transfer," where irrelevant or outdated information hinders the agent's performance.

How MemHarness Works

MemHarness introduces a five-stage process: environment observation, experience retrieval, memory critique, contextual memory reconstruction, and action generation.
Instead of acting immediately after retrieving a memory, the agent uses a unified policy model to critique the retrieved information against its current state. It then reconstructs the memory into context-grounded guidance, discarding or revising content that does not apply to the current situation. This entire process is trained end-to-end using Group Relative Policy Optimization (GRPO). Because the reconstruction step lacks ground-truth labels, the agent learns this ability implicitly by optimizing for task success.

Performance and Results

The researchers evaluated MemHarness on two benchmarks: ALFWorld, which involves household tasks, and WebShop, which focuses on online shopping.
MemHarness outperformed both pure reinforcement learning (RL) baselines and static memory-augmented methods. On ALFWorld, it achieved an 85.2% success rate, and on WebShop, it reached 75.6%. Ablation studies showed that removing the reconstruction stage caused performance to drop to the level of naive memory replay, suggesting that the ability to adapt memories is the primary driver of the framework's success.

Why It Matters

The authors suggest that their approach bridges the gap between explicit memory banks—which are easy to trace—and parametric memory, which is more adaptive but harder to inspect. By training the agent to reconstruct memories, the framework not only prevents negative transfer but also appears to improve the agent's intrinsic reasoning capabilities. The research indicates that the reconstruction objective serves as a form of latent guidance during training, helping the model become more robust in out-of-distribution scenarios where the environment differs from the agent's training data.