Measure Before You Manage: Evaluating Agent Working... | AI Research

"Measure Before You Manage: Evaluating Agent Working Memory in Coding Agents" investigates how the semantic roles of different data objects—such as instructions, code artifacts, and tool outputs—affect the performance and management of coding agents. The authors argue that because these objects have different sizes, lifecycles, and representation needs, treating them as a uniform pool of tokens is insufficient. The paper proposes a four-level evaluation framework to better assess memory-management strategies.

Understanding Semantic Heterogeneity

The researchers analyzed 55 archived coding-agent trajectories to characterize how working memory functions. They found that different object types exhibit distinct behaviors:

• Tool outputs account for the largest share of content volume (55.5%).

• Artifacts (source code) have a larger median size and longer residency than tool outputs, meaning their impact on memory cost is higher than a simple volume-based count suggests.

• Representation varies by type; for example, artifacts are more compressible than tool outputs.
  The authors conclude that because these objects are not uniform, memory-management policies—such as those governing eviction or compression—should be "semantically informed" rather than based on simple, uniform rules.

Evaluating Management Strategies

The study examined two semantically informed management approaches: an object-aware compression policy and a retrieval-based policy.

• Object-Aware Compression: This policy uses type-specific weights and rules to manage memory. While it showed performance gains during initial calibration, these gains did not consistently transfer to held-out tasks.

• Retrieval-Based Policy: This approach uses recency, relevance, and importance scores to manage memory. The evaluation showed that even when agents are assigned equal token budgets, they do not necessarily receive the same amount of "delivered context" or incur the same management costs.

The Four-Level Evaluation Framework

The authors suggest that evaluating memory management requires looking beyond nominal token budgets. They propose a framework consisting of four reporting levels: 1. Stored State: Tracking the type, size, representation, and residency of objects. 2. Delivered Context: Measuring the actual state provided to the agent, rather than relying on a shared token cap. 3. Management Work: Accounting for the computational costs of memory management, such as embedding generation or summary calls. 4. Task or Process Outcome: Using process metrics to evaluate performance, while being careful not to conflate repeated tool calls with successful task repair.

Limitations and Considerations

The authors note several limitations to their findings. The evidence base is small and repository-clustered, and the study relies on archived trajectories where historical conditions are not perfectly reconstructable. Furthermore, the researchers emphasize that their results are specific to the coding agents and settings studied. They caution that a "winning" policy was not identified; instead, the study serves to illustrate that apparent gains in memory management can be misleading if the evaluation does not account for the underlying semantic structure of the agent's working memory.