Back to AI Research

AI Research

CoReMem evaluates what coding agents should remember between tasks

Key Takeaways

  • CoReMem tests which notes help on the next coding task and charges the agent for memory overhead.
  • Its approach keeps the inherited repository fixed.
  • CoReMem studies which pieces of coding agent memory pay off on a later task.
  • It compares runs that share the same starting repository but see different memory.
  • Coding agents working on the same repository across multiple tasks can leave notes for their next session.

Coding agents working on the same repository across multiple tasks can leave notes for their next session. In the CoReMem paper, researchers ask how to tell whether those notes save more effort than they cost to produce and read. They study both what the agent records and whether the next agent uses it well.

Comparing notes against the same starting point

The researchers use successive tasks on an evolving repository. Edits from an earlier task carry forward, but the later task is unknown when the agent writes its notes. To estimate the value of a piece of memory, the authors compare runs of a later task with and without that segment, starting from the same inherited repository state. That makes it possible to separate the value of the note from the value of code that was already changed.

Measuring when memory pays for itself

The framework accounts for memory overhead and gives credit for completing the task with fewer steps. It also distinguishes steps that appear to use a detail supplied by memory from those that do not. The authors call this provenance-gated credit. An agent can use a detail from an earlier run as a starting point without treating the entire previous trace as instructions for the next task.

Using the result to evaluate a repository workflow

CoReMem's comparison is tied to the later task and the inherited state it was evaluated against. A useful note in one task sequence can be irrelevant in another. For teams deciding what an agent should keep, the practical evaluation is to hold the repository fixed, change only the notes and compare task resolution as well as the steps spent getting there. The study provides a way to measure this tradeoff. It does not establish a universal size or format for memory across repositories.

Comments