Do Agent Optimizers Compound? A Continual-Learning...

Do Agent Optimizers Compound? A Continual-Learning... | AI Research

Key Takeaways

Most research on AI agent optimization focuses on "one-shot" improvements, where an agent is tuned once against a fixed set of tasks.
Most reported gains from agent-optimization methods are one-shot: an agent is optimized against a fixed benchmark and the resulting improvement is reported as if it were a stable property of the method.
This does not test the setting that matters for deployed agents, where optimization is applied recursively as new failures and new tasks appear over time.
The central question this raises is whether optimizer-driven gains compound: after an agent has been optimized once, can it be optimized again on newly arrived tasks without eroding the gains the first round produced?
All three methods improve over the baseline agent in the conventional, static, single-phase setting.

Paper AbstractExpand

Most reported gains from agent-optimization methods are one-shot: an agent is optimized against a fixed benchmark and the resulting improvement is reported as if it were a stable property of the method. This does not test the setting that matters for deployed agents, where optimization is applied recursively as new failures and new tasks appear over time. The central question this raises is whether optimizer-driven gains compound: after an agent has been optimized once, can it be optimized again on newly arrived tasks without eroding the gains the first round produced? We study this question with a two-phase continual-learning evaluation built from hard tasks in Terminal-Bench 2.0, comparing three approaches to agent-harness optimization (GEPA, Meta Harness, and RELAI's Verifiable Continual Learning, RELAI-VCL) under identical optimization budgets. All three methods improve over the baseline agent in the conventional, static, single-phase setting. However, once new tasks are introduced, the methods diverge sharply: GEPA's optimized agent transfers below the unoptimized baseline, Meta Harness transfers well but fails to improve further once given a second optimization budget, and RELAI-VCL is the only method that both transfers positively to unseen tasks and continues improving after those tasks are folded into the optimization objective, reaching the highest pass rate at every evaluated stage and the highest lifelong average pass rate overall (76.4% vs. 66.0% for GEPA, 64.6% for Meta Harness, and 58.7% for the baseline). Our key observation was that optimization gains compounded only when regression control was built into the optimization loop, providing an inductive bias against shortcut solutions that fail to generalize.

Most research on AI agent optimization focuses on "one-shot" improvements, where an agent is tuned once against a fixed set of tasks. However, in real-world deployment, agents must be updated repeatedly as new tasks arise and previous ones evolve. This paper investigates whether these optimization gains actually "compound"—meaning an agent can be improved over time without losing the skills it previously mastered. By testing three different optimization methods on the Terminal-Bench 2.0 benchmark, the authors demonstrate that standard optimization techniques often fail to handle this recursive process, leading to either overfitting or performance plateaus.

The Challenge of Recursive Optimization

The researchers introduce a two-phase evaluation protocol to measure how well agents handle continuous improvement. In Phase 1, agents are optimized on an initial set of tasks. In Phase 2, they are tasked with learning new, additional problems while maintaining their performance on the original set. This setup exposes a critical flaw in many current methods: they often treat optimization as a static, one-time event. Without a mechanism to protect previously learned capabilities, an agent might "forget" how to solve old tasks while trying to master new ones, or it may simply stop improving altogether when faced with a broader, more complex set of requirements.

Comparing Optimization Strategies

The study compares three distinct approaches to agent-harness optimization:

GEPA: An evolutionary method that uses reflective, natural-language feedback to mutate agent prompts.
Meta Harness: An outer-loop method where an agentic proposer directly edits the agent's underlying harness code.
RELAI-VCL: A regression-aware approach that integrates a "no-regression" constraint directly into the search process.
While all three methods successfully improved agent performance in the initial, static phase, they diverged significantly once the researchers introduced new tasks and repeated the optimization process.

Why Regression Control Matters

The key finding is that optimization gains only compound when the system includes "regression control" within the search loop. RELAI-VCL was the only method to achieve both positive transfer to unseen tasks and continued improvement during the second phase. By rejecting any candidate edits that would cause the agent to regress on previously solved tasks, RELAI-VCL avoids the pitfalls of its competitors. In contrast, GEPA tended to overfit to the initial tasks, resulting in poor performance on new ones, while Meta Harness stalled, failing to make further progress when given a second budget.

Implications for Deployed Agents

The results highlight that a high score on a static benchmark is not a reliable indicator of how an agent will perform in a dynamic, production environment. For agents that need to be updated recursively, the ability to avoid "catastrophic forgetting" is just as important as the ability to learn new skills. The authors suggest that future agent-optimization research should move away from one-shot evaluations and instead adopt protocols that explicitly measure how well an agent retains and builds upon its knowledge over time.