On the Fragility of Self-Improving Agents: Variance...

On the Fragility of Self-Improving Agents: Variance... | AI Research

Key Takeaways

On the Fragility of Self-Improving Agents: Variance, Task Order, and Underspecification evaluates the reliability of memory-based self-improving agents—syste...
Memory-based self-improving agents--those that learn from an online stream of tasks and improve over time by maintaining a textual memory bank--have shown great promise in recent literature.
However, the reliability aspects of these methods have been critically overlooked.
Second, the agent's improvement is highly dependent on task order.
Prior works often adopt default orderings that impose an implicit curriculum, acting as a hidden prerequisite for success.

Paper AbstractExpand

Memory-based self-improving agents--those that learn from an online stream of tasks and improve over time by maintaining a textual memory bank--have shown great promise in recent literature. However, the reliability aspects of these methods have been critically overlooked. In this work, we conduct a comprehensive re-evaluation of two memory-based methods, broadening the scope of evaluation along two axes: (1) including multiple runs to quantify variance, and (2) randomly shuffling the tasks to investigate the effect of task order. Through these experiments, we make two observations that expose the fragility of current methods: First, agent evaluation is inherently noisy in complex environments and on multi-step tasks, and stacking a self-improving loop on top can further amplify this noise. Second, the agent's improvement is highly dependent on task order. Prior works often adopt default orderings that impose an implicit curriculum, acting as a hidden prerequisite for success. To better understand this fragility, we manually examine the agents' memory and hypothesize that task and environment underspecification contribute to this fragility. We validate this hypothesis by incorporating information that enables better specification, such as detailed rubrics and environment feedback, into the memory construction process. While this added information partially closes the performance degradation in previous experiments, significant gaps still remain, suggesting that other uncharacterized factors contribute to this fragility. Looking ahead, our work advocates for more rigorous evaluation protocols for self-improving agents by reporting results across multiple runs and stress-testing them under challenging conditions. Moreover, our findings on underspecification call for systems and interfaces that enable effective human oversight, preventing agents from failing in unforeseeable ways.

On the Fragility of Self-Improving Agents: Variance, Task Order, and Underspecification evaluates the reliability of memory-based self-improving agents—systems that learn from an online stream of tasks by storing textual memories. Researchers from Salesforce AI Research re-evaluated two existing methods, Agent Workflow Memory (AWM) and ReasoningBank (RBank), to determine if these agents perform consistently when subjected to varied conditions.

Evaluating Agent Reliability

The authors tested these agents across three web-browsing benchmarks: WebArena, VisualWebArena, and SCUBA. To assess reliability, the study moved beyond single-run evaluations, which are common in existing literature. Instead, the researchers conducted multiple runs to measure variance and shuffled the order of tasks to see if the agents relied on a specific, implicit curriculum. They found that self-improving agents are highly sensitive to these factors, often showing significant performance fluctuations that single-run tests fail to capture.

The Impact of Task Order and Variance

The study reports that self-improving agents often perform better when tasks are presented in a default, easy-to-hard order. When the researchers randomly shuffled the task order, the agents' performance degraded by 4.5% instead of showing the expected improvement. Additionally, the researchers observed that the self-improving loop can amplify noise; in 71% of the cases tested, the variance across multiple runs increased compared to a baseline agent without a memory bank. In some instances, the gap between the best and worst runs reached 10 percentage points.

The Role of Underspecification

By manually inspecting the agents' memory banks, the researchers identified "underspecification" as a primary driver of fragility. Agents frequently generated memories that were plausible but technically impossible to execute in the given environment, such as recommending API calls in a browser-only environment that does not support them. Furthermore, ambiguous task queries led agents to develop "contagious" but incorrect strategies—such as using the Haversine Formula for distance calculations when a website failed to load—which were then stored and retrieved for future tasks, compounding errors over time.

Limitations and Future Directions

The researchers attempted to mitigate these issues by providing the agents with more detailed rubrics and environment feedback. While these interventions closed 31% of the performance gap observed in shuffled task order settings, significant performance deficits remained. The authors conclude that current self-improving methods require more rigorous evaluation protocols, such as reporting results across multiple runs and stress-testing under challenging conditions. They also suggest that future systems should incorporate human oversight interfaces to identify and correct the "wrong lessons" agents learn during their self-improvement process.