Back to AI Research

AI Research

EvoAgentBench: Benchmarking Agent Self-Evolution vi... | AI Research

Key Takeaways

  • EvoAgentBench: Benchmarking Agent Self-Evolution via Ability Transfer AI agents are increasingly capable of solving complex, long-horizon tasks, but they oft...
  • Agent self-evolution in long-horizon LLM systems is largely procedural: useful experience is not merely stored information, but reusable procedures for searching, debugging, and verification.
  • Yet current evaluations do not isolate this form of transfer.
  • Agent benchmarks test single-episode task solving; memory benchmarks target information retention rather than procedural reuse.
  • We introduce EvoAgentBench, a benchmark for agent self-evolution via Ability-guided transfer across four agentic domains: web research, algorithmic reasoning, software engineering, and knowledge work.
Paper AbstractExpand

Agent self-evolution in long-horizon LLM systems is largely procedural: useful experience is not merely stored information, but reusable procedures for searching, debugging, and verification. Yet current evaluations do not isolate this form of transfer. Agent benchmarks test single-episode task solving; memory benchmarks target information retention rather than procedural reuse. We introduce EvoAgentBench, a benchmark for agent self-evolution via Ability-guided transfer across four agentic domains: web research, algorithmic reasoning, software engineering, and knowledge work. EvoAgentBench extracts trace-grounded Abilities from agent executions, canonicalizes them into operational units, and builds domain-specific Ability Graphs linking tasks that share procedural overlap. By design, every test task is backed by verified training-side Ability support. Across a 528/267 train/test split, two scaffolds, and three backbones, curated Ability content transfers reliably across model families, but no current automatic method sustains positive gain in all settings. EvoAgentBench shifts self-evolution evaluation from aggregate accuracy comparison to fine-grained diagnosis of experience encoding, routing, and uptake. The benchmark is publicly available at this https URL .

EvoAgentBench: Benchmarking Agent Self-Evolution via Ability Transfer
AI agents are increasingly capable of solving complex, long-horizon tasks, but they often struggle to learn from their past experiences. While current systems can store information, they frequently fail to convert that experience into reusable procedures—such as specific debugging steps or search strategies—that can be applied to new, unseen tasks. This paper introduces EvoAgentBench, a new evaluation framework designed to measure how well agents can "self-evolve" by transferring these procedural abilities across different domains.

A New Way to Measure Learning

Existing benchmarks often focus on whether an agent can solve a single task or recall specific information. EvoAgentBench shifts the focus to "procedural transfer." It evaluates whether an agent can take a lesson learned from a training task and successfully apply it to a different, but related, test task. To ensure the evaluation is fair and rigorous, the researchers developed a system where every test task is guaranteed to have "Ability support" from the training set, meaning the necessary procedural knowledge is available to be learned.

How the Benchmark Works

The construction of EvoAgentBench relies on a three-stage pipeline: 1. Trace Collection: The researchers collect execution logs from multiple AI models across four domains: web research, algorithmic reasoning, software engineering, and knowledge work. 2. Ability Extraction: Instead of using broad labels, the system extracts "Abilities"—specific, reusable operations like a particular validation workflow or a search strategy—directly from the agent’s actual performance traces. 3. Ability Graphs: These abilities are organized into an "Ability Graph," which maps tasks that share similar procedural requirements. This allows the benchmark to split data based on these shared skills rather than random task selection, ensuring that the test set truly challenges an agent’s ability to transfer knowledge.

Key Findings

The researchers tested several automatic self-evolution methods against a "diagnostic reference" (the benchmark’s own curated ability-based skills). Their findings highlight a significant gap in current technology:

  • Reliability Issues: While the diagnostic reference condition consistently improved performance across all domains, current automatic methods for self-evolution remain brittle.

  • Negative Transfer: The study identified instances of "negative transfer," where an agent’s performance actually worsened after attempting to use an injected artifact or learned procedure.

  • Cost vs. Accuracy: The researchers noted that the computational cost of these methods is not a reliable indicator of how much they will improve an agent’s accuracy, suggesting that developers should track both performance and overhead simultaneously.

Moving Forward

EvoAgentBench provides a diagnostic tool for researchers to pinpoint exactly where an agent’s self-evolution process breaks down. By isolating whether a failure stems from missing training data, flawed extraction of procedures, or poor retrieval of those procedures at test time, the benchmark offers a clearer path for developing more robust, self-improving AI agents.

Comments (0)

No comments yet

Be the first to share your thoughts!