GDPevo: Evaluating Agent Self-Evolution on Real Bus... | AI Research

GDPevo is a benchmark designed to evaluate how well AI agents can "self-evolve"—the process of updating their internal state (such as skills, memory, or prompts) based on prior experience to solve new, related tasks more effectively. The researchers, led by Leijun Zhou and colleagues from PrismShadow and New York University, developed this tool to address the lack of standardized, reliable ways to measure whether an agent is truly learning or simply memorizing data.

The Problem with Current Benchmarks

Existing benchmarks for agent evolution often suffer from three main issues. First, they lack coverage of complex, economically valuable domains like finance, law, and healthcare. Second, many benchmarks do not clearly separate training and testing in a way that proves an agent has actually learned a transferable skill; as a result, performance gains are often difficult to attribute to specific training experiences. Third, many benchmarks are static and public, making them vulnerable to data contamination, where an agent might "see" the test questions during its training phase.

How GDPevo Works

To solve these issues, the researchers introduced "rule hybridization." This method decomposes complex enterprise workflows into atomic, checkable business rules. The benchmark pipeline distributes subsets of these rules across five training tasks and then recombines them in five held-out test tasks. Because these rules are specific to the enterprise environment and not part of the model's general knowledge, an agent must successfully infer the rules during training to perform well on the test tasks.
The benchmark is fully automated. A pipeline generates the tasks, and if a set of tasks becomes exposed or contaminated, the system can generate a new, fresh version of the benchmark in as little as two days. The V1 release includes 120 tasks across 12 groups, while the V2 release expands this to 240 tasks across 24 groups, covering CRM, ERP, finance, healthcare, legal, and data-centric workflows.

Evaluation and Results

The researchers evaluated four different agents using four types of supervision: no-evolution (base), fewshot (using gold answers), reflect (using self-generated scores), and self (using only training questions).
The results showed that self-evolution consistently improved accuracy, with gains of up to 16.44 percentage points compared to the base agents. However, the researchers noted that even the best-performing evolved agents remained significantly below the "oracle ceiling" of 91.6% accuracy. This indicates that while current agents are capable of some level of self-evolution, their potential is far from being fully realized.

Key Considerations

GDPevo introduces several rigorous standards for evaluating agents:

• Deterministic Grading: Instead of relying on LLMs to judge performance, the benchmark uses code-based test cases to ensure scores are reproducible and that failures can be traced to specific rule violations.

• Cost as a Metric: The benchmark tracks token consumption, agent turns, and monetary costs, acknowledging that an effective agent must be efficient as well as accurate.

• Diagnostic Insights: The system provides transfer heatmaps and per-group breakdowns, allowing users to see exactly which domains an agent is learning from and whether that learning helps or hinders performance in other areas.
  The researchers conclude that while the evolution method (such as how a skill is created) matters, the primary driver of performance remains the underlying intelligence of the model itself.