Back to AI Research

AI Research

FinEvo-Bench: A Longitudinal Benchmark for Self-Evo... | AI Research

Key Takeaways

  • FinEvo-Bench is a longitudinal benchmark designed to measure how effectively AI agents learn from experience to improve their performance in professional fin...
  • Most agent benchmarks evaluate tasks independently and cannot measure whether experience from one task helps with later tasks.
  • Existing self-evolution benchmarks do not jointly cover professional workflows, open-ended deliverables, and multi-aspect evaluation.
  • We introduce FinEvo-Bench, a longitudinal benchmark with 120 real-case-grounded tasks, 20 business scenes across six financial domains.
  • Institution-provided professional procedures define the required operations and constraints.
Paper AbstractExpand

Most agent benchmarks evaluate tasks independently and cannot measure whether experience from one task helps with later tasks. Existing self-evolution benchmarks do not jointly cover professional workflows, open-ended deliverables, and multi-aspect evaluation. We introduce FinEvo-Bench, a longitudinal benchmark with 120 real-case-grounded tasks, 20 business scenes across six financial domains. Institution-provided professional procedures define the required operations and constraints. Eligible institution-provided and publicly documented cases supply the task facts. Each scene contains six related but substantively distinct cases that share a professional procedure and a manually reviewed rubric for task quality and financial compliance. We compare four self-evolving agent scaffolds using the same Qwen3.7-Max backbone and three independently shuffled, globally interleaved task streams. Paired non-evolving controls estimate each scaffold's self-evolution gain from retained experience, while an independent Claude Code scoring agent backed by Claude Opus 4.6 evaluates all outputs. Letta achieves the highest evolved score (91.65) and fewest compliance issues (0.09 per task); Codex achieves the largest self-evolution gain (+19.37). Across scaffolds, the evolving condition raises scores by 9.33-19.37 points and reduces compliance issues by 0.12-0.44 per task. Paired score gains at within-scene ranks 4-6 exceed those at ranks 1-3 by 6.10-8.70 points. In Claude Code, skill-only evolution produces higher task quality and fewer compliance issues than memory-only and combined memory-skill evolution. Across all four scaffolds, rubric feedback also yields higher scores and fewer compliance issues than reference-answer feedback. FinEvo-Bench measures both professional performance and self-evolution ability: how effectively an agent turns prior experience into later improvement.

FinEvo-Bench is a longitudinal benchmark designed to measure how effectively AI agents learn from experience to improve their performance in professional financial workflows. While most benchmarks evaluate agents on independent tasks, this research introduces a framework that tests whether an agent can retain and apply knowledge across a series of related, real-world financial cases.

Evaluating Self-Evolution

The researchers define self-evolution as the process where an agent extracts signals from completed tasks and feedback, updates its persistent state, and applies that knowledge to future work. To test this, the benchmark uses 120 tasks across 20 business scenes, such as credit reviews and claim analyses. Each scene contains six distinct cases that share a professional procedure but differ in specific facts and risks. By using globally interleaved task streams and paired non-evolving control agents, the study isolates the performance gains directly attributable to retained experience rather than the agent's base capabilities.

Methodology and Scoring

The benchmark employs a rigorous evaluation protocol to ensure consistency. Each agent's output is scored by an independent Claude Code agent backed by Claude Opus 4.6, which uses a manually reviewed rubric to assess both task quality and financial compliance. This automated judge was validated against human financial experts, achieving a high absolute agreement score (ICC(A,1) = 0.95). To prevent agents from simply memorizing answers, the benchmark provides feedback based on professional procedures and constraints rather than reference solutions, requiring agents to generalize lessons to new, substantively different cases.

Performance Results

The study compared four agent scaffolds—Claude Code, Codex, Letta, and GenericAgent—using a Qwen3.7-Max backbone. All four scaffolds showed improved performance when allowed to evolve, with scores increasing by 9.33 to 19.37 points and compliance issues decreasing by 0.12 to 0.44 per task compared to their non-evolving controls.
Key findings include:

  • Highest Performance: Letta achieved the highest overall evolved score (91.65) and the fewest compliance issues (0.09 per task).

  • Largest Gains: Codex demonstrated the most significant self-evolution gain (+19.37 points).

  • Longitudinal Improvement: All scaffolds showed greater performance gains in the latter half of the task streams (ranks 4–6) compared to the beginning (ranks 1–3), suggesting a cumulative benefit from retained experience.

  • Feedback Efficacy: Rubric-based feedback consistently resulted in higher scores and fewer compliance issues than reference-answer feedback across all tested scaffolds.

Considerations for Agent Design

The research highlights that there is no single "best" scaffold when balancing performance, self-evolution gains, and computational cost. For instance, while Letta and Codex performed well, they incurred higher token costs due to their memory management strategies. Conversely, GenericAgent was the most cost-efficient but showed the lowest overall performance and smallest evolution gains. Additionally, the study found that "skill-only" evolution—where agents distill experience into reusable skills—produced higher quality and better compliance than memory-only or combined memory-skill approaches. These results suggest that how an agent stores and retrieves experience is as critical to its professional utility as its underlying model architecture.

Comments (0)

No comments yet

Be the first to share your thoughts!