Back to AI Research

AI Research

Beyond Final Scores: A Systematic Evaluation of Age... | AI Research

Key Takeaways

  • Beyond Final Scores: A Systematic Evaluation of Agents for Long-Horizon AI Research and Development investigates the performance of autonomous AI agents in c...
  • Autonomous agents are increasingly capable of improving models, systems, and other technical artifacts through long-horizon experimentation.
  • These findings suggest concrete directions for improving model training, inference-time strategies, experience management, and harness design.
  • Beyond Final Scores: A Systematic Evaluation of Agents for Long-Horizon AI Research and Development investigates the performance of autonomous AI agents in complex, multi-step research tasks.
  • The authors, including researchers from Meituan and the University of Chinese Academy of Sciences, argue that relying on final scores is insufficient for understanding how agents function.
Paper AbstractExpand

Autonomous agents are increasingly capable of improving models, systems, and other technical artifacts through long-horizon experimentation. To understand the current state of this capability, however, evaluation must go beyond final scores, which neither reveal where progress is gained or lost nor indicate whether accumulated experience improves later decisions. We therefore present a systematic evaluation of seven frontier models on 36 long-horizon tasks based on a new framework that uses rule-based metrics to characterize within-run behavior through Solution Framing, Execution, and Feedback Control and controlled comparisons to assess experience reuse within and across tasks. The results show that current agents operate more like engineering optimizers than fully autonomous researchers: they can formulate and implement practical solutions, but their performance varies substantially across runs, their strongest solutions mainly adapt or combine established techniques, and genuine methodological novelty remains rare. Detailed analysis reveals that observed performance is shaped by multiple factors, including distinct process bottlenecks behind similar final outcomes, experience reuse that can help or mislead subsequent decisions, and harness designs that affect performance stability. These findings suggest concrete directions for improving model training, inference-time strategies, experience management, and harness design.

Beyond Final Scores: A Systematic Evaluation of Agents for Long-Horizon AI Research and Development investigates the performance of autonomous AI agents in complex, multi-step research tasks. The authors, including researchers from Meituan and the University of Chinese Academy of Sciences, argue that relying on final scores is insufficient for understanding how agents function. Instead, they propose a framework to evaluate the internal research process, including how agents frame solutions, execute tasks, and respond to feedback.

Evaluating the Research Process

To move beyond final scores, the researchers decomposed the research loop into three core capabilities: Solution Framing (C1), Execution (C2), and Feedback Control (C3). Using 36 expert-curated tasks—ranging from model development to CUDA optimization—the team evaluated seven frontier models. They used rule-based metrics derived from verifier outcomes and trajectory signals rather than subjective LLM judgments. This allowed the team to pinpoint exactly where an agent’s performance succeeds or fails, such as whether a model struggles to propose a viable direction or fails to implement a correct solution.

Performance and Reliability

The study found that while several models can reach competitive final results, they differ significantly in consistency. The gap between the strongest and weakest models was 0.237 on average performance (avg@3) but only 0.122 on peak performance (best@3). This indicates that many models are capable of finding high-quality solutions but struggle to do so reliably across repeated runs. The researchers conclude that current agents function more like engineering optimizers than autonomous researchers, as their strongest solutions typically adapt existing techniques rather than generating novel methodologies.

The Role of Experience and Harnesses

The evaluation revealed that research performance is not solely determined by the underlying model. The researchers found that accumulated experience can either help or hinder an agent; in some cases, it allows the agent to preserve useful discoveries, while in others, it leads to misleading conclusions or traps the agent in local optima. Additionally, the "harness"—the system surrounding the agent—was found to influence run-to-run stability. The authors suggest that future improvements should focus on better experience management, inference-time strategies, and more robust harness designs to close the gap between average and peak performance.

Franklin Analysis

The evidence suggests that current evaluation methods for autonomous research are incomplete. By showing that two models can achieve similar final scores while possessing vastly different strengths in execution or feedback control, the paper demonstrates that a single score masks the underlying mechanics of AI performance. The data supports the conclusion that "reliability" is a more significant differentiator among current frontier models than "peak capability," providing a clear direction for developers to focus on consistency and process-level refinement rather than just final output quality.

Comments (0)

No comments yet

Be the first to share your thoughts!