Act As a Real Researcher: A Suite of Benchmarks Eva... | AI Research

Act As a Real Researcher: A Suite of Benchmarks Evaluating Frontier LLMs and Agentic Harnesses in Research Lifecycle
This paper introduces the AARR (Act As a Real Researcher) benchmark series, a new framework designed to evaluate how well AI agents can perform the nuanced, professional tasks required in scientific research. While current AI agents are proficient at coding and executing experiments, they often lack the "researcher-like" qualities—such as integrity, scientific judgment, and the ability to handle uncertainty—that are essential for true scientific work. The authors propose AARRI-Bench (Act As a Real Research Intern) as the first step in this series, focusing on tasks that are straightforward for human researchers but challenging for current AI systems.

Evaluating Research-Specific Qualities

Unlike existing benchmarks that primarily measure whether an agent can complete a task or write code, AARRI-Bench assesses the quality of the research process itself. The benchmark categorizes 82 manually crafted tasks along two dimensions: the type of challenge (Context, Mindset, Hands-on, and Interaction) and the level of autonomy required (from basic adaptation to open-ended innovation). By focusing on these areas, the researchers aim to identify where AI agents struggle to mimic the professional behavior, skepticism, and independent decision-making of human interns.

The Performance Gap

The study tested various combinations of AI models and agent "harnesses" (the scaffolding that allows models to interact with tools and environments). The results show that even the best-performing configuration—the Mini-SWE-Agent paired with Claude Opus 4.7—achieved only a 68.3% success rate. The researchers found that agents frequently overlook subtle, critical details that a human would easily notice. This indicates that current systems are still far from being able to replace human researchers, as they often struggle with the nuanced reasoning required in a research environment.

Minimalist Design vs. Complex Scaffolding

A surprising finding from the experiments is that more complex agent scaffolding does not necessarily lead to better performance. The minimalist Mini-SWE-Agent outperformed more feature-heavy systems like Claude Code and Hermes Agent. The authors suggest that overly rigid or complex frameworks may actually hinder highly intelligent models by creating unnecessary cognitive overhead. In contrast, a simpler interface allows the model to navigate research environments with greater flexibility.

Key Takeaways for Future Development

The research highlights that the primary bottleneck for autonomous research agents remains the intrinsic reasoning capability of the underlying AI model. While scaffolding is important, the authors conclude that developing "researcher-like" AI requires a deeper focus on research behavior and methodology rather than just building increasingly complex agent architectures. Future stages of the AARR series will continue to explore these gaps, moving from the intern level to research assistants and eventually to independent research scientists.