Back to AI Research

AI Research

From Atomic to Agentic: Towards Interpretable Evalu... | AI Research

Key Takeaways

  • From Atomic to Agentic: Towards Interpretable Evaluation of LLMs' Agentic Mathematical Capabilities introduces AgenticMathBench (AMB), a benchmark designed t...
  • Large Language Models (LLMs) are evolving from performing end-to-end mathematical reasoning to integrating agentic intelligence.
  • However, most existing math benchmarks evaluate only final answers.
  • This outcome-oriented evaluation provides limited diagnostic value for identifying process-level failures or rigorous logic, failing to guide the transformation of LLMs into robust agents.
  • To bridge this gap, we present a process-level benchmark designed to evaluate the inherent agentic mathematical reasoning abilities of LLMs.
Paper AbstractExpand

Large Language Models (LLMs) are evolving from performing end-to-end mathematical reasoning to integrating agentic intelligence. However, most existing math benchmarks evaluate only final answers. This outcome-oriented evaluation provides limited diagnostic value for identifying process-level failures or rigorous logic, failing to guide the transformation of LLMs into robust agents. To bridge this gap, we present a process-level benchmark designed to evaluate the inherent agentic mathematical reasoning abilities of LLMs. Our framework aligns problem-solving agentic behaviors with a structured taxonomy of reusable mathematical atomic capabilities. We design a comprehensive suite of planning, action, and feedback tasks across both textual and multimodal contexts, supported by an automated pipeline that synthesizes high-quality trajectories and produces fine-grained annotations via controlled LLM rewriting. Experiments reveal that models with similar end-to-end accuracy can exhibit markedly different agentic capability profiles. This demonstrates that process-level evaluation is crucial for interpreting the true potential of LLMs and guiding the development of next-generation mathematical agents.

From Atomic to Agentic: Towards Interpretable Evaluation of LLMs' Agentic Mathematical Capabilities introduces AgenticMathBench (AMB), a benchmark designed to evaluate the process-level mathematical reasoning of Large Language Models (LLMs). While traditional benchmarks focus on final answers, this framework decomposes mathematical problem-solving into atomic, reusable capabilities to diagnose specific failures in planning, action, and feedback. Methods and results are detailed in the full paper on arxiv.org.

Evaluating Agentic Intelligence

The researchers, including Jiayi Kuang and colleagues from Sun Yat-sen University, Tencent Youtu Lab, and the University of Illinois Chicago, argue that current benchmarks fail to capture the "agentic" potential of LLMs. By aligning mathematical skills with agentic functions, the framework evaluates models on three core axes:

  • Planning: The ability to select atomic capabilities, create a solution roadmap, and make dynamic decisions during a task.

  • Action: The execution of specific, decoupled mathematical tasks such as calculation, formalization, and spatial perception.

  • Feedback: The ability to evaluate trajectories, localize errors, and suggest corrections. The agentic story also surfaces in NVIDIA DeepStream 9.1 Adds Agentic Skills..., adding another angle. The same Reasoning question is explored in GRAIN, which adds a research perspective.

Benchmark Construction

To support this evaluation, the team developed an automated pipeline that synthesizes high-quality mathematical trajectories. They collected and filtered 27 existing datasets, converting them into a unified schema. The benchmark includes both text-only and multimodal contexts. To ensure the evaluation is interpretable, the researchers decoupled these agentic functions so that each can be diagnosed in isolation rather than relying on a single end-to-end success metric.

Performance Insights

Experiments conducted on various models—including general open-source, math-specialized, and commercial models—revealed that end-to-end accuracy is an insufficient indicator of a model's true agentic capability. Key findings include:

  • Planning Gaps: While many models can generate a plan offline, they struggle with "next-step planning," or dynamic decision-making during execution.

  • Feedback Limitations: Even high-performing commercial models struggle to convert their identification of an error into an effective, actionable fix.

  • Capability Profiles: Models with similar end-to-end accuracy scores often exhibit markedly different strengths and weaknesses across the atomic capability taxonomy. The same Large Language Models question is explored in StrategyBench, which adds a research perspective.

Implications for Future Development

The researchers conclude that process-level evaluation is necessary to guide the development of robust mathematical agents. Because the benchmark reveals failure modes that are hidden by traditional outcome-oriented testing, it provides a clearer path for training models to improve their reasoning processes rather than just their final answers. The authors note that memory-related tasks were excluded from this benchmark because they are difficult to map to specific mathematical atomic capabilities and challenging to evaluate in a controlled, comparable manner.

More from Franklin

GRAINresearchStrategyBenchresearchLearning to Use ToolsresearchNVIDIA DeepStream 9.1 Adds Agentic Skills...newsGoogle brings Gemini Spark agentic AI...newsProQurAItool

Comments