Learning to Use Tools: Reinforcement Learning for T... | AI Research

Learning to Use Tools: Reinforcement Learning for Tool-Integrated Mathematical Reasoning explores how integrating external calculator tools and reinforcement learning (RL) can improve the accuracy of large language models (LLMs) on complex mathematical tasks. Researchers Minghui Xu and Zi Wang focus on the "Countdown" task, where models must combine numbers using arithmetic to reach a specific target, finding that calculator integration significantly reduces arithmetic and verification errors.

Addressing Reasoning Failures

The authors identified that a substantial portion of incorrect responses in mathematical reasoning stems from simple calculation errors rather than a lack of logical planning. To address this, they developed a training pipeline that teaches models to use a calculator tool. By inserting and tags into the reasoning process, the model can perform arithmetic checks and adjust its reasoning based on the tool's output. This approach ensures that if a model makes an arithmetic mistake, it can use the calculator to correct the error and continue the reasoning process accurately.

Training with Reinforcement Learning

After establishing a supervised fine-tuning (SFT) baseline, the researchers applied several on-policy reinforcement learning methods—RLOO, RLOO++, GRPO, and DAPO—to further refine the model's performance. These methods use final-answer rewards, which are automatically verifiable in the Countdown task, to provide training signals. The goal is to increase the probability of the model generating correct reasoning traces. Among the tested methods, Tool-DAPO proved the most effective, improving pass@1 accuracy from 35.8% for the Tool-SFT baseline to 66.0%.

Evaluation and Benchmarking

To ensure a reliable assessment of their methods, the authors constructed a new 1,024-problem held-out benchmark. This dataset was designed to have no exact overlap with the training data, providing a more robust evaluation than the standard 50-problem public test set. The results show that tool integration consistently improves performance across both SFT and RL baselines, yielding roughly 10 percentage-point gains in pass@k metrics.

Insights and Limitations

The study suggests that reinforcement learning primarily improves performance by sharpening the model's existing solution distribution—increasing the probability of correct trajectories that the model is already capable of producing—rather than necessarily creating entirely new reasoning capabilities. A key limitation noted is that RL is less effective when the model fails to produce any correct reasoning traces among its sampled responses, as there is little signal to reinforce. Additionally, the authors observed that some RL methods, such as GRPO, can become unstable if they do not properly filter out groups of responses that provide no useful preference signal, leading to a potential collapse in tool usage and accuracy.