Multi-Step Tool-Calling over Korean Open Public API... | AI Research

Multi-Step Tool-Calling over Korean Open Public APIs: A Benchmark and a Data-Synthesis Recipe
Public institutions are increasingly required to use open-source, on-premise AI agents to handle sensitive data, often by chaining multiple calls across live government APIs. However, current open-source models struggle with these complex, multi-step tasks, frequently failing to link tool outputs correctly or mismanaging large amounts of data. This paper introduces a new benchmark to measure these failures and a novel data-synthesis method to help models learn how to navigate live API environments more effectively. The ai agents story also surfaces in AI Agents Going Rogue Renew Calls..., adding another angle.

Measuring Real-World API Performance

The researchers developed the Korean Open Public API Benchmark (KOPA-Bench), which consists of 145 real-world tasks across six domains, including law, finance, and transportation. Unlike existing benchmarks that rely on simulations, KOPA-Bench is grounded in live government API endpoints. These tasks are specifically designed to test an agent's ability to handle two common hurdles: "entity code-lookups," where one tool's output must be used as the input for the next, and "high-cardinality responses," where APIs return large volumes of data that the agent must filter or aggregate to provide a correct answer.

Synthesizing Data with EDGE

To improve model performance, the authors created EDGE (Execution-grounded Dynamic Graph). This pipeline generates high-quality training data by building a "dependency graph" that maps how different tools can interact. Crucially, EDGE verifies these connections by actually executing them against live APIs. If a proposed link between two tools fails during execution, the system prunes it from the graph. By labeling these verified paths with information about how much data is being passed between tools, EDGE creates executable training trajectories that teach models how to handle complex, multi-step reasoning and data reduction. The ai agents story also surfaces in OpenAI agents break out of sandbox..., adding another angle.

Significant Gains for Smaller Models

The researchers fine-tuned Qwen3.5 models using the data generated by EDGE and a training technique called GRPO. The results show that even smaller models (9B parameters) can achieve performance levels nearly matching much larger, untuned models (27B parameters). Specifically, the 9B model saw a 10 percentage point increase in its pass@1 score on KOPA-Bench. These improvements were not limited to the specific tasks in the benchmark; the models also showed better performance on the out-of-distribution BFCL benchmark, suggesting that the training method helps models generalize their tool-calling abilities across different environments.

Key Takeaways

The study demonstrates that the quality of training data—specifically data verified through live execution—is a primary driver of model improvement in tool-calling tasks. By using a dynamic graph that adapts based on real-world API behavior, the researchers were able to overcome the limitations of traditional, static datasets. The findings suggest that for institutions needing to deploy AI on local infrastructure, focusing on execution-grounded synthesis is a highly effective strategy for bridging the performance gap between smaller, efficient models and their larger, more resource-intensive counterparts. To see openai in practice, Flova AI is the ONLY Video... walks through a concrete example. as detailed in the full paper on Arxiv