ToolCUA: Towards Optimal GUI-Tool Path Orchestratio... | AI Research

Computer Use Agents (CUAs) are designed to automate desktop tasks by interacting with graphical user interfaces (GUIs) through clicks and typing. While these agents can also use high-level tools (like API-based file operations), they often struggle to decide when to use a tool versus when to stick to standard GUI actions. This confusion leads to inefficient, brittle, or failed task execution. The paper introduces ToolCUA, an end-to-end agent trained to master this "hybrid action space" by learning how to orchestrate GUI actions and tool calls for optimal performance.

Scaling Data Without Manual Effort

A major hurdle in training these agents is the lack of high-quality examples showing how to switch between GUI actions and tool calls. Collecting such data manually is expensive and difficult. To solve this, the researchers developed an "Interleaved GUI-Tool Trajectory Scaling Pipeline." This system takes existing, static GUI-only data and uses advanced models to synthesize a library of tools based on the actions already present in those recordings. By converting GUI-only sequences into hybrid trajectories—where some steps are replaced by tool calls—the team created a large, diverse dataset for training without needing to manually build or instrument complex environments.

Training for Better Decision-Making

Once the data was prepared, the team used a two-stage training paradigm. First, they performed "Tool-Bootstrapped GUI Reinforcement Finetuning," which uses supervised learning to teach the agent the basics of tool usage, followed by single-turn reinforcement learning to help the agent make better decisions at critical "switching points" (where it must choose between a GUI action or a tool). Second, they employed "Online Agentic Reinforcement Learning" in a live environment. During this phase, the agent is guided by a "Tool-Efficient Path Reward," which provides feedback based on two factors: whether the tool was actually appropriate for the task and whether the total number of steps was minimized.

Achieving State-of-the-Art Results

The effectiveness of ToolCUA was tested on the OSWorld-MCP benchmark, a standard for evaluating computer-use agents. ToolCUA achieved an accuracy of 46.85%, representing a 66% relative improvement over the baseline model. Notably, the agent performed better than models limited to GUI-only actions, proving that its ability to orchestrate a hybrid action space leads to more efficient and reliable automation. The results also showed that the agent could generalize its skills to unseen applications and platforms, such as different Linux tasks and Windows desktop apps.

Key Takeaways

The research suggests that the primary challenge for modern digital agents is not just the ability to use tools, but the ability to know when to use them. By moving away from simple step-by-step imitation and toward trajectory-level optimization, ToolCUA demonstrates that agents can learn to replace long, error-prone sequences of GUI clicks with precise, efficient tool calls. This approach provides a scalable path forward for creating more capable and reliable digital assistants for real-world desktop environments.