Back to AI Research

AI Research

Rethinking Inference-Time Scaling in Local Computer... | AI Research

Key Takeaways

  • Rethinking Inference-Time Scaling in Local Computer-Use Agents: Failure Modes and Compute Tradeoffs investigates how to improve autonomous computer-use agent...
  • We present a systematic empirical study of inference-time scaling in local CUAs across contextual, temporal, structural, and parallel dimensions.
  • We evaluate Qwen3-VL-8B/30B-A3B, UI-TARS-1.5-7B, and OpenCUA-7B on the OSWorld benchmark.
  • Our results show that additional computation often yields diminishing returns while changing failure modes.
  • Temporal scaling similarly reduces max-step stalls, yet does not substantially improve task success, indicating that longer horizons often extend erroneous trajectories rather than correct them.
Paper AbstractExpand

Deploying autonomous computer-use agents (CUAs) locally is increasingly important for privacy, cost efficiency, and practical usability, yet improving their performance under strict hardware constraints remains challenging. While recent studies show that inference-time scaling can improve frontier computer-use agents through additional computation during execution, its effectiveness for resource-constrained local models remains poorly understood. We present a systematic empirical study of inference-time scaling in local CUAs across contextual, temporal, structural, and parallel dimensions. We evaluate Qwen3-VL-8B/30B-A3B, UI-TARS-1.5-7B, and OpenCUA-7B on the OSWorld benchmark. Our results show that additional computation often yields diminishing returns while changing failure modes. Contextual scaling provides historical grounding that improves trajectory stability and task accuracy, but its gains saturate as token cost increases and failures shift from repetitive or stalled trajectories toward premature false successes. Temporal scaling similarly reduces max-step stalls, yet does not substantially improve task success, indicating that longer horizons often extend erroneous trajectories rather than correct them. We further find that structural decomposition can introduce planning and formatting overhead in local two-stage agents, while parallel scaling partially mitigates these failures at a substantial computational cost. Overall, our findings suggest that efficient local CUAs require selective compute allocation, failure-aware control mechanisms, and agentic frameworks designed around the capabilities and limitations of local models.

Rethinking Inference-Time Scaling in Local Computer-Use Agents: Failure Modes and Compute Tradeoffs investigates how to improve autonomous computer-use agents (CUAs) that run locally on hardware with limited resources. While larger, cloud-based models often use "inference-time scaling"—allocating more computation during execution to improve accuracy—this study examines whether these techniques are effective for smaller, open-source models like Qwen3-VL, UI-TARS, and OpenCUA when tested on the OSWorld benchmark.

Scaling Dimensions and Methodology

The researchers evaluated how four different scaling dimensions affect task success and computational cost:

  • Contextual Scaling: Increasing the number of previous screenshots and actions (history length) provided to the model.

  • Temporal Scaling: Increasing the maximum number of steps allowed to complete a task.

  • Structural Decomposition: Comparing single-agent architectures to two-stage frameworks that separate planning from action grounding.

  • Parallel Scaling: Generating multiple candidate plans to select the best possible action.
    The study measured performance using task success rates, average steps per task, and total prompt token usage to determine if the added compute actually translates into better results.

Performance and Failure Patterns

The study found that adding computation does not always lead to better performance; instead, it often changes how the agent fails.

  • Contextual Gains: Providing a small amount of history (up to four screenshots) is essential for stability and prevents agents from getting stuck in repetitive loops. However, increasing history beyond this point yields diminishing returns and leads to more "premature false successes," where the agent incorrectly decides a task is finished.

  • Temporal Limitations: Increasing the step budget does not significantly improve success rates. It primarily allows the agent to continue down an incorrect path for longer, shifting failures from "max-step stalls" to premature false successes.

  • Structural Overhead: Two-stage agents, which separate planning and grounding, performed worse than single-agent models in this study. The decomposition introduced new failure modes, such as formatting errors where the planner failed to output the correct syntax for the grounding model.

Efficiency Tradeoffs

Parallel scaling—generating multiple plans—can help recover some of the performance lost in two-stage architectures by bypassing invalidly formatted plans. However, this comes at a high cost in token consumption. The researchers conclude that for local models, uniform scaling is inefficient. Instead, they suggest that developers should use moderate history lengths, set bounded temporal budgets, and prioritize structurally simple agents.
The findings suggest that future development of local CUAs should focus on failure-aware control mechanisms and agentic frameworks specifically designed to work within the inherent reasoning limitations of smaller, local models.

Comments (0)

No comments yet

Be the first to share your thoughts!