JarvisGUI: Towards Cross-Device GUI Agents with Dyn... | AI Research

JarvisGUI: Towards Cross-Device GUI Agents with Dynamic Task Composition introduces a new benchmark designed to evaluate how well AI agents can perform tasks that span multiple devices and operating systems. While current AI agents are often tested on single-device tasks, real-world digital workflows frequently require users to coordinate actions across smartphones, personal computers, and servers. This paper highlights that existing benchmarks fail to account for the complexities of transferring data, maintaining state, and managing long-term dependencies across different platforms, leading to an overly optimistic view of how ready these agents are for real-world use.

Bridging the Cross-Device Gap

The researchers developed JarvisGUI to address the limitations of current benchmarks, which typically focus on isolated environments. The new benchmark creates a heterogeneous ecosystem that includes Android, Windows, and Ubuntu. By requiring agents to perform tasks that involve moving information, sharing files, and maintaining context across these different platforms, the benchmark exposes a significant capability gap in current open-source GUI agents. It reveals that these agents struggle with the advanced reasoning and dependency management necessary for multi-device workflows. The same large language models question is explored in Harness-of-Harness, which adds a research perspective.

How the Benchmark Works

To create a reliable and scalable evaluation system, the researchers formulated GUI tasks as input-output transformations using a lightweight type system. This allows for the automatic composition of multi-step, cross-device workflows. The system is organized into four layers:

• Infrastructure Layer: Uses Docker-based virtualization to ensure reproducible and isolated environments.

• Environment Control Layer: Provides a unified interface to mask the differences between Android, Windows, and Ubuntu.

• Model Interaction Layer: Uses a "Planner-Grounder" architecture to translate natural language instructions into actionable sequences across devices.

• Evaluation Execution Layer: Manages the lifecycle of tasks and performs automated scoring by inspecting the final state of the environment.

Data Collection and Quality Control

The benchmark was built using a multi-stage pipeline that combines human annotation with automated synthesis. Human experts designed template tasks, which were then stitched together into complex task graphs using the researchers' type system. To ensure high quality, the team used an "LLM-as-a-judge" framework to filter tasks based on realism, coherence, and evaluability. This was further validated by human experts, who confirmed that the vast majority of the generated tasks were accurate and reflected genuine user needs. The same ai evaluation question is explored in Multi-Step Tool-Calling over Korean Open Public..., which adds a research perspective.

Key Findings

The evaluation of representative open-source GUI agents using JarvisGUI revealed that there is a substantial disparity between an agent's proficiency on a single device and its competence in a cross-device setting. The results demonstrate that current agents face persistent challenges in preserving context and reasoning across different platforms. By identifying these bottlenecks, the researchers provide a clearer picture of the hurdles that must be overcome to deploy GUI agents in realistic, multi-device environments. The same ai systems question is explored in Efficient Test-Time Adaptation through Human-AI Interaction, which adds a research perspective. as detailed in the full paper on Arxiv