Back to AI Research

AI Research

ASIL: Replacing Screenshot-and-Click with Structure... | AI Research

Key Takeaways

  • ASIL (Agent-Software Interaction Layer) is a new interface designed to improve how AI agents operate software.
  • Powerful code agents can execute scripts, call tools, and manage files, yet many important applications remain accessible primarily through graphical user interfaces.
  • We argue that screenshot-and-click is an inefficient interface for software-operating agents: screenshots are state-incomplete, and GUI actions are brittle, semantically weak, and poorly matched to long-horizon planning.
  • We instantiate ASIL across 15 applications and a benchmark of 300 single-application and 80 multi-application tasks.
  • ASIL reaches above 80 with closed models while executing fewer than five actions per task.
Paper AbstractExpand

Powerful code agents can execute scripts, call tools, and manage files, yet many important applications remain accessible primarily through graphical user interfaces. We argue that screenshot-and-click is an inefficient interface for software-operating agents: screenshots are state-incomplete, and GUI actions are brittle, semantically weak, and poorly matched to long-horizon planning. We introduce ASIL (Agent-Software Interaction Layer), an agent-native interface that exposes software through structured JSON observations and code-executable semantic actions, realized through the deepest feasible access path for each application. We instantiate ASIL across 15 applications and a benchmark of 300 single-application and 80 multi-application tasks. ASIL reaches above 80 with closed models while executing fewer than five actions per task. Under a repaired runtime and a 50-step screenshot budget, the same tasks yield 6.6 and 26.6 strict success under screenshot-and-click control, rising to 15.0 and 53.3 on an easier OSWorld-comparable band. Against application-native interfaces on matched tasks, ASIL exceeds LibreOffice's UNO API by 28-38 strict points but only matches this http URL 's MCP content contract. The structured modality also suits training: small-scale SFT raises Qwen3.5-2B from 58.0 to 72.1 and Qwen3.5-9B from 66.6 to 80.4, and resource-limited on-policy RL further raises them to 74.4 and 82.2.

ASIL (Agent-Software Interaction Layer) is a new interface designed to improve how AI agents operate software. Instead of relying on the traditional "screenshot-and-click" method—where an agent interprets visual pixels and performs manual mouse movements—ASIL provides agents with structured JSON data representing the software's internal state and allows them to execute high-level, semantic commands.

The Problem with Screenshot-and-Click

Current GUI agents function by imitating human behavior, which researchers Rui Xie and Lu Chen argue is inefficient for software automation. Screenshots are "state-incomplete," meaning they fail to capture hidden panels, background processes, or metadata that an agent needs to make accurate decisions. Furthermore, GUI actions like clicking or dragging are brittle; if a layout or theme changes, the agent’s previously learned motor sequences often fail. This approach requires long, complex chains of actions to complete simple tasks, increasing the likelihood of errors and high latency.

How ASIL Works

ASIL replaces visual observations and coordinate-based clicks with a structured protocol. The agent receives a JSON object that explicitly details the software’s current state, such as active documents, editable entities, and environment context. In response, the agent emits semantic actions—such as "modify_file," "set_value," or "invoke_function"—that directly trigger software operations.
The researchers realize this interface through three primary access paths:

  • File-backed execution: Modifying structured file formats directly.

  • Native scripting: Executing code within the application’s runtime (e.g., Blender Python).

  • Service/API execution: Using REST or WebSocket endpoints to communicate with the software.

Performance and Training

In a benchmark of 380 tasks across 15 applications—including creative tools like GIMP and Inkscape, and productivity suites like LibreOffice—ASIL significantly outperformed screenshot-based agents. While screenshot-and-click agents often struggled with long, multi-step sequences, ASIL agents frequently completed tasks in fewer than five actions.
The structured nature of ASIL also makes it highly effective for training. The researchers found that using verified ASIL trajectories for supervised fine-tuning (SFT) and on-policy reinforcement learning (RL) led to consistent performance gains. For example, the Qwen3.5-9B model improved from a baseline score of 66.6 to 82.2 after training with these methods.

Limitations and Scope

ASIL is not a universal solution for all software. The researchers note that the approach is only applicable to software that exposes at least one open read path and a semantic action path. Applications that are entirely opaque—lacking parseable files, scripting interfaces, or service APIs—remain outside the scope of this framework. Additionally, the researchers emphasize that while ASIL provides a more stable interface, the initial setup requires human-reviewed interface profiles to map software functions into the ASIL contract.

Comments (0)

No comments yet

Be the first to share your thoughts!