Back to AI Research

AI Research

Plover: Steering GUI Agents through Plan-Centric In... | AI Research

Key Takeaways

  • Plover: Steering GUI Agents through Plan-Centric Interaction GUI automation agents often struggle in real-world environments because they operate as "black b...
  • Graphical user interface (GUI) automation remains challenging in real-world environments, where dynamic layouts, unexpected dialogs, and evolving interface states can cause autonomous agents to drift from user intent.
  • We present Plover, a plan-centric vision-based GUI automation system that externalizes task plans and replanning as persistent, inspectable, and revisable artifacts.
  • A formative study with six participants informed the interaction design.
  • We then evaluate Plover through benchmark failure-case repair and scenario-based workflow analyses.
Paper AbstractExpand

Graphical user interface (GUI) automation remains challenging in real-world environments, where dynamic layouts, unexpected dialogs, and evolving interface states can cause autonomous agents to drift from user intent. Recent vision-based multimodal agents improve flexibility by operating directly over screenshots and natural language instructions, but planning and adaptation often remain internal, limiting users' ability to inspect, supervise, or correct system behavior. We present Plover, a plan-centric vision-based GUI automation system that externalizes task plans and replanning as persistent, inspectable, and revisable artifacts. Through a planner--executor architecture, Plover supports explicit supervision of evolving execution, localized correction through editable plans, natural-language guidance, and screenshot-grounded interventions, while preserving prior progress during repair. A formative study with six participants informed the interaction design. We then evaluate Plover through benchmark failure-case repair and scenario-based workflow analyses. Our results show that many autonomous GUI-agent failures are structurally repairable when plans remain visible and interventions are localized, and that explicit replanning helps make GUI automation more transparent, controllable, and adaptable.

Plover: Steering GUI Agents through Plan-Centric Interaction
GUI automation agents often struggle in real-world environments because they operate as "black boxes." When these agents encounter unexpected pop-ups or layout changes, they often adapt internally without informing the user, leading to errors that are difficult to diagnose or fix. Plover is a new system designed to solve this by making the agent’s task plan a visible, editable, and persistent artifact. Instead of forcing users to restart a task when an agent makes a mistake, Plover allows them to inspect the agent's logic, intervene with specific corrections, and repair the workflow while preserving the progress already made.

A Plan-Centric Approach

Traditional GUI agents typically treat a task as a single, opaque instruction-to-action flow. Plover changes this by using a planner-executor architecture that externalizes the task plan. By keeping the plan visible, the system allows users to see exactly what the agent intends to do next. If the agent’s plan does not align with the user's intent, the user can step in to edit the plan directly, provide natural language guidance, or use screenshot-based annotations to point out exactly where the agent went wrong.

Intelligent Replanning

A core feature of Plover is "Intelligent Replanning," which functions in two ways. First, it supports user-driven corrections, where a user identifies a mistake and the system updates only the remaining steps of the plan, keeping the successful parts of the workflow intact. Second, it includes system-driven recovery; if the agent detects that it is no longer making progress, it can trigger a replanning event to propose a new path forward. This ensures that the agent does not silently drift or repeat failed actions, but instead communicates its recovery strategy to the user.

Improving Reliability and Control

To evaluate the system, researchers conducted a formative study and a series of performance analyses. They found that many common GUI agent failures—such as clicking the wrong button or getting stuck in a loop—are actually recoverable if the user has the right tools to intervene. In a test of 26 failed tasks, Plover enabled users to repair the workflows, resulting in 17 complete successes and 6 partial successes. The results suggest that by transforming GUI automation from a fully autonomous process into a collaborative, repairable one, users can maintain better control over long-horizon tasks in complex, dynamic software environments.

Key Considerations

While Plover significantly improves transparency and control, it is designed specifically for high-friction or failure-prone workflows where human oversight is valuable. The system acknowledges that not every task requires constant supervision; for simple, low-risk tasks, fully autonomous execution remains appropriate. The primary goal of Plover is to provide a safety net for complex tasks where errors are likely to occur, ensuring that human intervention is precise, localized, and efficient rather than a total restart of the automation process.

Comments (0)

No comments yet

Be the first to share your thoughts!