MissClick: Exploiting Digit-Serialized Coordinates...

MissClick: Exploiting Digit-Serialized Coordinates... | AI Research

Key Takeaways

MissClick is a white-box adversarial attack designed to disrupt GUI visual grounding models by exploiting how these systems translate screen coordinates into...
Recent GUI visual grounding models generate screen coordinates as sequences of digit tokens that are parsed into numerical values and mapped to executable clicks.
The security implications of this coordinate generation process have been largely overlooked.
This observation motivates attack objectives that account for the numerical and place-value structure of coordinate outputs rather than treating them as ordinary text.
Moreover, untargeted and targeted attacks impose different success conditions--displacing the click outside the correct region versus into an attacker-specified region--and therefore benefit from different objectives.

Paper AbstractExpand

Recent GUI visual grounding models generate screen coordinates as sequences of digit tokens that are parsed into numerical values and mapped to executable clicks. The security implications of this coordinate generation process have been largely overlooked. We observe that each coordinate digit is predicted as a categorical token, yet after parsing, changing a hundreds-place digit by one changes the corresponding numerical coordinate component by 100 units, which can induce a large displacement of the executed click. This observation motivates attack objectives that account for the numerical and place-value structure of coordinate outputs rather than treating them as ordinary text. Moreover, untargeted and targeted attacks impose different success conditions--displacing the click outside the correct region versus into an attacker-specified region--and therefore benefit from different objectives. We propose MissClick, a simple and effective white-box adversarial attack with two goal-specific objectives: MissClick-U maximizes soft-coordinate displacement for untargeted disruption, while MissClick-T minimizes a place-weighted target-digit loss for targeted hijacking. Compared with existing attacks against GUI grounding models on OS-Atlas and UGround across desktop, web, and mobile platforms, MissClick-U achieves untargeted success rates of 75.07\% and 72.93\% (+16.62 and +30.72 pp), and MissClick-T achieves targeted success rates of 44.86\% and 62.67\% (+31.73 and +47.06 pp). Attack objective comparison further shows that soft-coordinate displacement yields the highest untargeted attack success rate, whereas place-weighted target-digit optimization yields the highest targeted attack success rate, revealing distinct objective preferences for the two attack goals.

MissClick is a white-box adversarial attack designed to disrupt GUI visual grounding models by exploiting how these systems translate screen coordinates into executable clicks. Researchers Yu Ran, Wentao Zhao, Xin Zhang, and Yi Pan identify that because these models generate coordinates as sequences of digit tokens, they are vulnerable to perturbations that manipulate the numerical value of those digits. By accounting for the decimal place value of each digit, the attack can induce significant, unintended click displacements.

The Vulnerability of Digit-Serialized Coordinates

GUI grounding models typically function by predicting screen coordinates as a series of individual digit tokens. Once generated, these tokens are parsed into numerical values (e.g., hundreds, tens, and ones places) that determine where a click occurs. The authors observe that existing adversarial attacks often treat these outputs as standard text, failing to account for the fact that a small change in a high-order digit—such as the hundreds place—results in a much larger physical shift on the screen than a change in the ones place. MissClick leverages this numerical structure to create more effective adversarial perturbations.

Two Goal-Specific Attack Objectives

The researchers propose two distinct objectives for MissClick, tailored to the specific goals of an attacker:

MissClick-U (Untargeted): This objective aims to disrupt the model by maximizing the distance between the intended click and the actual executed click. It uses "soft coordinates," which are differentiable representations of the model’s digit probability distributions, to push the output away from the ground-truth region.
MissClick-T (Targeted): This objective aims to hijack the click to a specific, attacker-chosen location. It uses a "place-weighted target-digit loss," which applies cross-entropy to the target digits while weighting them by their decimal place value. This ensures the optimization process prioritizes the high-order digits that have the greatest impact on the final coordinate value.

Performance and Effectiveness

In experiments conducted on the OS-Atlas and UGround models across desktop, web, and mobile platforms, MissClick demonstrated significant improvements over existing attack methods. For untargeted attacks, MissClick-U achieved success rates of 75.07% and 72.93%, representing gains of 16.62 and 30.72 percentage points, respectively. For targeted attacks, MissClick-T achieved success rates of 44.86% and 62.67%, marking improvements of 31.73 and 47.06 percentage points over prior approaches.

Considerations for GUI Security

The authors note that their findings are specific to models that use coordinate-generation pipelines. Models that utilize coordinate-free action heads do not expose the digit-token interface required for this attack and are therefore outside the scope of this research. Furthermore, the study focuses on perturbations applied to screenshot pixels and does not address other potential security vectors like prompt injection or agent-level attacks. The research highlights that treating coordinate outputs as mere text ignores the underlying numerical logic that governs how GUI agents interact with digital interfaces.