Back to AI Research

AI Research

PointerBench tests the precise GUI targets that broad task scores can hide

Key Takeaways

  • A diagnostic benchmark covers spreadsheet sub-elements, text characters and interfaces from 100 professional applications.
  • A model may understand an instruction yet click the wrong character, cell boundary or small interface element.
  • The [PointerBench paper](https://arxiv.org/abs/2610.31274) studies that fine-grained grounding problem separately from whether an agent eventually completes a long workflow.
  • The benchmark contains 1,500 screenshot-instruction pairs.
  • The authors curate pixel-exact target geometry and compare general-purpose multimodal models with 39 specialist GUI-grounding configurations.

A model may understand an instruction yet click the wrong character, cell boundary or small interface element. The PointerBench paper studies that fine-grained grounding problem separately from whether an agent eventually completes a long workflow.

Three kinds of precise target

The benchmark contains 1,500 screenshot-instruction pairs. PointerBench-Sheets targets thin spreadsheet sub-elements and geometry, PointerBench-Text examines character-level grounding in long documents, and PointerBench-Pro covers mixed interfaces across 100 professional applications.
The authors curate pixel-exact target geometry and compare general-purpose multimodal models with 39 specialist GUI-grounding configurations. Those conditions make the benchmark useful for diagnosing where a target was missed, rather than attributing every failed task to planning or tool selection.

Text is the hardest reported setting

The paper finds text grounding especially challenging. It reports success on PointerBench-Text of 69.6% for Claude Fable 5, 53.4% for GPT-5.6 Sol and 25.8% for the leading native specialist UI-Venus-2-9B. Their respective average success across the benchmark is 83.2%, 77.7% and 54.5%.
These are model names and results as reported in the study. The difference between the text subset and the overall score is important: a strong average can conceal a weak capability that matters to a particular application.

Match the evaluation to the action

For a workflow that edits a document or spreadsheet, inspect the relevant target category rather than relying only on an end-to-end completion rate. A reproduction should preserve screenshots, instruction wording, target geometry and the scoring rule so near misses can be examined consistently.
The benchmark does not establish reliable operation across every screen resolution, interface change or application state. It provides a more specific diagnostic question: can the model locate the exact element the instruction names? That question is worth testing before giving a computer-use agent a task where a small targeting error changes the meaning of the action.

Comments