How Benchmarks Mis-Score Computer-Use Agents | AI Research

How Benchmarks Mis-Score Computer-Use Agents examines why current evaluation methods for computer-use agents (CUA) are unreliable. The authors argue that benchmark scores are outputs of a flawed pipeline rather than direct observations of agent capability, leading to inaccurate leaderboards and misdirected engineering efforts.

The Evaluation Pipeline Problem

The authors define CUA evaluation as a four-stage process: task construction, trajectory observation, scoring, and reporting. Reliability issues arise at each stage. During construction, tasks can become stale or contaminated by external data. During scoring, "brittle" scripted oracles—which check only specific strings, URLs, or machine states—often reject valid alternative solutions. Finally, reporting often relies on a single scalar success rate that obscures the specific cause of failure, making it impossible to distinguish between a bad plan, a broken environment, or a simple execution error.

Audit of Benchmark Verdicts

To test the reliability of these evaluations, the researchers audited 150 trajectories from five public benchmarks (OSWorld, WebArena, VisualWebArena, WorkArena, and AssistantBench) that had been labeled as failures. They found that 15.3% of these "FAIL" verdicts were incorrect. Specifically, 10.7% were evaluator false negatives—where the agent actually completed the task but the automated checker failed to recognize it—and 4.7% were caused by broken tasks, such as dead search engines or empty system directories.

Diagnostic Taxonomy for Failures

For the genuine failures identified in the audit, the authors applied a three-tier diagnostic taxonomy to categorize why agents failed:

• Tier 1 (Planning and Specification): These are strategy failures, such as planning loops or attempting to use features that do not exist.

• Tier 2 (Execution and Grounding): These involve errors in selecting the correct GUI coordinates, losing context, or using invalid tool arguments.

• Tier 3 (Verification and Feedback): These occur when an agent fails to notice that its actions are not changing the state of the environment, leading to repetitive, ineffective behavior.
  The audit revealed that Tier 3 feedback-blindness (29.5%) and Tier 1 planning errors (35.2%) are the most common causes of genuine agent failure, while Tier 2 execution errors account for only 13.9%.

Franklin Analysis

The evidence suggests that the current reliance on scalar success rates is insufficient for diagnosing agent performance. Because 15.3% of the audited failure verdicts were incorrect, the authors conclude that researchers should distrust binary scores and instead prioritize trajectory-level evidence. The authors note that while LLMs can assist in auditing these trajectories, human-level agreement remains a challenge, as evidenced by the moderate inter-rater reliability scores (κ = 0.41 for LLMs and κ = 0.59 for human groups) observed during the study. This indicates that even with better diagnostic frameworks, interpreting agent behavior remains a complex task requiring careful, independent validation.