OSReward: Instituting Standardized Evaluation for C...

OSReward: Instituting Standardized Evaluation for C... | AI Research

Key Takeaways

OSReward introduces a standardized benchmark to evaluate how reliably vision-language models (VLMs) act as judges for computer-using agents (CUAs).
Computer-using agents (CUAs) are advancing rapidly across the digital world.
A CUA trajectory records the agent's actions, states, and reasoning.
Verifying whether it fulfilled the task instruction is central to CUA evaluation, data curation, and reinforcement learning.
Neither human-written verifiers nor human annotators can provide such verification at scale, so the field increasingly turns to vision-language models (VLMs) as judges of CUA trajectories.

Paper AbstractExpand

Computer-using agents (CUAs) are advancing rapidly across the digital world. A CUA trajectory records the agent's actions, states, and reasoning. Verifying whether it fulfilled the task instruction is central to CUA evaluation, data curation, and reinforcement learning. Neither human-written verifiers nor human annotators can provide such verification at scale, so the field increasingly turns to vision-language models (VLMs) as judges of CUA trajectories. But a fundamental question has long gone unexamined: are these VLM judges reliable enough? To study it systematically, we introduce OSReward, a realistic, high-quality benchmark that evaluates VLM judges on CUA trajectories. The trajectories come from diverse agent backbones executing human-verified instructions across platforms, then rigorously labeled with ground-truth verdicts through multi-stage human annotation. Building on it, we derive OSReward-Hard, a challenge set concentrating genuinely hard cases, and OSReward-Multi for fine-grained efficiency and alignment scoring. The most comprehensive evaluation of VLM judges to date finds even state-of-the-art models fall short of an ideal judge, sharing a systematic leniency bias that mislabels failed runs as successes. The few reliable enough to trust are too expensive to run at scale, while affordable open models trail far behind. To close this gap, we construct and release OS-Shepherd-100K, an open corpus of reasoning-annotated trajectory judgments for the CUA community. On it, we train OS-Shepherd (9B and 35B), open reward models that supply low-cost, stable, and reliable reward signals, matching commercial judges at 30-60% lower cost than the frontier. Extensive analyses further inform the design of reliable CUA reward at scale. Our code, benchmark, dataset, and model checkpoints are available at this https URL .

OSReward introduces a standardized benchmark to evaluate how reliably vision-language models (VLMs) act as judges for computer-using agents (CUAs). As these agents perform tasks across web, mobile, and desktop environments, verifying their success is essential for training and data curation. However, the researchers found that current VLM judges often suffer from a "leniency bias," where they incorrectly label failed agent trajectories as successful.

Evaluating VLM Judges

The researchers built OSReward from scratch using dedicated cross-platform infrastructure. Unlike previous studies that reused existing agent trajectories—which can confound judge errors with poor agent performance—OSReward uses 1,019 human-verified, high-quality trajectories. These trajectories were generated by four different agent model families and subjected to a rigorous, multi-stage human annotation process.
The benchmark includes three specific views:

OSReward: The full set of trajectories covering diverse platforms and task types.
OSReward-Hard: A challenge set of 284 trajectories that were difficult even for human annotators to judge, designed to expose specific failure modes in VLM judges.
OSReward-Multi: A subset of successful trajectories that includes fine-grained labels for alignment and efficiency.

Performance and Bias

In an evaluation of 27 different VLM judges, the researchers found that even state-of-the-art models struggle to act as reliable judges. While top-tier models perform adequately on the full set, their accuracy drops significantly on the OSReward-Hard subset, with the mean judge accuracy falling to 52%.
A consistent finding across these models is a systematic leniency bias. Judges are frequently fooled by "false successes," where an agent claims to have completed a task but has actually failed. The study notes that the most reliable judges, such as Claude-Opus-4-8 and GPT-5.5, are too expensive for large-scale use, while more affordable open models currently lack the necessary accuracy.

OS-Shepherd Models

To address the gap between reliability and cost, the researchers developed OS-Shepherd-100K, an open corpus of 100,000 reasoning-annotated trajectory judgments. Using this data, they trained two open reward models: OS-Shepherd-9B and OS-Shepherd-35B.
These models were trained using a two-stage recipe that specifically targets the false-success failure mode identified in the benchmark. According to the researchers, these models provide a stable, reliable reward signal at 30–60 times lower cost than frontier commercial judges, making them a viable option for academic and large-scale agent training.

Franklin Analysis

The evidence provided by the authors suggests that the reliability of VLM judges is a critical bottleneck for the development of autonomous computer-using agents. By creating a standardized, human-verified benchmark, the researchers demonstrate that current industry-standard models are not as objective as previously assumed. The transition from using expensive, proprietary models to the open-source OS-Shepherd models represents a practical shift toward making high-quality reward signals accessible for research at scale.