A changing task can break in several places
The EnvOS paper studies computer-use agents working with information that changes during a task. An appointment may move or a policy may be updated after an agent has already read the old value.
A failure in that setting is not automatically an agent failure. The update might not have fired correctly, the task might have become impossible or the grader might have reached the wrong conclusion. EnvOS therefore checks the environment and scoring machinery alongside the agent.
Certify the change and retain the evidence
Before a run, the framework checks that the intended change occurs, that the task remains solvable and that the grader distinguishes correct from incorrect behavior. Afterward, separate code rechecks the verdict using archived evidence.
In a clinic-booking experiment, Claude Opus 5 succeeds on none of sixteen tasks without an extra observation, two when reminded to recheck the source and fifteen when shown the updated value. The reported advantage over the reminder survives correction for the tested comparisons. That is a result for this model and interface, not a universal statement that reminders never work.
The auditor has limits too
The authors also test corrupted episodes. EnvOS rejects all twenty-six deliberately corrupted cases in one set, but catches twenty-five of thirty-nine in an externally generated set. This is stronger than the simple log checks in that comparison, while still leaving undetected cases.
For an agent evaluation, retain evidence of both what changed and what the agent actually received. Separate the final task result from the validity of the intervention and the scoring process. The framework’s useful contribution is an auditable test setup, rather than a claim that either an agent or its benchmark grader can be assumed correct without inspection.
Comments