A later instruction can change what counts as success
The DynaRev paper studies web tasks whose requirements change during execution. A user might add a constraint, replace an earlier preference or make a new requirement conditional on something the agent discovers.
That differs from a fixed instruction spread over several messages. The active goal itself can change, so a successful answer to the initial request may no longer satisfy the user’s final requirements.
Deliver the revision at a meaningful moment
DynaRev contains 181 tasks and uses semantic milestones to trigger revisions at task-defined stages. This aims to make the timing meaningful even when different agents follow different paths through a website.
The benchmark covers additive revisions, unconditional overrides and runtime-conditioned overrides. Its evaluation examines correctness, responsiveness and efficiency using the trajectory, rather than reducing the entire interaction to one final success label.
Preserve useful state while updating the instruction
Five-model evaluations show partial satisfaction of requirements and variation across models, task sources and revision types. The authors also report gains from instruction rewriting with browser state preserved, while browser resets degrade performance in the examined configurations.
Those findings do not establish one revision strategy as optimal for every interface. They show why testing a user change in isolation can miss the state and progress the agent had already accumulated.
For an operational evaluation, retain which requirements remain active after each revision and the moment at which the change was delivered. Check whether the agent follows the new condition without silently dropping an earlier constraint that still applies. DynaRev supplies a testbed for that behavior, while the actual site state, revision timing and resource cost remain important parts of interpreting a result.
Comments