UserSWE evaluates coding agents in conversations where a developer’s information arrives over several turns. The study argues that a complete issue report presented once does not fully represent interactive bug fixing. It also examines verdicts that can accept an incorrect patch or reject a correct one when the evaluation evidence is incomplete.
A user that can reveal information gradually
The UserSWE paper builds multi-turn tasks from repository history and existing single-turn benchmarks. Information pools constrain what the simulated user can reveal at each turn, while a persona process governs how the user communicates.
The authors calibrate that process on 20,859 development sessions. The result is still a simulated user, not a live developer study, but it gives the benchmark a more explicit interaction model than simply repeating the original issue text.
Ground each verdict in execution
Tests run after every turn. Two agents from different model families write and gate the verdict, with test results as the primary evidence and model judgment reserved for aspects the tests cannot settle.
That separation is relevant to coding evaluation because a plausible explanation of a patch is not the same as evidence that the repository behaves correctly. It also makes the verdict procedure part of the benchmark to inspect, rather than treating the final success label as self-explanatory.
The turn budget changes the ranking
The authors evaluate eleven agents on 72 tasks from 42 repositories. They report different leaders at one turn, intermediate budgets and eight turns. The study therefore does not support reducing interactive coding ability to a single one-turn ranking.
For teams comparing coding assistants, a useful evaluation should specify the allowed interaction budget and the evidence required to declare a bug fixed. Additional turns can change both what the user reveals and how an agent revises its work. UserSWE makes those factors measurable within its constructed tasks, while leaving transfer to other repositories and real developer behavior as separate questions.
Comments