A coding agent can satisfy a clear ticket while still struggling to deliver something a customer needs. The FDE-Gym paper studies that distinction with tasks modeled on forward-deployed engineering, where requirements must be clarified while working inside a customer environment.
Evaluate delivery rather than a finished specification
FDE-Gym contains 95 tasks drawn from 20 customer scenarios across 18 sectors. Industry experts adapted the scenarios from delivery projects. Agents interact with scripted stakeholders and can modify production-like systems inside a simulation.
The benchmark grades improvement in the customer's success metric. Scores are normalized between an unchanged system and a reference solution, with penalties for interactions such as messages, trial runs and waiting. This combines the value of the result with the cost of obtaining it.
Understand the best reported result
The best evaluated configuration resolves 14 of the 95 tasks. The paper defines a resolved task as a score of at least 0.8 without an incident. Its best mean score is 0.56, which is a normalized delivery score rather than a claim that 56% of tasks were fully solved.
The authors identify unstated constraints as a major bottleneck. They also show that gathering information is not sufficient: on 23 tasks where the best configuration obtained every key fact, it achieved business uplift of at least 0.8 on only ten.
Use the benchmark's distinction in your own evaluations
The experiments cover 13 model configurations from seven providers. They establish results within the benchmark's customer simulations, rather than demonstrating that a model can independently run a real customer project.
For teams evaluating agents, the useful question is whether a successful test includes requirement discovery, implementation and incident handling. A trial with a complete specification can leave the first of those untested. FDE-Gym offers a way to inspect that gap while keeping the customer interaction and grading criteria visible.
Comments