A language model can write a plausible instrument description without producing a design that works. Lijie Ding and Changwoo Do build NeutronGym to evaluate the latter: agents configure neutron instruments, and a physics simulator grades the resulting observables.
The NeutronGym paper combines a tool environment, a small published-instrument benchmark and procedural training families. Its central training result concerns setting numeric parameters within fixed layouts. The scored tasks do not ask agents to invent a new instrument topology.
The simulator supplies the reward
Agents use 22 validating tools exposed through MCP to inspect components, build an instrument and run simulations. McStas compiles the instrument and ray-traces neutron behavior; a language model does not judge whether the design is good.
The environment grades progress along four levels: syntax, runtime, structure and science. A candidate must survive engineering checks before its physical performance earns higher credit. For the target-matching families, it must reproduce observables from a hidden design within prescribed tolerances.
That grading still requires defensive task design. The authors report nine ways to score without doing the intended design work and reject four task designs. One early family looked successful until a hand-coded rule copying limits from the prompt achieved nearly the trained model's pass rate.
The admitted families face fixed-answer and prompt-reading probes. These checks reduce specific shortcuts, while the paper acknowledges that class-level degeneracy remains an open problem.
Training improves a bounded design task
On the guide-matching family, reinforcement learning raises Qwen3-8B from 11.3% to 76.7% on 300 held-out instances. A second training seed reaches 69.0%, demonstrating that the headline result varies across training runs.
The held-out split includes parameter regimes disjoint from training, rather than merely new random seeds. Three additional gated families also improve with the recipe, but each rests on one training seed. Those results support further experiments, not a broad assertion that an eight-billion-parameter model has mastered scientific instrument design.
The reward ladder's partial credit is important: removing it costs about 60 percentage points in the reported analysis. At the agent's simulation budget, the trained model reaches a result close to a classical optimizer given the closed-form physics. Frontier models remain ahead on this family.
Reproduction remains unreliable
The curated McStasBench slice contains 16 scored tasks per model and arm. The best reported model completes seven in the one-shot setting and five through the tool loop. No agent meets the improvement target in the scored improvement task or its development counterpart.
The authors caution that one episode per task and such a small benchmark cannot establish a reliable model ranking. A reproduction pass certifies the graded beam-delivery observables, not every class-defining performance characteristic of an instrument.
NeutronGym demonstrates how simulator-backed tasks can provide more concrete feedback than fluent explanations. Its strongest contribution includes the failed task designs and shortcut probes: teams training scientific agents need to test the reward itself before interpreting a high pass rate as physical reasoning.
Comments