Same Simulation, Different Evidence examines an agent failure that is easy to miss: a tool computes the right result, but the assistant answers from a different representation of that result. The paper studies context management using PHREEQC, a deterministic geochemistry simulator.
The experiment separates computation from evidence use
The authors use 158 execution-verified multiple-choice items and rerun 11,007 logged simulator calls. They compare putting a report in context with giving an agent a section index for retrieving it from a file.
Under retrieval, GPT-5.1 and GPT-5.2 lost 11.3 and 7.9 percentage points of accuracy. Claude Opus 4.6 and Sonnet 4.6 showed no detectable cost in that comparison. These are model- and setup-specific findings, rather than a universal penalty for retrieval.
Why correct computation was not enough
The reruns showed that the two GPT systems computed the keyed value about as often as before. What changed was the evidence used in the final answer. They sometimes skipped the report and relied on higher-precision output files, then rounded to a neighbouring answer option.
For GPT-5.4, more frequent correct simulation masked an evidence-use loss, leaving the overall change indistinguishable from zero. A stable final score could therefore conceal opposing changes in two parts of the workflow.
A useful lesson for harness design
Naming the authoritative output improved accuracy by 6.3 to 7.9 points over plain retrieval for the three GPT models studied. Inlining outputs that fit also helped GPT-5.1 and GPT-5.2, while a rounding instruction alone did not solve the problem.
Franklin’s interpretation is to trace both the tool result and the source used for the answer. When an agent is wrong, distinguish a bad computation from selecting the wrong file, field or precision. That produces a more actionable diagnosis than changing a prompt solely because the aggregate score fell.
Read the paper for the controlled comparisons and qualifications.
Comments