An agent component can improve outcomes without proving why it helped

Replacing one component of an AI agent can improve the final outcome without proving that its local decisions became better. Shuyang Zhang and Jianshuo Chang examine that distinction in a critical scoping review of component-replacement evidence, posted to arXiv on September 30, 2026. Their question concerns attribution: what a comparison can establish about the source of an observed gain.

A replacement changes more than one decision

The authors describe an agent's execution as a trajectory. Changing a component can affect later observations, resource use, and opportunities to recover from mistakes. A better final task score may therefore reflect several consequences of the replacement, rather than a direct improvement in the quality of one decision.
The review maps 348 studies and examines ninety comparison records, including eighty-eight from forty included studies and two from supplementary studies. Eight selected cases organize its synthesis around the replaced decision, executed conditions, comparable measurements, controls, and explanations that remain possible.
Of 222 studies reporting local decision metrics, 142 also report measured task endpoints and forty-nine report proxies. The authors caution that reporting both kinds of measurement does not establish that they come from matched comparisons. The counts describe measurement coverage, not a success rate for agent improvements.

Different endpoints support different claims

The abstract gives three examples of the distinction. Outcome Monitors reports a package-level completion gain, while attribution to detector quality remains limited. That supports a statement about the package's observed outcome without settling which part caused the gain.
First-chunk selection reports a local improvement assessed against an offline proxy endpoint. An offline proxy addresses a different claim from executing the changed system and measuring its final task result. The review does not treat the two as interchangeable.
Evidence-Carrying Termination reports fewer premature unsupported terminations and completion non-inferiority. Non-inferiority does not establish completion superiority: avoiding a particular failure can be valuable without demonstrating a higher overall completion rate. The distinction prevents a narrower result from becoming an unsupported broader headline.

Examine the comparison before explaining the gain

Across cases, the authors identify candidate mechanisms involving recovery and disruption, intervention timing, and downstream use. They argue that attribution depends on comparison controls, label definitions, and the information available to the controller. Those details determine which alternative explanations a study can rule out.
The review derives eight claim-specific reporting items, but its abstract does not enumerate them. It does state the central limit: online execution, or simultaneous improvement in local and task metrics, is insufficient on its own to show that better local decisions explain the task-level gain.
This is a scoping-review preprint, not a new leaderboard result or a deployment recommendation. Its contribution is a more precise way to read agent evaluations: keep the measured outcome separate from the proposed mechanism, and ask whether the experimental comparison supports both.