Interface-Induced Trajectory Censoring | AI Research

Interface-Induced Trajectory Censoring investigates a critical disconnect between how AI models are trained and how they are evaluated. The paper demonstrates that the "tool-call rate"—a key metric used to judge an agent's ability to use external software—is often misleading. Because the serving interface (the software layer that sits between the model and the environment) may fail to recognize the specific format a model uses to request a tool, it can "censor" valid calls before they ever reach the execution stage. This creates a situation where a model appears incapable of using tools, even when it is successfully generating well-formed requests.

The Measurement Trap

The core problem is that an agent evaluation does not measure the model in isolation; it measures the entire stack, including the chat template, the parser, and the execution protocol. When these components are mismatched, the system may report zero tool calls. The author shows that this is not necessarily a bug in the parser itself, but a failure of the "contract" between the model’s training and the interface’s expectations. For example, a model might emit a perfectly valid JSON call, but if the interface is configured to look for a specific XML-style tag that the model was not trained to produce, the interface will simply discard the output. To see meta in practice, How to Make Cinematic Commercials walks through a concrete example.

Scale-Dependent Distortion

The research reveals that this censoring effect grows as models get larger. In tests using Qwen2.5-Coder, the server-side tool-call rate remained at zero across all model sizes. However, when the researchers analyzed the raw output, they found that larger models were actually emitting valid tool calls at a high rate (up to 80% for the 32B model). Because the interface was misaligned, the "silent fraction"—the number of valid calls that were ignored—increased as the model became more capable. This suggests that researchers might be underestimating the tool-use capabilities of larger models simply because their evaluation infrastructure is filtering out the evidence.

Impact on Training

This issue is not limited to final evaluations; it also distorts the training process. When using reinforcement learning (RL) to train agents, the training loop relies on receiving feedback from tool executions. If the interface censors the model's tool calls, the training loop never sees a successful execution, meaning the model receives no signal to reinforce that behavior. The study found that at certain scales, models were generating complete, valid calls that were never accepted or executed, effectively leaving the agent with no "tool-mediated" experience to learn from. The ai agents story also surfaces in AI Agents Going Rogue Renew Calls..., adding another angle.

A Practical Solution

To address this, the author argues that researchers should stop treating tool-call rates as a simple property of the model. Instead, they should verify the entire stack before drawing conclusions about model performance. The paper provides a 98-line preflight check designed to catch these silent failures. By separating the model’s intent from the interface’s parsing, developers can determine whether a failure is due to the model’s lack of capability or a simple mismatch in the communication protocol between the model and the server. The same ai evaluation question is explored in Wrong Prediction, Right Answer, which adds a research perspective. as detailed in the full paper on Arxiv