QuoteBench: How Matched Scores Can Hide Command-Pat... | AI Research

QuoteBench: How Matched Scores Can Hide Command-Path Failures investigates why current benchmarks for LLM coding agents often fail to distinguish between a model’s ability to generate correct code and the failures that occur when that code is processed by downstream systems. The researchers, Shangao Li, Yao Zhang, Volker Tresp, and Yuanyuan Yang, demonstrate that "matched" execution scores—where a model is evaluated based on the final outcome of its generated commands—can mask significant errors introduced by the interface between the model and the shell.

The Problem with Matched Scores

Current agent benchmarks typically evaluate success by running a generated command and checking the final state of the system. However, LLM coding agents often operate through interfaces that serialize, wrap, or reparse model output before it reaches a shell. This means a command might be perfectly generated by the model but corrupted by the "transport" path (such as being reparsed inside double quotes when sent to a remote server or container). Because standard benchmarks only look at the final result, they cannot tell if a failure was caused by the model’s poor coding or by the execution environment’s handling of that code.

How QuoteBench Works

To isolate these variables, the authors created QuoteBench, a set of 56 one-shot tasks covering 14 operation families, such as handling hostile filenames, multiline text, and complex shell metacharacters. The researchers use a "crossed design" to independently vary two factors:

• Generation Contract: How the model is instructed to format its output (e.g., raw Bash versus a disclosed boundary that warns the model about downstream parsing).

• Execution Transport: How the command is actually run (e.g., direct execution versus a "nested" path that adds an extra layer of parsing, simulating remote or containerized environments).
  By fixing the model's output and replaying it through different transport paths, the researchers can decompose a matched score into two parts: "transport damage" (failures caused by the environment) and "contract-conditioned compensation" (improvements gained when the model is warned about the environment).

Key Findings

The study reveals that aggregate success scores often hide large, opposing effects. For example, in the case of the GPT-5.6-sol model, a matched score gap of -3.6 points was found to actually consist of -64.3 points of damage caused by the transport path, offset by +60.7 points of compensation from the model adapting its generation.
Across eight configurations, replaying replies through an added parser lowered success rates by 55.4 to 73.2 percentage points. While some models were able to recover significant performance when the boundary was disclosed to them, others showed little to no improvement. The authors conclude that raw generation is nearly saturated at the frontier, meaning that the ability to adapt to the command path—rather than raw coding ability—is what currently separates the most capable models.

Implications for Evaluation

The authors argue that evaluations of command-issuing agents should stop treating matched scores as an intrinsic property of a model. Instead, they recommend that researchers and system builders report the full configuration, including the generation contract, the execution path, the operating point, and the final-state validator. Because trivial fixes like correct escaping or using temporary scripts can eliminate these failures, the researchers emphasize that the primary contribution of QuoteBench is the measurement of the command path itself, rather than the development of a new repair technique.