Clean Engineering, Unstable Measurement: A Preregis... | AI Research

Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared Endpoints
This paper investigates a fundamental, often overlooked assumption in AI research: that using a language model as a "judge" to score or rank other models is a stable, reliable measurement process. The authors conducted two rigorous, preregistered research campaigns to test whether the same request, sent to the same model name, consistently produces the same result. They found that even when the engineering behind the requests was perfect—with every technical detail, schema, and metadata field verified—the measurement instrument itself was fundamentally unstable, failing to meet the strict reliability thresholds required for scientific research. The ai search story also surfaces in Stanford AI discovery identifies natural weight..., adding another angle.

The Failure of the "Judge" Instrument

The researchers treated the language model as a scientific instrument, setting strict, predefined thresholds for reliability before running any tests. They performed over 52,000 request attempts, ensuring that every request was byte-identical and that the execution environment was perfectly controlled. Despite this, the "judge" failed to provide consistent results. When asked to rank candidate responses, the model’s agreement with itself was significantly lower than the required standard. Even when the exact same request was replayed the next day, the results fluctuated, proving that a fixed model name does not guarantee a fixed, reliable measurement.

Why the Instability Occurs

The study identifies three primary mechanisms that cause this instability. First, the model’s internal "label-to-meaning" mapping can bias the output as much as the actual signal being measured. Second, the differences between candidate responses are often so small that they fall well below the instrument’s own "noise floor," making the judge’s output essentially random. Finally, because these models run on shared infrastructure, the underlying serving environment is not truly deterministic. Even when the input is identical, the way the system processes concurrent requests leads to different outputs, a problem that is amplified when the judge is asked to perform complex tasks like ranking multiple candidates. The same ai search question is explored in Measure Before You Manage, which adds a research perspective.

Testing the Limits of Reliability

To understand the scope of the problem, the researchers performed several follow-up experiments. They found that waiting for a different day did not improve consistency, and switching to different service providers yielded similar instability across the board. Even when they tried self-hosting the model, they found that performance only remained stable when the server was completely quiet; as soon as the load increased, the results became inconsistent again. Furthermore, the researchers discovered that the model does not act as a precise "magnitude meter"—it tracks the type of error in a response rather than the size of the error, making it an unreliable tool for measuring progress.

A New Discipline for AI Evaluation

The authors conclude that because shared inference endpoints are not frozen instruments, researchers must stop assuming that a model name is a constant. They propose a new "instrument-first" discipline for AI evaluation. Before using a model to gate training data or drive leaderboards, researchers should conduct a pilot study to measure the instrument's noise floor and calibrate its performance. The paper provides a set of design rules and a reporting checklist to help the research community move toward more rigorous, transparent, and reproducible evaluation practices, emphasizing that engineering perfection in the code does not compensate for a lack of reliability in the measurement tool itself. The same large language models question is explored in When Does Bigger Help? A Controlled..., which adds a research perspective. as detailed in the full paper on Arxiv