Bayesian Inference and Decision Audits for Public A... | AI Research

Bayesian Inference and Decision Audits for Public Archives of Frontier AI Evaluations
This paper addresses the problem that current AI leaderboards are often treated as definitive, final rankings, even though they are actually selective, time-indexed snapshots. Because these leaderboards often hide unpublished or under-performing systems, they provide a "lossy" view of AI progress. The author proposes a new framework that treats these public evaluation histories as a Bayesian inference problem, using an archive-and-adjudication protocol to reconstruct the actual history of AI performance and test whether frontier claims hold up under rigorous audit.

Reconstructing Evaluation Histories

The paper argues that a terminal ranking is not the same as a longitudinal history. To understand how AI capabilities truly evolve, the author reconstructs "evaluation-trace archives" from sources like LiveBench and the Open LLM Leaderboard v2. By collecting repeated snapshots—including metadata like reporting rules, benchmark versions, and system identities—the research creates a more complete picture of performance over time. This approach allows researchers to distinguish between genuine technological progress and artifacts caused by how results are reported or selected.

The Observability Boundary

A core challenge identified in the paper is the "observability boundary." The author demonstrates that terminal-only records are often insufficient to determine how quickly an AI system reaches a specific performance ceiling. By constructing a synthetic example, the paper shows that a single terminal leaderboard result can be compatible with multiple, vastly different historical paths. For instance, one model might reach a performance threshold in 23 steps, while another reaches it in 75 steps, even if they end up at the same final score. This proves that without access to the full history of snapshots, it is impossible to accurately infer the timing and trajectory of AI development.

Decision Audits and Model Testing

Rather than relying on traditional significance testing, the paper uses a decision-theoretic approach. It frames the evaluation of AI models as a series of potential actions—such as "continue," "refresh," or "harden"—that a decision-maker might take based on the available evidence. By applying this to a "selection-aware" frontier model, the author tests whether the model can accurately predict future observations, calibrate uncertainty, and transfer preferences. The results indicate that the current candidate model fails these criteria, demonstrating that the proposed audit protocol is effective at rejecting unsupported claims that might otherwise be accepted if viewed only through a standard, terminal leaderboard.

Limitations and Scope

The author emphasizes that this protocol is not a universal substitute for all statistical testing. The findings are bounded by the specific parameterizations and loss structures used in the study. While the framework provides a robust way to audit AI claims, the paper notes that prior-sensitivity analysis and broader model robustness checks remain outside the current scope. The primary goal is to establish a reproducible, auditable standard for evaluating AI progress that moves beyond the limitations of static, terminal-only reporting.