Can We Trust Item Response Theory for AI Evaluation? | AI Research

Can We Trust Item Response Theory for AI Evaluation?
This paper investigates whether Item Response Theory (IRT)—a statistical framework traditionally used for human educational testing—is reliable when applied to AI benchmarks. While IRT is increasingly used to rank AI models and diagnose benchmark quality, the data used in AI evaluation differs significantly from human testing. Specifically, AI benchmarks often feature a small number of models evaluated against a massive number of items, with capability distributions that are frequently skewed or clustered. This study systematically tests whether these differences lead to biased or unreliable results when using common IRT estimation tools.

The Mismatch Between Human and AI Testing

In traditional psychometrics, IRT models are designed for scenarios with many test-takers and a limited number of questions. AI benchmarking often inverts this, evaluating a small set of models on thousands of items. Furthermore, standard IRT models typically assume that human ability follows a normal distribution (a bell curve). AI models, however, often show non-normal, multimodal, or skewed performance patterns. The authors argue that these fundamental differences can violate the statistical assumptions required for IRT to function accurately, potentially leading to distorted rankings and flawed diagnostics.

Testing the Estimators

To evaluate the reliability of IRT in this new context, the researchers conducted a massive simulation study covering 18,000 different conditions. They used data from six popular LLM benchmarks to simulate response matrices and tested four common estimation methods: Marginal Maximum Likelihood (MML-EM), Markov chain Monte Carlo (MCMC), Variational Inference (VI), and a neural pseudo-Siamese estimator (PSN). By comparing the results of these estimators against known "ground truth" parameters, the team measured how well each method recovered model rankings, predicted performance, and identified item characteristics.

Key Findings on Reliability

The study reveals a trade-off between computational feasibility and accuracy. Classical estimators, such as MML-EM, often struggle to remain computationally feasible as the number of benchmark items grows. Conversely, more scalable methods—like VI and neural estimators—can produce unreliable results when the number of models is small or when the underlying capability distribution is not normally distributed. The findings suggest that simply applying standard IRT tools to AI benchmarks without careful validation can lead to misleading conclusions about model capabilities and the quality of the benchmarks themselves.

Practical Guidance for Researchers

The paper concludes by providing a roadmap for the trustworthy use of IRT in AI evaluation. It emphasizes that researchers must be cautious about the sample size of models being evaluated and the specific distribution of their capabilities. The authors suggest that before relying on IRT-based claims, practitioners should perform diagnostics to ensure their chosen model and estimation method are appropriate for their specific data regime. By identifying the conditions under which IRT models fail or succeed, this study provides the necessary criteria for researchers to determine when these statistical tools will genuinely improve AI benchmarking and when they might introduce unintended bias.