Messier: A High-Resolution Corpus for Cross-Benchma...

Messier: A High-Resolution Corpus for Cross-Benchma... | AI Research

Key Takeaways

Messier is a unified corpus designed to standardize the evaluation of AI agents across fragmented benchmarks.
Evaluating AI agents in interactive environments is hindered by fragmented tasks, scaffolds, verifiers, and scoring rules.
Existing efforts focus on narrow settings, remain limited in scale, or require costly reruns, leaving much of the empirical record incomparable.
We introduce Messier, a unified corpus of 957,253 records that span 30 benchmarks, 714 agents, 11,891 tasks, and 74,205 verifiers.
Messier consolidates public benchmark scores and supplements them with five-agent runs across six underrepresented professional and scientific domains, including a recent legal benchmark.

Paper AbstractExpand

Evaluating AI agents in interactive environments is hindered by fragmented tasks, scaffolds, verifiers, and scoring rules. Existing efforts focus on narrow settings, remain limited in scale, or require costly reruns, leaving much of the empirical record incomparable. We introduce Messier, a unified corpus of 957,253 records that span 30 benchmarks, 714 agents, 11,891 tasks, and 74,205 verifiers. Messier consolidates public benchmark scores and supplements them with five-agent runs across six underrepresented professional and scientific domains, including a recent legal benchmark. Each record is standardized by model, scaffold, environment, task, verifier, and aggregation rule, with SOC/NAICS classifications for occupational and industry analysis. Using this corpus, we show frontier progress is uneven across benchmark types, with "function calling" saturated, "programming" improving the fastest, and "enterprise workflows" remaining the most challenging. Furthermore, counterfactual rescoring shows that strict all-pass aggregation in multi-verifier tasks can obscure progress and artificially alter agent rankings. From these standardized records, we derive capability scales that align with Epoch's Evaluation Capability Index rankings at Spearman \r{ho} = 0.81 and can be specialized by domain, occupation, action space, or verifier type. Messier provides a foundational, reusable infrastructure for agent capability scaling, benchmark auditing, and fine-grained analysis of evaluation failures.

Messier is a unified corpus designed to standardize the evaluation of AI agents across fragmented benchmarks. By consolidating 957,253 records from 30 benchmarks, 714 agents, and 74,205 verifiers, the authors aim to solve the problem of incomparable results caused by inconsistent naming, scaffolding, and scoring rules in current agent research.

Standardizing Agent Evaluation

The authors address the "reconciliation problem" where benchmarks use different definitions for tasks, environments, and success conditions. Messier organizes these into a uniform data model that maps model identifiers, scaffolds, and release dates into standardized categories. Each record includes SOC (Standard Occupational Classification) and NAICS (North American Industry Classification System) codes to allow for industry-specific analysis. The corpus includes both existing public data and new evaluations conducted by the authors on six benchmarks, including legal review and clinical workflows, using a uniform five-agent grid.

The Impact of Aggregation Rules

The research indicates that how a benchmark calculates a final score can significantly alter the perceived capability of an agent. Many benchmarks use "all-pass" aggregation, where every individual verifier must succeed for the task to be marked as passed. The authors show that this method can report near-zero success rates even when an agent satisfies a majority of the individual criteria. By providing access to verifier-level outcomes rather than just final scores, Messier allows researchers to perform counterfactual rescoring to see how different aggregation rules change agent rankings.

Frontier Progress Across Domains

Using the corpus to track frontier performance between 2024 and 2026, the authors identify uneven progress across different types of tasks:

Function Calling: This domain is nearing saturation, with a frontier score of 0.97.
Programming: This area shows the fastest improvement, with an absolute gain of 0.64 over the two-year period.
Enterprise Workflows: This remains the most challenging domain, with the lowest performance at a 0.57 frontier score.

Capability Indexing

The authors derived capability scales from the standardized records that align with Epoch’s Evaluation Capability Index (ECI) rankings at a Spearman correlation of ρ = 0.81. This suggests that the Messier corpus can serve as a foundational infrastructure for auditing benchmarks and analyzing why specific evaluations fail. The dataset is designed to be extensible, allowing future researchers to add new benchmarks through a modular builder system without needing to modify the core consolidation logic.