Back to AI Research

AI Research

WorldCupArena: Fine-Grained Evaluation of Language... | AI Research

Key Takeaways

  • WorldCupArena is a dynamic benchmark designed to evaluate how well language models and deep-research agents can forecast future events.
  • Predicting a football match before kickoff requires more than knowing past results: a model must use changing information and make a clear prediction before the answer is available.
  • We present WorldCupArena, a dynamic benchmark for language models and deep-research agents.
  • The 2026 FIFA World Cup is its first evaluation, and the same process can be reused for future leagues and cups.
  • Before each match, a model either receives a common evidence package or searches for information itself.
Paper AbstractExpand

Predicting a football match before kickoff requires more than knowing past results: a model must use changing information and make a clear prediction before the answer is available. We present WorldCupArena, a dynamic benchmark for language models and deep-research agents. The 2026 FIFA World Cup is its first evaluation, and the same process can be reused for future leagues and cups. Before each match, a model either receives a common evidence package or searches for information itself. It predicts the result and score, likely players and events, match statistics, and the outcome of the competition. After the match, these predictions are compared with the recorded result. We report result accuracy, exact-score accuracy, and a scoreline score that gives some credit when a predicted score is close but not exact, together with scores for the other prediction tasks. Across 104 matches and 13 systems, models with similar result accuracy differ more clearly on detailed predictions. Compared with betting-market and human-fan baselines, the best system shows only small gains in result and exact-score accuracy, but a clearer gain in Scoreline. New schedules can be added as they begin, allowing the benchmark to evaluate future models without using outcomes that are already known. Code, prompts, predictions, and evaluation scripts are open sourced at this https URL .

WorldCupArena is a dynamic benchmark designed to evaluate how well language models and deep-research agents can forecast future events. Unlike traditional benchmarks that test models on questions with already-known answers, this project focuses on football matches, requiring systems to analyze changing information and make predictions before a game begins. By evaluating models on the 2026 FIFA World Cup, the researchers created a framework that tests not just the final result, but also detailed predictions like scores, player lineups, match statistics, and the progression of the entire tournament.

How the Benchmark Works

The evaluation process follows a strict protocol for every match. Twenty-four hours before kickoff, models are provided with a standard evidence package (such as team form and news) or are tasked with searching for their own information. They then submit a comprehensive forecast, including win probabilities, expected scores, and specific match events. Once the match concludes, these predictions are compared against official records. The benchmark uses five distinct layers of evaluation: core match results, player lineups, specific events (like goals or cards), tactical statistics, and the overall outcome of the competition.

Evaluating Model Performance

The study tested 13 different systems across all 104 matches of the 2026 World Cup. A key finding is that looking only at "result accuracy"—who won or lost—is insufficient for distinguishing between high-performing models. While many models achieved similar success rates in predicting the winner, they showed significant differences in their ability to predict precise scorelines and detailed match statistics. The researchers also implemented a "Scoreline" metric, which provides partial credit for close misses, offering a more nuanced view of a model's predictive capability than a simple right-or-wrong score.

Search and Baseline Comparisons

One of the study's goals was to determine if allowing models to search the web for real-time information improves their forecasting accuracy. Interestingly, the results showed no consistent performance gain from adding search capabilities compared to models provided with a standardized evidence package. Furthermore, when compared to betting markets and human-fan predictions, the best-performing AI systems showed only minor improvements in result accuracy. However, these AI models did demonstrate a clearer advantage in the "Scoreline" category, suggesting they are better at narrowing down the range of likely outcomes even when they do not predict the exact final score.

Future Applications

Because the benchmark is designed to be modular, it is not limited to the 2026 World Cup. The same evaluation structure can be applied to future leagues, cups, or other sports, allowing researchers to continuously test new models as they are released. By publishing the code, prompts, and evaluation scripts, the authors have created a reusable tool that ensures future models can be judged on their ability to handle real-world uncertainty rather than relying on static, historical data.

Comments (0)

No comments yet

Be the first to share your thoughts!