Forecasting Scientific Progress with Artificial Intelligence

This paper investigates whether modern artificial intelligence can accurately predict the trajectory of scientific discovery. While AI is increasingly used to assist in research, its ability to act as a forecasting tool remains unproven. To address this, the authors introduce a new evaluation framework called CUSP (Cutoff-conditioned Unseen Scientific Progress), which tests AI models on their ability to assess feasibility, reason through scientific mechanisms, design solutions, and predict the timing of future breakthroughs.

The CUSP Benchmark

The researchers developed CUSP as a multi-disciplinary, event-level benchmark to rigorously evaluate AI performance. By analyzing 4,760 distinct scientific events, the framework challenges AI systems to look beyond simple pattern recognition and engage in complex forecasting tasks. The goal is to determine if these models can move past identifying plausible research directions to actually predicting whether a specific scientific advance will occur and when it will take place.

Key Findings on Predictive Capability

The study reveals that while frontier AI models are capable of identifying potentially viable research paths, they struggle significantly with the practical aspects of forecasting. Specifically, the models fail to reliably predict whether a scientific advance will be realized and consistently miscalculate the timing of these events. The researchers noted that performance is highly domain-dependent; for instance, the models were better at predicting the timing of progress in AI research compared to fields like biology, chemistry, and physics.

Limitations and Reliability

A critical discovery is that these forecasting limitations are not merely a result of the models' training data cutoffs. Performance remained largely insensitive to whether an event occurred before or after the model’s training cutoff, suggesting that simply having more data does not solve the problem. Furthermore, the models exhibited systematic overconfidence and strong response biases, which makes their uncertainty estimates unreliable.
Ultimately, the research concludes that current AI systems are not yet effective tools for predicting scientific progress. The models benefit more from having access to post-event information than from genuine forward-looking prediction, and even with additional pre-cutoff knowledge, they fail to reach the accuracy required for reliable scientific forecasting.