Back to AI Research

AI Research

Can LLMs Discover Scientific Laws in Real and Paral... | AI Research

Key Takeaways

  • Can LLMs Discover Scientific Laws in Real and Parallel Worlds?
  • This paper investigates whether Large Language Models (LLMs) can genuinely discover scientific...
  • Scientific equation discovery has long been central to scientific progress, proceeding through iterative cycles of hypothesis generation, observational testing, and refinement under scientific constraints.
  • As LLM capabilities advance and their role in AI for Science expands, it remains an open problem whether they can genuinely discover scientific laws and how this ability should be evaluated.
  • Existing evaluations, however, often either simplify discovery through synthetic settings or reuse published targets that may already be familiar to LLMs.
Paper AbstractExpand

Scientific equation discovery has long been central to scientific progress, proceeding through iterative cycles of hypothesis generation, observational testing, and refinement under scientific constraints. As LLM capabilities advance and their role in AI for Science expands, it remains an open problem whether they can genuinely discover scientific laws and how this ability should be evaluated. Existing evaluations, however, often either simplify discovery through synthetic settings or reuse published targets that may already be familiar to LLMs. We therefore introduce SCILAWS-BENCH, a benchmark for scientific law discovery built from published research and real scientific data. It comprises 118 problems drawn from 381 scientific papers, covering 291 candidate laws and roughly 8M real data points across six scientific disciplines. Each problem is instantiated in two complementary settings: (1) SCILAWS-REAL asks models to propose laws from fixed real observations and evaluates held-out predictive fit and scientific validity derived from the source literature, and (2) SCILAWS-PARALLEL asks models to actively query residual-calibrated worlds and recover synthesized hidden laws derived from published forms. This two-setting task design preserves each problem's scientific context while separately evaluating fixed-record law discovery and active recovery of a newly synthesized hidden law. We find that predictive fit can diverge from scientific validity, memorization shapes whether models reproduce or move beyond published formulas, and our best-of-N study reveals a selection bottleneck. Our work provides a paper-grounded benchmark and new empirical perspectives for evaluating AI for scientific discovery. Project page: this https URL

Can LLMs Discover Scientific Laws in Real and Parallel Worlds?
This paper investigates whether Large Language Models (LLMs) can genuinely discover scientific laws or if they are simply relying on memorized information from their training data. To test this, the authors introduce a new benchmark called SciLaws-Bench, which consists of 118 scientific problems derived from 381 research papers. By moving away from well-known textbook equations and focusing on active, data-driven research, the researchers aim to evaluate how well AI models can perform empirical induction—the process of deriving new scientific insights from raw, noisy observations.

A Two-Part Evaluation Strategy

To ensure a rigorous assessment, the researchers designed two distinct testing environments. The first, SciLaws-Real, provides models with fixed, real-world data and asks them to propose a mathematical law. This setup evaluates both how well the model’s formula fits the data and whether the proposed law is scientifically valid according to the context of the original research.
The second environment, SciLaws-Parallel, creates an active, queryable world. In this setting, the model starts without data and must decide which experiments to run to uncover a "hidden" law. This allows the researchers to see if the AI can recover a novel, synthesized structure rather than just repeating a formula it might have encountered during its training.

The Challenge of Memorization

A major hurdle in AI for science is distinguishing between true discovery and simple recall. The authors conducted a "cold-recall" audit to see if models could output the correct formulas before even looking at the data. They found that while models are very good at recalling "canonical" textbook equations, they struggle significantly with the "moat" of less-popularized, real-world research problems. This suggests that for many scientific tasks, the models are forced to engage in actual reasoning rather than relying on their internal database of known facts.

Key Findings and Limitations

The study reveals that while current frontier LLMs show promise, they are not yet reliable scientific discovery tools. Three primary issues stand out:

  • Fit vs. Validity: Models often prioritize mathematical "goodness of fit," sometimes proposing formulas that fit the data well but are physically impossible or scientifically nonsensical.

  • Structural Recovery: While models can reproduce known laws, they rarely succeed at discovering entirely new, complex structures. When they do move beyond published formulas, they often struggle to get the details right.

  • The Selection Bottleneck: Even when an AI generates several potential laws, it often fails to pick the best one. The models are better at generating candidates than they are at evaluating which candidate is the most scientifically sound.
    Ultimately, the research demonstrates that while LLMs are becoming capable assistants in scientific workflows, their ability to perform independent, high-level scientific discovery remains limited and requires further development.

Comments (0)

No comments yet

Be the first to share your thoughts!