Back to AI Research

AI Research

FrontierPhysics evaluates agents across the full physics research workflow

Key Takeaways

  • Expert-authored tasks test literature review, research execution and manuscript quality, revealing limits beyond isolated question answering.
  • Answering a physics question is different from carrying a research project through literature review, experimentation and a defensible manuscript.
  • [FrontierPhysics](https://arxiv.org/abs/2611.06910) is designed to evaluate that longer process.
  • The benchmark contains 56 tasks developed by 42 physicists across nine research domains.
  • An agent must review relevant literature, develop a plan, conduct the required research and produce structured results and a scientific manuscript.

Answering a physics question is different from carrying a research project through literature review, experimentation and a defensible manuscript. FrontierPhysics is designed to evaluate that longer process.

What each task asks an agent to do

The benchmark contains 56 tasks developed by 42 physicists across nine research domains. An agent must review relevant literature, develop a plan, conduct the required research and produce structured results and a scientific manuscript.
The evaluation combines deterministic verifiers with expert-authored rubrics. That gives it a way to examine both concrete correctness requirements and qualities of research that a single exact-answer check would miss.
The paper distinguishes these tasks from reproducing a paper when the solution structure is already supplied. Its aim is to test more of the decisions that make an open-ended research workflow difficult.

What the evaluated systems could complete

The authors evaluate 22 models in 26 configurations. No configuration passes more than 46 of its 168 rollouts in the reported evaluation.
A rollout is one attempt under a particular agent configuration; that denominator should remain attached to the result. It is not a count of distinct physics problems solved by a model outside this benchmark.
Manual trajectory review identifies insufficiently rigorous reasoning and limited expert intuition as major weaknesses. The researchers distinguish those issues from simply lacking physics facts or failing to execute software.

Reading the result as a research tool

For a scientist considering an agent assistant, the useful question is which stages can be delegated while preserving an expert’s ability to inspect assumptions, calculations and conclusions. A polished manuscript is only one output of that process.
This preprint offers a demanding evaluation setting and documented failure categories. It does not establish that an agent is ready to conduct independent physics research safely or that a successful benchmark attempt will generalize to a new research question.

Comments