Fisher-R1: Training LLM Agents for Reliable Hypothe...

Fisher-R1: Training LLM Agents for Reliable Hypothe... | AI Research

Key Takeaways

Fisher-R1 is an open-weight large language model (LLM) agent designed to perform reliable scientific hypothesis testing.
Reliable hypothesis testing is the foundation of many empirical scientific claims.
Large language model (LLM) agents are increasingly used to automate this process, as they can inspect datasets, generate code, and produce analyses end-to-end.
However, we show that they frequently make subtle inferential errors that lead to incorrect conclusions despite correctly executed analyses.
Existing benchmarks fail to capture this failure mode, as they rarely assess whether a reported p-value is statistically valid given the assumptions underlying the data.

Paper AbstractExpand

Reliable hypothesis testing is the foundation of many empirical scientific claims. Large language model (LLM) agents are increasingly used to automate this process, as they can inspect datasets, generate code, and produce analyses end-to-end. However, we show that they frequently make subtle inferential errors that lead to incorrect conclusions despite correctly executed analyses. Existing benchmarks fail to capture this failure mode, as they rarely assess whether a reported p-value is statistically valid given the assumptions underlying the data. We address this gap by building P-Bench, a benchmark comprising 425 open-ended, realistic hypothesis-testing tasks spanning economics, biology, and medicine. Each task requires an agent to select a statistical method, compute a p-value, and draw a conclusion given only a scientific hypothesis and a dataset. We further introduce Fisher-R1, an open-weight LLM agent trained for rigorous hypothesis testing using synthetic tasks and reinforcement learning. On P-Bench, Fisher-R1-14B substantially improves over its backbone and outperforms strong proprietary and open-source baselines, including GPT-5.4 and DeepSeekV4-Pro, achieving a 21% average relative improvement in single-trial success over DeepSeek-V4-Pro, with gains up to 26% on the most challenging tasks. Our results demonstrate that current LLM agents lack reliable statistical reasoning for hypothesis testing and that reinforcement learning on tasks with verified statistical reward substantially improves reliability.

Fisher-R1 is an open-weight large language model (LLM) agent designed to perform reliable scientific hypothesis testing. Researchers Jiacheng Miao, Jin Mu, Guanhua Chen, and James Zou developed this agent to address the tendency of current LLMs to produce technically valid code while making flawed inferential choices, such as selecting inappropriate statistical tests for a given dataset.

The Challenge of Automated Inference

While LLM agents can inspect data and write code, they often fail to perform rigorous hypothesis testing. The authors note that an agent might correctly identify data outliers but still proceed with a linear regression that produces a misleading result. Existing benchmarks for data analysis focus on code execution or factoid retrieval rather than the validity of the statistical method chosen or the accuracy of the resulting p-value. This creates a risk of "false discoveries" in high-stakes fields like medicine, biology, and economics.

P-Bench: A New Benchmark

To measure these inferential errors, the authors introduced P-Bench, a collection of 425 open-ended hypothesis-testing tasks. Each task includes a scientific question, a dataset, and a hidden answer key derived from canonical reference analyses. These tasks span various statistical methods—such as Cox regression, t-tests, and rank-based tests—and include adversarial data conditions like heteroskedasticity and clustered observations. The benchmark is designed to ensure that agents cannot succeed by relying on a single, fixed statistical recipe.

Training Fisher-R1

The researchers trained Fisher-R1 using a two-stage process: 1. Supervised Fine-tuning (SFT): The model was trained on 3,851 expert-generated trajectories that follow a structured workflow: exploratory data analysis, assumption checking, method selection, and conclusion drafting. 2. Reinforcement Learning (RL): The model was further optimized using synthetic tasks with verified answer keys. The RL process used a reward function that evaluates the accuracy of the reported p-value and the consistency of the final decision (reject or fail-to-reject the null hypothesis).

Performance Results

On P-Bench, the 14B-parameter version of Fisher-R1 outperformed several proprietary and open-source models, including GPT-5.4 and DeepSeek-V4-Pro. According to the paper, Fisher-R1-14B achieved a 21% average relative improvement in single-trial success over DeepSeek-V4-Pro, with performance gains reaching 26% on the most challenging tasks. These results suggest that reinforcement learning on tasks with verified statistical rewards is an effective way to improve the reliability of LLM agents in scientific reasoning.