Back to AI Research

AI Research

Reconstruction: A Blind Benchmark for Recovering Re... | AI Research

Key Takeaways

  • Reconstruction is a blind benchmark designed to test whether large language models (LLMs) can accurately recover the core research idea of a published paper...
  • Can a language model recover the true research idea of a published paper when given only that paper's pre-publication bibliography?
  • A strict anti-leakage protocol-temporal citation cutoff, anonymous reference IDs, and frozen per-paper bibliographies, which prevents prompt-time leakage of the seed idea.
  • Across six scientific domains and 643 evaluated papers, seven frontier models achieve only modest Match rates (approx.
  • We then evaluate a reference-only multi-agent (top 4) pipeline that combines cross-model review with a Swiss tournament over aligned hypothesis slots, without external web search.
Paper AbstractExpand

Can a language model recover the true research idea of a published paper when given only that paper's pre-publication bibliography? We introduce Reconstruction, a blind idea-recovery benchmark that withholds the seed paper and all contemporaneous or future literature, and asks models to propose hypotheses that an independent large language model judge matches against the held-out ground-truth idea. A strict anti-leakage protocol-temporal citation cutoff, anonymous reference IDs, and frozen per-paper bibliographies, which prevents prompt-time leakage of the seed idea. Across six scientific domains and 643 evaluated papers, seven frontier models achieve only modest Match rates (approx. 3-15%). We then evaluate a reference-only multi-agent (top 4) pipeline that combines cross-model review with a Swiss tournament over aligned hypothesis slots, without external web search. Cross-model review plus tournament selection raises Match rates to approx. 23-42% across all six domains, which is an observed approx. 2.4x lift over the best single-model baseline. This draft reports the protocol, anti-leakage design, and current results as an arXiv timestamp.

Reconstruction is a blind benchmark designed to test whether large language models (LLMs) can accurately recover the core research idea of a published paper using only the references available before that paper was written. By withholding the seed paper and all subsequent literature, the benchmark evaluates a model's ability to synthesize existing knowledge to predict scientific outcomes.

The Reconstruction Task

The researchers, including Shaolong Chen, Ritankar Das, and colleagues, formalize the task by creating a "blind" context for each seed paper. This context contains only literature published before the seed paper’s release date. Models are tasked with generating five distinct hypotheses based on these references. An independent LLM judge then compares these generated hypotheses against the actual title and abstract of the held-out seed paper to determine if they match. The primary metric is the "Match rate," representing the percentage of hypotheses that successfully align with the ground-truth research idea.

Anti-Leakage Protocol

To ensure the validity of the results, the authors implemented a strict anti-leakage protocol. This includes:

  • Temporal Cutoff: Only literature published before the seed paper’s date is included.

  • Information Isolation: Models are prevented from accessing the seed paper or any contemporaneous or future research.

  • Anonymous References: Bibliography entries are provided as opaque IDs with title and abstract text, removing venue-specific shortcuts that might reveal the paper's identity.

  • Judge Recusal: To avoid bias, the judge model must be different from the model that generated the hypothesis.

Performance and Multi-Agent Results

The study evaluated seven frontier models across 643 papers in six scientific domains: Machine Learning, Astronomy, Chemistry, Materials, Medicine, and Physics. Single-model performance was modest, with Match rates typically ranging between 3% and 15%.
The researchers also tested a "reference-only multi-agent" pipeline using the top four performing models. This approach combined cross-model review with a Swiss-system tournament to select the best hypotheses. This pipeline significantly improved performance, raising Match rates to approximately 23–42% across all domains. This represents an observed 2.4x lift compared to the best single-model baseline.

Limitations and Considerations

The authors note that the multi-agent results are based on a post-selected ensemble of the top four models, rather than a selection rule validated on a held-out data split. Additionally, the study excludes papers that failed to yield five hypotheses during the generation phase, meaning the reported results reflect a specific subset of the initial collection. The researchers also observed that while the multi-agent pipeline improves accuracy, it also increases the dispersion of scores among judges, as different models may evaluate the higher-quality, more specific hypotheses differently.

Comments (0)

No comments yet

Be the first to share your thoughts!