Back to AI Research

AI Research

Ideas Have Genomes: Benchmarking Scientific Lineage... | AI Research

Key Takeaways

  • Scientific ideas do not emerge in a vacuum; they are built upon the foundations of previous research, inheriting mechanisms and addressing limitations in a p...
  • Scientific ideas rarely start from a blank page.
  • They inherit mechanisms, repair known limitations, and recombine pieces of earlier work, much like biological genomes.
  • Current benchmarks still say little about whether AI systems can follow this inheritance structure.
  • We present IdeaGene-Bench (IG-Bench), a benchmark for scientific lineage reasoning and lineage-grounded idea generation.
Paper AbstractExpand

Scientific ideas rarely start from a blank page. They inherit mechanisms, repair known limitations, and recombine pieces of earlier work, much like biological genomes. Current benchmarks still say little about whether AI systems can follow this inheritance structure. We present IdeaGene-Bench (IG-Bench), a benchmark for scientific lineage reasoning and lineage-grounded idea generation. IG-Bench is organized around the IdeaGene framework: each paper or proposal is represented as a set of minimal, typed, evidence-grounded Idea Genome objects, and a GenomeDiff aligns these objects to record inheritance, mutation, loss, external import, and novel insertion under six operational evolutionary dynamics. The benchmark contains 1,961 golden lineage traces, 1,085 curated Idea Genome objects, and 920 pairwise GenomeDiff records across 10 scientific domains. It supports two evaluations. IG-Exam (42 task types, 1,029 instances) tests closed-form lineage reasoning across Idea Genome abstraction, inheritance tracing, evolutionary reasoning, and lineage verification. IG-Arena evaluates generation with a lineage-conditioned Population-Evolution Score(PES), asking whether a proposal can be inserted as a coherent descendant of a given lineage population: it should inherit the right Idea Genome objects, vary meaningfully from nearby work, and offer selection value for future research. Experiments on 14 LLM-based scientists expose a compositional bottleneck. The strongest system reaches only 27.3% exact accuracy on lineage reasoning, and structured lineage context reshuffles system rankings rather than helping every participant uniformly.

Scientific ideas do not emerge in a vacuum; they are built upon the foundations of previous research, inheriting mechanisms and addressing limitations in a process that mirrors biological evolution. The paper Ideas Have Genomes: Benchmarking Scientific Lineage Reasoning and Lineage-Grounded Idea Generation introduces a new framework and benchmark, IG-Bench, to evaluate whether AI systems can understand and replicate this complex lineage structure.

The IdeaGene Framework

To track how scientific ideas evolve, the researchers developed the IdeaGene framework. This approach treats scientific papers and proposals as sets of "Idea Genome" objects—minimal, typed, and evidence-based components. By using a tool called GenomeDiff, the researchers can map how these objects change over time. This includes tracking inheritance, mutations, the loss of old concepts, the import of external ideas, and the insertion of entirely new ones. By organizing research into these evolutionary dynamics, the framework provides a structured way to analyze the history and development of scientific thought across 10 different domains.

Evaluating AI Reasoning and Generation

The benchmark supports two primary types of evaluation:

  • IG-Exam: This tests an AI’s ability to perform "closed-form" reasoning. It uses over 1,000 instances to see if a model can correctly identify how ideas are inherited, verify lineage, and understand the evolutionary steps between different scientific works.

  • IG-Arena: This evaluates the creative side of AI. Using a "Population-Evolution Score" (PES), it measures whether a generated research proposal acts as a coherent descendant of existing work. A successful proposal must inherit the right foundational ideas, show meaningful variation from previous research, and offer genuine value for future scientific inquiry.

The Compositional Bottleneck

When testing 14 different LLM-based "scientists" against these benchmarks, the researchers discovered a significant performance gap. The most capable system achieved only 27.3% exact accuracy on lineage reasoning tasks. Furthermore, the study found that providing structured lineage context did not improve performance uniformly across all models; instead, it tended to reshuffle the rankings of the systems. This suggests that current AI models struggle with the compositional nature of scientific progress, finding it difficult to synthesize and evolve ideas in the same way human researchers do.

Comments (0)

No comments yet

Be the first to share your thoughts!