Back to AI Research

AI Research

EarthVerse: Benchmarking Scientific Agents Across D... | AI Research

Key Takeaways

  • EarthVerse is a benchmark designed to evaluate how well scientific AI agents can conduct complex, multi-source investigations into Earth-system events and na...
  • Earth-system analysis reconstructs changing physical processes from observations that differ in source, scale, timing, and modality.
  • Natural hazards make this work consequential because incomplete evidence can change estimates of severity, exposure, and mechanism.
  • We introduce EarthVerse, a benchmark that evaluates scientific agents through package-scoped investigations.
  • Its 405 reproducible tasks are grounded in 199 documented events and 19 hazard families.
Paper AbstractExpand

Earth-system analysis reconstructs changing physical processes from observations that differ in source, scale, timing, and modality. Natural hazards make this work consequential because incomplete evidence can change estimates of severity, exposure, and mechanism. We introduce EarthVerse, a benchmark that evaluates scientific agents through package-scoped investigations. Its 405 reproducible tasks are grounded in 199 documented events and 19 hazard families. Agents inspect heterogeneous event packages, choose compatible evidence, execute transparent calculations, reconcile source differences, and preserve provenance in the final answer. We provide executable ground truth that decomposes each task into fine-grained answer units, together with task-specific rubrics that assess the supporting research process while allowing multiple valid paths. We evaluate 25 model and agent systems under a controlled tool-using protocol, then use controlled studies to locate failures in evidence access, tool selection, memory, reasoning, interaction, and scientific execution. Across systems, the best mean answer-unit accuracy is 84.65%, while the highest Strict@95 is only 34.81%. The gap shows that current agents often complete individual steps without maintaining a consistent chain across evidence, scales, units, calculations, and physical interpretation. EarthVerse provides a reproducible basis for measuring end-to-end scientific reliability in dynamic Earth systems.

EarthVerse is a benchmark designed to evaluate how well scientific AI agents can conduct complex, multi-source investigations into Earth-system events and natural hazards. The researchers behind EarthVerse aim to move beyond simple question-answering tasks by testing whether agents can gather, reconcile, and synthesize heterogeneous data—such as satellite imagery, station records, and impact reports—to form a scientifically reliable and auditable conclusion. Methods and results are detailed in the full paper on arxiv.org.

The Challenge of Scientific Reliability

Current Earth-science benchmarks often provide agents with pre-selected data, testing only their ability to interpret a specific image or passage. EarthVerse changes this by requiring agents to perform "package-scoped investigations." For each of the 405 tasks, an agent is given a collection of files related to a specific disaster—such as a heat wave or flood—and must determine which sources are relevant, reconcile conflicting data, and maintain a consistent chain of evidence throughout their analysis. The researchers note that a single error in scale, unit, or time window can invalidate an entire scientific conclusion, even if individual calculations are correct. The same AI Evaluation question is explored in CAFE, which adds a research perspective.

How the Benchmark Works The benchmark is grounded in 199 documented disasters and extreme events across 19 hazard families. Each task requires the agent to:

  • Identify evidence: Select relevant files from a package that typically contains about 34 candidates.

  • Execute calculations: Perform transparent operations like spatial overlaps, ratios, and thresholding.

  • Maintain provenance: Ensure that every claim is supported by the chosen evidence and that the research process is auditable.
    To score performance, the researchers provide "executable ground truth" and task-specific rubrics. They evaluate agents based on both the accuracy of their final answers and the quality of their research process, ensuring that multiple valid paths to a solution are accepted as long as they are scientifically sound. The AI Search story also surfaces in EU Regulators Demand Apple and Google..., adding another angle. The same AI Evaluation question is explored in SciMIF, which adds a research perspective.

Performance and Reliability Gaps In an evaluation of 25 model and agent systems,

the researchers identified a significant "reliability gap." While the best-performing systems achieved a mean answer-unit accuracy of 84.65%, the highest "Strict@95" score—which measures the percentage of tasks completed with near-total accuracy—was only 34.81%. This indicates that while current agents are often capable of completing individual steps of a research task, they frequently struggle to maintain a consistent, error-free chain of reasoning across an entire investigation.

Why This Matters Earth-system analysis is essential for disaster response and climate risk assessment, where accurate estimates of severity and exposure are critical. The researchers argue that for AI to be useful in these fields, it must be capable of more than just isolated reasoning; it must be able to construct a coherent evidence base from messy, real-world data. EarthVerse provides a reproducible framework to measure this end-to-end reliability, helping to identify specific failure points in how agents access evidence, select tools, and manage long-horizon scientific workflows.

More from Franklin

CAFEresearchSciMIFresearchOntoAligner-EnsembleresearchEU Regulators Demand Apple and Google...newsEgo LitetoolUnibase memorytool

Comments