AutoSynthesis: An agentic system for automated meta... | AI Research

AutoSynthesis is an end-to-end multi-agent system designed to automate the complex, time-consuming process of conducting a meta-analysis. While meta-analyses are essential for turning individual research studies into reliable, evidence-based knowledge for fields like medicine, education, and policy, they are traditionally manual, expensive, and difficult to scale. AutoSynthesis addresses this by using AI agents to perform the entire workflow—from formulating search strategies and screening literature to extracting quantitative data and performing statistical synthesis—resulting in a transparent report that follows established PRISMA guidelines.

How the System Works

The framework operates as a multi-agent system that mimics the steps a human researcher would take. It begins by taking a research question in natural language and using LLM-generated queries to retrieve relevant literature from scientific databases. The system then screens these papers for eligibility, extracts specific quantitative data, and calculates standardized effect sizes (such as Cohen’s d or Hedges’ g). Beyond simple synthesis, the system performs random-effects meta-analysis, assesses between-study heterogeneity, conducts publication bias diagnostics, and evaluates the risk of bias in individual studies. Throughout the process, the system maintains a record of its decisions, allowing users to audit the reasoning behind every inclusion or exclusion.

Key Results and Performance

To test the system, the researchers tasked AutoSynthesis with analyzing the persuasive power of large language models, comparing its output against an expert-conducted meta-analysis. The system successfully retrieved and screened dozens of records, ultimately identifying a set of eligible studies that closely matched the human benchmark. The pooled effect estimates produced by AutoSynthesis were within a narrow margin of the expert-conducted results, and the system successfully replicated key statistical findings, such as the presence of substantial between-study heterogeneity and evidence of small-study effects.

The Role of Human Expertise

The authors emphasize that AutoSynthesis is not intended to replace human researchers. Instead, it is designed to act as a powerful tool to support evidence-based decision-making. Human judgment remains critical for defining research questions, setting inclusion criteria, and interpreting the final results within a broader context. The system is particularly well-suited for "living" meta-analyses, where it can continuously integrate new evidence as it is published, significantly reducing the manual labor required to keep scientific reviews up to date. By automating the more repetitive aspects of the workflow, AutoSynthesis helps researchers maintain methodological rigor and provides a scalable way to synthesize evidence in fields where timely information is vital.