Back to AI Research

AI Research

ExtractBench: A Benchmark for Schema-Guided Enterpr... | AI Research

Key Takeaways

  • ExtractBench is a benchmark designed to evaluate how well AI agents perform schema-guided document extraction in enterprise settings.
  • We present ExtractBench, a benchmark for schema-guided extraction and, to our knowledge, the first to score value accuracy, record completeness at scale, grounding, and measured cost together.
  • The evaluation system contains 4,869 pages across 370 enterprise documents, 8 business domains, and 67 document types, with clear tags differentiating their challenge scenarios.
  • The scalable schema and ground-truth curation pipeline combines independent-system agreement for real documents, known values for synthetic lists, and human verification for forms.
  • We report order-insensitive value F1 for value accuracy, plus two grounding metrics for source traceability: word- and page-level F1.
Paper AbstractExpand

Enterprise workflows increasingly rely on agents for \emph{schema-guided extraction}: given a document and a user-defined schema, the agent faithfully follows the schema to produce the correct output with source evidence as grounding metadata. We present ExtractBench, a benchmark for schema-guided extraction and, to our knowledge, the first to score value accuracy, record completeness at scale, grounding, and measured cost together. The evaluation system contains 4,869 pages across 370 enterprise documents, 8 business domains, and 67 document types, with clear tags differentiating their challenge scenarios. The scalable schema and ground-truth curation pipeline combines independent-system agreement for real documents, known values for synthetic lists, and human verification for forms. We report order-insensitive value F1 for value accuracy, plus two grounding metrics for source traceability: word- and page-level F1. Commercial VLMs perform well on short documents but often truncate record lists on long ones, while coding agents retain higher accuracy at much higher cost. LlamaExtract Agentic Plus ranks first on all three metrics, with accuracy comparable to coding agents at a fraction of the cost. Dataset and evaluation code are available on \href{ this https URL }{HuggingFace} and \href{ this https URL }{GitHub}.

ExtractBench is a benchmark designed to evaluate how well AI agents perform schema-guided document extraction in enterprise settings. The researchers aim to address the limitations of existing benchmarks by simultaneously measuring value accuracy, record completeness, visual grounding, and operational cost across a wide variety of real-world document types.

The Challenge of Enterprise Extraction

Enterprise workflows often require agents to extract specific data from documents based on a user-defined schema. While previous benchmarks have focused on fixed templates or narrow tasks, they often fail to account for the complexities of real-world business documents. These include long, multi-page tables, handwritten notes, scan artifacts, and the need for source evidence (grounding) to allow human verification. ExtractBench categorizes these difficulties into five independent axes: task challenges, perception challenges, table structure, document length, and business domain. By separating these variables, the benchmark allows for a more granular analysis of where and why an extraction system fails.

How the Benchmark Works

The dataset consists of 370 documents totaling 4,869 pages, covering 8 business domains and 67 document types. To ensure high-quality ground truth without relying on manual labeling for every field, the authors use a three-part curation pipeline:

  • Real documents: An ensemble of extraction systems proposes values, which are then adjudicated to ensure consensus.

  • Synthetic long lists: Documents are generated programmatically with known values and layouts, ensuring exact ground truth for complex, long-form data.

  • Scanned forms: Human annotators verify values and bounding boxes for documents containing handwriting or complex scan noise.
    The benchmark evaluates performance using order-insensitive value F1 for accuracy and two grounding metrics—word-level and page-level F1—to determine if the system can correctly trace extracted data back to its source on the page.

Performance and Cost Analysis

The researchers evaluated 14 different methods, including commercial vision-language models (VLMs), coding agents, and specialized APIs. The results indicate a significant performance gap between models, particularly regarding document length. For instance, while some commercial VLMs perform well on short documents, their accuracy drops sharply on long ones due to record truncation.
LlamaExtract Agentic Plus ranked first across all three primary metrics. The study notes that this agent achieved 94.4% accuracy on long documents, compared to 27.9% for Gemini 3.5 Flash. Furthermore, the authors highlight a cost-performance trade-off: LlamaExtract Agentic Plus outperformed Codex GPT-5.5 (95.6% vs. 93.6% overall accuracy) while operating at a lower cost of 8.1 cents per page compared to 27.8 cents per page.

Limitations and Considerations

The authors note that many commercial VLMs and coding agents do not return word-level bounding boxes, which limits their utility in workflows requiring strict source traceability. Additionally, the benchmark highlights that even high-performing systems struggle with specific structural challenges, such as "enormous" tables or tables packed within a single cell. The researchers emphasize that for enterprise-scale adoption, the cost per page is a critical factor, as even small differences in price can determine the financial viability of an AI-driven document workflow.

Comments (0)

No comments yet

Be the first to share your thoughts!