Back to AI Research

AI Research

CorporateBench: Large-Scale Q&A Benchmarking with T... | AI Research

Key Takeaways

  • CorporateBench (CB) is a human-validated, multi-task benchmark designed to evaluate how well Large Language Models (LLMs) reason over enterprise-scale docume...
  • LLMs are increasingly able to answer complex questions about enterprise-scale document collections.
  • But evaluation is hard: companies don't want to share internal communications, and synthetic datasets have been overly simple.
  • We present CorporateBench (CB), a human-validated multi-task Q&A benchmark whose scale approaches the conditions LLMs encounter in corporate communication networks, with evaluation corpora surpassing 230,000 documents.
  • CB evaluates LLMs across two dimensions (information extraction and knowledge base querying) through four synthetically generated firms ranging from 12 to 10,000 employees.
Paper AbstractExpand

LLMs are increasingly able to answer complex questions about enterprise-scale document collections. But evaluation is hard: companies don't want to share internal communications, and synthetic datasets have been overly simple. We present CorporateBench (CB), a human-validated multi-task Q&A benchmark whose scale approaches the conditions LLMs encounter in corporate communication networks, with evaluation corpora surpassing 230,000 documents. CB evaluates LLMs across two dimensions (information extraction and knowledge base querying) through four synthetically generated firms ranging from 12 to 10,000 employees. Each corpus is sampled from a temporally evolving knowledge base describing a consistent world, guaranteeing cross-document logical consistency even across hundreds of thousands of documents. We evaluate five LLMs on CB, revealing increasingly poor performance as input size approaches realistic scales. CB provides LLM developers a metric for corporate communication reasoning, filling a crucial gap in the benchmarking ecosystem.

CorporateBench (CB) is a human-validated, multi-task benchmark designed to evaluate how well Large Language Models (LLMs) reason over enterprise-scale document collections. By simulating complex, temporally evolving corporate environments, the benchmark provides a way to test model performance on tasks that require synthesizing information across thousands of documents, addressing the lack of realistic, non-proprietary datasets for enterprise AI.

Simulating Corporate Complexity

To overcome the limitations of synthetic datasets—which are often too simple—and real-world data—which is restricted by non-disclosure agreements—the authors created a procedural generation pipeline. This pipeline builds a "knowledge base" (KB) for four synthetic companies ranging from 12 to 10,000 employees. These KBs define organizational hierarchies, project assignments, and meeting schedules. From these foundations, the researchers generated over 230,000 documents, including emails that maintain logical consistency with the underlying KB. This allows for deterministic ground truth, as the answers to questions can be verified directly against the original graph.

Evaluating Long-Context Reasoning

The benchmark tests models across two primary dimensions: information extraction and knowledge base querying. Extraction tasks require models to reconstruct structured data from raw text, while QA tasks evaluate reasoning over that data. The researchers tested five LLMs on these tasks, finding that model performance consistently degrades as the input size increases toward realistic enterprise scales. The benchmark achieves a ratio of 87.6 documents per question, a significant increase in complexity compared to previous benchmarks, which often rely on fewer documents per query.

Task Structure and Performance

CorporateBench organizes its evaluation into five specific tasks:

  • Extraction: Models must recover relations and classify topics from the document corpora.

  • KB QA: Models answer factual questions about the company, such as identifying employees who began working on specific projects after a certain date.

  • Topic QA: Models determine if specific employees discussed certain topics in their communications.

  • Integrated QA: Models synthesize information across multiple documents to identify specific workers or project statuses.
    The results indicate that while models can perform well on smaller datasets, their ability to maintain precision and recall in extraction and reasoning tasks drops as the company size and document volume grow to the "Extra Large" (10,000 employee) scale.

Limitations and Considerations

The authors note that while their simulation mimics real-world corporate structures, it is still a synthetic environment. The workload simulation, for instance, limits employees to 1 to 3 tasks per quarter to ensure uniform relationship counts, which is less dense than a typical real-world workload. Additionally, while the benchmark provides a metric for corporate reasoning, the researchers emphasize that a model might successfully answer a query while failing to correctly build the underlying knowledge base during the extraction phase. The dataset and code are released to provide a reproducible standard for developers to measure how well their models handle the interconnectedness of enterprise communication.

Comments (0)

No comments yet

Be the first to share your thoughts!