Back to AI Research

AI Research

ContractScrub: A benchmark for final review of lega... | AI Research

Key Takeaways

  • ContractScrub is a new benchmark designed to evaluate how well Large Language Models (LLMs) perform "contract scrubbing"—the final, meticulous review of lega...
  • Legal work, with its heavy reliance on processing large amounts of text, is often considered one of the domains most exposed to the use of LLMs.
  • Scrubbing also seems to align naturally with the general capabilities expected of frontier LLMs around long-context reasoning, consistency checking, and named entity recognition (NER).
  • Despite the economic value and potential for automation, no formal evaluations of LLMs performing contract scrubbing have been conducted.
  • ContractScrub is a new benchmark designed to evaluate how well Large Language Models (LLMs) perform "contract scrubbing"—the final, meticulous review of legal agreements for errors and inconsistencies.
Paper AbstractExpand

Legal work, with its heavy reliance on processing large amounts of text, is often considered one of the domains most exposed to the use of LLMs. Contract ``scrubbing,'' the final review of transactional agreements for errors and inconsistencies, is a particularly suitable task for automation, because it is routine, painstaking work requiring detailed attention to long documents. Scrubbing also seems to align naturally with the general capabilities expected of frontier LLMs around long-context reasoning, consistency checking, and named entity recognition (NER). Despite the economic value and potential for automation, no formal evaluations of LLMs performing contract scrubbing have been conducted. We introduce ContractScrub, the first benchmark designed to evaluate contract scrubbing capabilities, comprising contracts hand-crafted by experienced lawyers over diverse error categories such as misuse of defined terms, incorrect references, and inconsistent language. Frontier models perform surprisingly poorly with only one model reaching 0.75 macro average recall despite strong performance on seemingly related general benchmarks, demonstrating the practical limits of current models and the importance of narrowly targeted, domain-specific benchmarks for measuring real-world impact.

ContractScrub is a new benchmark designed to evaluate how well Large Language Models (LLMs) perform "contract scrubbing"—the final, meticulous review of legal agreements for errors and inconsistencies. While LLMs are frequently used for legal text processing, this research identifies a gap in how these models perform on the routine, high-stakes task of identifying drafting defects before a contract is finalized.

The Task of Contract Scrubbing

Contract scrubbing is a repetitive but essential legal process where practitioners check for issues like misused defined terms, incorrect cross-references, and inconsistent language. These errors are often subtle and embedded in long documents. The authors, including researchers from Thomson Reuters and Imperial College London, argue that while this task aligns with the general capabilities of frontier LLMs—such as long-context reasoning and named entity recognition—there has been no formal evaluation of how these models perform in a real-world legal setting.

How the Benchmark Works

The researchers constructed the ContractScrub dataset using 44 contracts sourced from the CUAD dataset. To ensure high-quality, realistic testing, the benchmark was developed through a three-stage pipeline: 1. Selection: Contracts were screened for structural integrity and length (typically 10–15 pages). 2. Annotation: Experienced lawyers with at least 8 years of practice identified existing drafting errors. 3. Augmentation: The lawyers purposefully inserted additional, realistic drafting errors to create a "gold standard" for evaluation.
The final dataset contains 3,014 annotated tasks across nine specific categories, such as "Undefined Capitalized Terms" and "Incorrect Section References." Models are evaluated on their ability to identify these defects, with a focus on recall, as missing a legal error is generally more costly than flagging a potential false positive.

Performance Results

The study evaluated nine frontier and open-weight models, including versions of GPT, Claude, Qwen, and Gemini. The results indicate that contract scrubbing is significantly more difficult for LLMs than their performance on general-purpose benchmarks might suggest.
The best-performing model achieved a macro-average recall of 0.750, and all models recorded F1 scores below 0.650. Performance was inconsistent across categories: models were relatively proficient at identifying explicit lexical signals, such as defined terms, but struggled significantly with categories requiring the inference of intent within a specific context, such as "Incorrect Capitalization in Context."

Why Domain-Specific Benchmarking Matters

The authors conclude that general-purpose legal benchmarks often test narrow skills in isolation, failing to capture the complexity of reviewing an entire, long-form document. The findings suggest that even advanced models fall short of the performance expected based on their general capabilities when applied to specialized, high-precision tasks. This highlights the necessity of using narrowly targeted, domain-specific benchmarks to measure the real-world impact of AI in professional fields like law.

Comments (0)

No comments yet

Be the first to share your thoughts!