InsufficiencyBench: Evaluating LLM legal advice on... | AI Research

InsufficiencyBench is a new legal benchmark designed to evaluate whether Large Language Models (LLMs) can recognize when a user’s legal query is missing critical information. Rather than assuming a query is fully specified, this benchmark tests if a model can identify missing facts, ask for clarification, and avoid providing premature legal conclusions based on unstated assumptions.

The Problem of Premature Legal Closure

Current legal AI benchmarks typically provide models with fully formed, complete scenarios. In real-world practice, however, users often omit facts that are legally material to the outcome. When LLMs are presented with these underspecified queries, they often engage in "premature legal closure." This occurs when a model silently presumes missing information—such as the user's jurisdiction or employer size—and provides a confident, fluent answer that may be legally incorrect for the user's actual situation. The authors note that this behavior is distinct from hallucination, as the model may state the law correctly for its own assumed facts while giving dangerous advice for the user's reality.

How InsufficiencyBench Works

The benchmark consists of 202 items, including 58 base queries and 144 deficient variants, across six legal domains and 24 US jurisdictions. Legal experts annotated these queries to identify "required elements"—facts that must be known to provide a safe and accurate legal response.
The researchers established a taxonomy of eight canonical categories of missing information, organized by how they disrupt legal reasoning:

• Switch: Missing elements that change the governing legal framework (e.g., jurisdiction).

• Gating: Missing elements that determine if a framework applies at all (e.g., employer size).

• Fatal Prerequisite: Missing elements that prevent a claim from proceeding within an otherwise applicable framework (e.g., being current on rent for a habitability claim).
  To test a model, the researchers present a deficient query and evaluate whether the model detects the missing elements, explains why they are necessary, and refrains from offering a final conclusion until those gaps are filled.

Performance of Frontier Models

The authors evaluated ten frontier models using InsufficiencyBench and found significant gaps in performance. No model achieved an F2 score higher than 0.46 on the task of identifying missing elements, and the median recall was 0.44. The study observed that models generally fall into two suboptimal behaviors: they either hedge their answers indiscriminately or they answer silently under fabricated presumptions. The researchers concluded that no current model successfully balances identifying missing information with providing direct, accurate answers when a query is complete.

Franklin Analysis

The evidence suggests that current LLM training, which heavily incentivizes providing an answer, is fundamentally at odds with the requirements of legal intake. Because legal materiality is defined by rigid structures like statutes and doctrines rather than user preference, the failure to recognize missing information creates a high risk of providing incorrect legal advice. The benchmark provides a clear metric for future research to move away from evaluating simple response correctness and toward evaluating the model's ability to manage the intake process responsibly.