A study of legal-authority verification asks whether models can abstain when none of the available answers is correct. Its focus is a closed-choice task: select the relevant holding, or recognize that it is missing. That is a different requirement from producing a fluent explanation of a legal question.
Remove the correct answer deliberately
The paper extends CaseHOLD by replacing the correct holding in 1,670 of 5,314 items. It compares four decision models, which return option probabilities, with six generative language models.
The modification creates cases where choosing any supplied holding is an error. That makes abstention directly measurable instead of assuming that a high score on ordinary multiple-choice questions implies reliable recognition of missing authority.
Strong ordinary accuracy can coexist with wrong commitments
When no option is correct, the authors report that generative models still choose one in 38 to 86 percent of items. The most cautious decision model does so in 30 percent. The latter is an improvement in the tested comparison, but it is still a substantial rate of wrong commitments.
On standard CaseHOLD, a decision model reaches 88.1 percent accuracy against 88.2 percent for the best compared generative model, at a reported fifty-fifth of its cost. Those accuracy and cost figures belong to the paper’s particular task and setup, not all legal AI workloads.
Check the failure type, not only the total
The paper reports that accepting a generative model’s answer only when a decision model agrees halves wrong citations in the evaluated setting. It also explores distilling decision probabilities into a local model.
For evaluators, the useful distinction is between getting an available answer right and refusing an unsupported choice. Agreement can reduce some errors without guaranteeing that both systems are correct. This is a benchmark study of legal-authority selection, not a substitute for checking an actual citation or determining how a legal rule applies to a real case.
Comments