Persuadability and LLMs as Legal Decision Tools | AI Research

Persuadability and LLMs as Legal Decision Tools
As Large Language Models (LLMs) are increasingly considered for roles as legal assistants or even first-instance decision-makers, it is vital to understand how they process competing arguments. A core requirement for any legal decision-maker is the ability to be persuaded by the merits of a case while remaining independent enough to avoid being swayed simply by the skill of an advocate. This paper explores the "persuadability" of various LLMs by testing how the quality of an advocate—represented by different AI models—influences the decisions made by a "Judge" model in complex legal scenarios.

Measuring Judicial Persuadability

To test how models handle conflicting legal arguments, the researchers created a trilateral experiment. They selected "hard" legal questions—cases where appellate courts had split decisions—to ensure there was no single, obvious answer. They then tasked "Advocate" models to generate arguments for both sides of these disputes. These arguments were presented to "Judge" models, which were instructed to decide the case. By measuring how often a Judge model favored one Advocate over another, the researchers could quantify the model's persuadability. If a model were perfectly impartial, it would favor each Advocate equally; a deviation from this 50/50 split indicates that the model is being influenced by the identity and rhetorical quality of the advocate.

The Impact of Argument Quality

The study found that all tested models exhibited a statistically significant level of persuadability. Depending on the specific Judge model, the identity of the Advocate had an average effect of 8% to 21% on the outcome. In some instances, the strongest Advocate models were able to secure a win rate of over 90% against weaker counterparts. This suggests that LLMs are not merely neutral arbiters; their final decisions are notably sensitive to the quality of the arguments presented to them.

Substance Versus Style

A key question for the researchers was whether these models were being persuaded by the legal substance of the arguments or merely by the "form"—the fluency and rhetorical power of the advocate. To test this, they provided some Advocates with summaries of the actual legal arguments used in the original court cases, while others were left to generate arguments based only on the facts. The results showed that providing these "hints" slightly reduced the influence of the advocate's identity, suggesting that legal content plays a role in persuasion. However, the difference was generally small, indicating that the rhetorical style of the advocate remains a powerful factor in how these models reach their conclusions.

Implications for Legal AI

The findings highlight a tension in using LLMs for legal tasks: while a decision-maker must be open to persuasion to ensure a fair hearing, they must also maintain the intellectual autonomy to form independent judgments. The study reveals that larger models and those with advanced reasoning architectures are not always less persuadable than smaller ones, and in some cases, the relationship is reversed. These results suggest that as we integrate AI into administrative and judicial settings, we must carefully consider how these models weigh competing arguments, as their susceptibility to the quality of advocacy could fundamentally alter the fairness and consistency of legal outcomes.