Necessary or Sufficient? Evaluating LLM Explanation... | AI Research

Necessary or Sufficient? Evaluating LLM Explanations With Behavioural Evidence
This research investigates whether the explanations provided by Large Language Models (LLMs) actually reflect the factors that drive their decisions. When an AI system makes a recommendation or flags a risk, it often provides a list of reasons. Operators rely on these explanations to monitor the system and catch errors. This paper tests whether these "cited factors" are truly necessary (changing them would change the outcome) or sufficient (they are enough to preserve the outcome on their own) by comparing the model's claims against its actual behavior during controlled experiments. The ai search story also surfaces in EU Regulators Demand Apple and Google..., adding another angle.

Testing Model Logic Through Intervention

To determine if an explanation is accurate, the researchers used a "black-box" intervention method. They took the inputs that led to a specific decision and systematically altered them. For necessity, they changed a single feature to see if the model’s output changed. For sufficiency, they stripped away all other information to see if that single feature was enough for the model to reach the same conclusion. By doing this, they could assign a numerical score to every feature, revealing which ones truly influenced the model, regardless of what the model claimed in its text explanation.

The Gap Between Claims and Influence

The study evaluated eight models from the Claude, GPT, and Gemini families across two tasks: recommending financial advisors and monitoring prompts for risks. The results show that while the models' explanations contain useful information, they are not always reliable. In many cases, the models failed to rank their reasons according to their actual influence. Furthermore, the researchers found that models frequently left out highly influential factors entirely. For example, in advisor recommendations, an uncited feature was often more influential than the least important feature the model actually chose to list. The ai search story also surfaces in Google AI Releases TimesFM 3 for..., adding another angle.

Why Reasoning Effort Matters

The researchers tested whether increasing a model's "reasoning effort"—specifically in GPT-5.4—would improve the quality of these explanations. They found that higher reasoning effort led to fewer instances where the model ignored a highly influential feature. However, this did not necessarily mean the model became better at ranking the features it did choose to include. This suggests that while more intensive processing can help a model identify the most important factors, it does not guarantee that the model will present those factors in a logical or accurate order.

Implications for AI Oversight

The findings highlight a "salience gap," where a model might repeatedly cite a specific factor—like a "jailbreak attempt"—simply because it is a prominent risk category, even if that factor had less actual impact on the specific decision than other, uncited elements. Because these explanations are used by human operators to diagnose errors or decide when to escalate a concern, these discrepancies are significant. The framework developed in this paper provides a way for developers to perform reliability checks on AI agents, ensuring that the explanations provided for oversight are grounded in the model's actual decision-making behavior rather than just plausible-sounding text. To see anthropic in practice, This AI Agent Runs Your Tasks... walks through a concrete example. as detailed in the full paper on Arxiv