ConceptSMILE: Auditing the Trustworthiness of Conce... | AI Research

Concept-based explainable AI (C-XAI) aims to make complex machine learning models more transparent by using human-understandable concepts—such as clinical biomarkers or semantic attributes—rather than just raw pixels or abstract data. However, even if a model provides an explanation that seems logical to a human, it may still be relying on hidden biases, shortcuts, or unstable features. The paper introduces ConceptSMILE, an independent auditing framework designed to test whether these concept-based explanations are actually trustworthy, reliable, and faithful to the model's true decision-making process.

How ConceptSMILE Works

ConceptSMILE is a model-agnostic framework, meaning it can be applied to various types of AI systems, including vision-language models and concept bottleneck models. It functions by treating concept-based explanations as claims that must be verified through systematic testing.
The process involves four main steps: 1. Perturbation: The framework systematically alters input regions of the data. 2. Measurement: It observes how these changes affect the model’s concept-level responses. 3. Weighting: It applies locality weighting to focus on the most relevant data shifts. 4. Surrogate Modeling: It fits an XGBoost surrogate model to approximate how the AI behaves locally, allowing researchers to evaluate the explanation's reliability through metrics like faithfulness, stability, and consistency.

Evaluating Reliability in Medical Imaging

To demonstrate the framework, the researchers applied ConceptSMILE to retinal fundus images, a high-stakes domain where accurate and transparent reasoning is critical. They compared two different ways of extracting concepts: one using MedSAM (a tool for visual segmentation) and another using a vision-language model (VLM) to generate semantic concepts.
The audit revealed that reliability is not uniform across different methods. The MedSAM pathway demonstrated stronger spatial attribution and achieved the highest surrogate fidelity. Conversely, the VLM pathway showed greater stability when faced with specific image artifacts and proved more faithful in identifying blood vessel features.

Why Auditing Matters

The core takeaway of this research is that human-readable explanations are not inherently self-verifying. A model might produce an explanation that looks correct while actually being driven by "label leakage"—where the model uses hidden information about the final answer—or by spurious correlations, such as camera-specific artifacts or text marks on an image.
By providing an independent layer of auditing, ConceptSMILE helps bridge the gap between a model simply appearing interpretable and actually being reliable. It shifts the focus of XAI from merely generating explanations to rigorously testing whether those explanations hold up under controlled, systematic scrutiny.