A DeBERTa audit compares medical classifications with five token explanations

A classifier can assign a medical abstract to the right category while highlighting words that do not explain its decision well. Conversely, two explainers can disagree about the same prediction. A DeBERTa-v3 auditing study examines both problems together, comparing prediction quality with token-level explanations across five medical-abstract categories.
The task is document classification, not diagnosis of individual patients. The model classifies medical abstracts into five diagnostic categories without task-specific training. The authors use an existing natural-language-inference model and audit the explanations attached to its decisions.

Turning categories into statements

The inference engine treats an abstract as a premise and a category description as a hypothesis. It scores whether the abstract entails that description. Rather than representing a category with a short label alone, the study supplies five enriched hypotheses per category, spelling out relevant conditions and terminology.
The corpus includes neoplasms, digestive-system diseases, nervous-system diseases, cardiovascular diseases and general pathological conditions. The last category covers systemic or nonspecific states, making it harder to separate from the others. For the experiment, the authors select a balanced sample of 1,000 abstracts per category. That balances the audit sample; it does not establish that the category distribution matches a hospital’s records.
Zero-shot here means no training specifically for this classification task. It does not mean an untrained model: the underlying DeBERTa checkpoint has already been trained for inference and topic-classification tasks. Its prior representations remain part of the downstream decision.

Comparing five explanations of one prediction

The framework compares SHAP, LIME, occlusion, Input × Gradient and Attention × Gradient. These approaches look for relevance in different ways: perturbing an input, following gradients, or combining gradients with the transformer’s attention information.
The pipeline standardizes each method’s output using its top 20 tokens. Pairwise Jaccard overlap measures how much the resulting token sets agree. A high overlap means the methods select similar words; it does not establish that those words provide a correct causal account of the model.
That distinction is explicit in the study’s design. The framework compares explanations against each other rather than against a ground-truth explanation. There is no such reference explanation for this model’s decisions over the free-text medical abstracts. The authors also distinguish an explanation that appears plausible from one that faithfully describes the computation.

What disagreement helps expose

The abstract reports stronger convergence in well-defined categories and less stable explanations under diagnostic ambiguity. Its qualitative error audit identifies lexical hypersensitivity, semantic overlap and loss of attribution coherence as failure mechanisms. A broad pathological category can overlap with more specific categories instead of supplying a clean alternative.
The methodology uses SHAP maps and waterfall plots to examine misclassified examples from that ambiguous group. This joins the prediction audit to the explanation audit: the question is not only which label was wrong, but whether the words receiving credit reveal an unstable or overly lexical decision.
For someone evaluating a medical-text classifier, the useful methodological lesson is to examine failures by category and compare multiple explanation families on the same examples. A single attractive heatmap cannot settle whether a prediction is dependable. Neither can agreement among five heatmaps substitute for clinical validation. This study provides an auditing framework and evidence about a particular document-classification setup, not a demonstrated patient-care system.