The demand for explainable AI in clinical settings has grown from an academic debate into a concrete regulatory requirement. The EU AI Act categorises most diagnostic, prognostic, and treatment recommendation AI systems as high-risk, imposing requirements for transparency, human oversight, and accuracy robustness. In the United States, the FDA's Predetermined Change Control Plan framework implicitly requires that developers understand their models well enough to predict how changes to training data or algorithms will affect outputs — a capability impossible to demonstrate without meaningful interpretability.
Why Saliency Maps Are Not Enough
The most widely deployed explainability methods in clinical AI — SHAP (SHapley Additive exPlanations), LIME (Local Interpretable Model-agnostic Explanations), and gradient-based attention maps — provide post-hoc approximations of model behaviour rather than true mechanistic transparency. They show which input features most influenced a specific prediction, but they do not reveal why the model has learned to weight those features, whether those weightings are stable under distribution shift, or whether a clinician could reliably predict the model's outputs given those explanations.
Research has consistently shown that clinicians presented with saliency maps as explanations for AI-generated diagnoses do not reliably improve their diagnostic accuracy relative to clinicians who receive the AI prediction without any explanation. In some cases, misleading saliency maps actively degrade clinician performance by redirecting attention to spurious correlations that the model has overfit.
Explainability that does not improve a clinician's ability to appropriately rely on or appropriately override an AI recommendation is not explainability — it is a compliance artefact.
A Clinically Grounded Taxonomy of Explainability
At GHAI Foundation, we define explainability in clinical AI across three levels. Global interpretability refers to the developer's ability to characterise the model's overall decision logic — the most significant learned features, the population subgroups where performance diverges, and the conditions under which the model should not be trusted. Local interpretability refers to the model's ability to provide case-specific reasoning that a qualified clinician can evaluate. Functional transparency refers to the clinician's ability to predict, given an explanation, how the model would respond to changes in specific input values.
We require that clinical AI systems submitted for certification under our framework demonstrate all three levels, using standardised evaluation protocols including clinician simulation studies and adversarial input testing. The goal is not philosophical transparency for its own sake — it is ensuring that the humans who rely on these tools in clinical decision-making can do so rationally and appropriately, knowing when to trust and when to override.
