When a major US health system used an algorithm to identify patients requiring complex care management, an independent analysis later found the tool systematically assigned lower risk scores to Black patients than to equally ill White patients. The algorithm was optimised to predict health costs, not health needs — and because systemic inequities mean Black patients historically spend less on healthcare despite comparable illness burdens, the model learned to encode racial inequality as a feature.
This example, now widely cited in the algorithmic fairness literature, is not an anomaly. It is a representative illustration of a systemic problem in how clinical AI systems are built, evaluated, and deployed.
The Scope of the Problem: What the Evidence Shows
A 2023 systematic review published in The Lancet Digital Health examined 68 validated clinical AI algorithms across seven disease categories. Across all categories, model performance was significantly lower — by an average of 11.3 percentage points in AUC — for non-White patients compared to the White patients who typically dominated training datasets. For dermatology AI specifically, the disparity was as high as 26 percentage points, driven by a near-absence of darker skin tones in training corpora.
Sex-based disparities are similarly pervasive. Cardiovascular risk models trained predominantly on data from male patients routinely underestimate coronary risk in women, who present with different symptom profiles and biomarker patterns. These are not edge cases. They are predictable consequences of building AI systems using datasets that reflect historical inequities in healthcare access and documentation.
An AI system that achieves 93% accuracy overall while performing at 79% for a specific demographic subgroup is not a good AI system. It is a system that has automated and amplified existing disparities at machine speed.
Defining Fairness: A Framework Beyond Overall Accuracy
Fairness in algorithmic systems is not a single property — it is a family of related but distinct criteria that can sometimes be mutually incompatible. Demographic parity requires equal positive prediction rates across groups. Equalised odds requires equal true positive and false positive rates. Calibration requires that predicted probabilities match actual outcomes across groups. No single metric captures all dimensions of algorithmic fairness, and the appropriate metric depends on the clinical context.
For screening tools, where false negatives carry severe consequences (missed cancer diagnoses, delayed sepsis treatment), equalised odds with a particular focus on true positive rate parity is the priority. For resource allocation tools, demographic parity across predicted risk scores may be the appropriate constraint. The GHAI Foundation's fairness audit framework requires that developers specify, justify, and then demonstrate conformance with the fairness criterion most appropriate to their specific deployment context — not simply report overall accuracy.
Our Recommendations for Regulators and Developers
First, subgroup performance reporting must be mandatory, not optional. Any regulatory submission for a clinical AI tool should include stratified performance metrics across race, sex, age group, and geographic setting at minimum. Second, training dataset demographic composition must be disclosed and independently verified before market authorisation. Third, post-market surveillance should include ongoing monitoring of algorithmic performance across demographic groups in real-world deployments, with automatic triggers for re-evaluation when performance gaps widen.
