↓ Skip to main content
Expert LinkedIn
AI AI in Healthcare AI Ethic

The Most Dangerous Number in Healthcare AI Is the Average

10 min read
Dr. Sophia Patel
Dr. Sophia Patel AI in Healthcare Expert & Machine Learning Specialist

The most dangerous number in healthcare AI is often the one that looks the safest.

An average accuracy score can tell a hospital that a model is ready while hiding the exact patients, care settings, and workflows where the model becomes unreliable. In April, I argued in The Slide That Could Save Everyone - If the Model Works for Everyone that pathology AI could sell a democratization story while still underperforming for the people it claimed to help. In July, Clinical AI’s Next Bottleneck Isn’t Accuracy. It’s Who Pays - and by What Standard. made a related point about markets and reimbursement: a strong benchmark does not settle real deployment. The next layer is measurement itself. Too many teams still ask whether the model works on average. In medicine, average performance is exactly where risk can hide.

A pristine clinical diagnostic panel sits under cool hospital light with one smooth central performance gauge, while a cracked lower layer reveals diverging patient-image panels and warning traces across different skin tones and care settings.
A reassuring average score can sit on the surface while subgroup failures accumulate underneath.

A good average can still be a bad clinical decision
#

The healthcare AI literature now has a familiar habit. A paper reports an impressive headline number. A vendor repeats it in a deck. A buyer sees something that looks mature. Then, if you read the evaluation details, the confidence gets thinner.

The 2026 systematic review Responsible artificial intelligence in medical imaging is useful precisely because it refuses to stop at the headline. Across 24 studies in X-ray, CT, MRI, mammography, ultrasound, dermoscopy, retinal imaging, and abdominal CT, the authors found plenty of eye-catching metrics. Several papers reported accuracy or sensitivity above 90%. But they also warned that many of those results depended on internal validation, curated public datasets, class-balanced splits, augmentation, or limited demographic reporting. In plain English: a beautiful number inside a tidy test environment can still be a weak promise in a messy hospital.

That warning is not theoretical anymore. In the U.S.-focused systematic review Racial and Ethnic Disparities in the Diagnostic Performance of AI Systems for Medical Imaging, researchers identified 27 eligible studies published between 2015 and 2026. Several reported reduced diagnostic performance, lower sensitivity, higher underdiagnosis, or subgroup disparities for racial and ethnic minority populations, especially Black and Hispanic patients. Once you know that pattern exists across 27 studies, it becomes much harder to pretend subgroup failure is an edge case.

The real question is no longer whether healthcare AI can produce a strong average. It clearly can. The question is whether the average is telling you the truth you actually need before clinical deployment.

The gaps appear as soon as you slice the data
#

The clearest example I found came from Equity and Generalizability of Artificial Intelligence for Skin-Lesion Diagnosis Using Clinical, Dermoscopic, and Smartphone Images. Across more than 70,000 test images, the pooled AUROC was 0.88. On a vendor slide, that number sounds reassuring. But the study becomes more interesting once it is broken apart.

Performance was highest in specialist settings, with AUROC at 0.90. It dropped to 0.85 in community care and 0.81 in smartphone environments. It also fell from 0.89 in lighter skin tones to 0.82 in darker skin tones. That is exactly why I distrust top-line averages in clinical AI. The average tells you the model is promising. The slices tell you where the clinical risk starts.

This is the part many health systems still underweight. They buy or pilot a model as if validation were a single destination. In practice, healthcare AI needs at least three kinds of proof at once: subgroup proof, setting proof, and workflow proof. A model that performs well in a specialist clinic may degrade in community care. A model that performs well on lighter skin may lose reliability on darker skin. A model that looks strong in one imaging pipeline may drift once the camera, scanner, or patient mix changes.

That is not a minor technical footnote. It is the core deployment question.

Medicine has already paid for this blind spot before
#

AI did not invent the danger of trusting a medical average. Medicine already has a painful example.

The scoping review Impacts of Skin Color and Hypoxemia on Noninvasive Assessment of Peripheral Blood Oxygen Saturation examined 10 studies on pulse oximetry. Eight reported statistically significant higher pulse-oximeter readings in darker-skinned patients with hypoxia than arterial blood gas measurements showed. Occult hypoxemia was more prevalent in Black and Hispanic patients than in White patients.

That history matters for AI because it shows how easily a measurement system can look broadly acceptable while failing the people who most need accurate detection. Pulse oximeters did not become controversial because they failed on average. They became controversial because the average hid who was being misread.

Healthcare AI teams should take that lesson seriously. A model does not become safe because its summary metric clears a threshold. It becomes safer when the institution can explain where performance drops, how those drops are monitored, and what happens when the model meets the patients or settings it understands worst.

Fairness fixes are not one-and-done
#

Even when developers try to correct bias directly, the result is not always as clean as the headline suggests.

In Adversarial debiasing for age-equitable diabetes prediction, researchers improved recall for the smallest age group, those older than 50, from 0.5556 to 0.7778 while keeping overall ROC-AUC nearly flat. If you stop there, the intervention looks like a straightforward win. But the same study found that the recall parity gap on the primary test split widened from 0.0996 to 0.2153, and fairness gains varied materially across random seeds.

That is an uncomfortable but useful result. It means fairness work cannot be treated like a cosmetic patch you apply once before launch. You can improve one subgroup and still worsen the balance elsewhere. You can reduce bias on one test partition and watch the effect wobble on another. In healthcare, where subgroup sizes are often small and clinically uneven, that instability is not a lab curiosity. It is exactly why slice-level monitoring has to continue after deployment.

Strong AUROC still does not buy clinical trust
#

Some of the most misleading numbers in healthcare AI are the ones that are genuinely strong.

The systematic review and meta-analysis Evaluating Artificial Intelligence Models for ICU Length of Stay Prediction found a pooled AUROC of 0.9005 across 33 studies. That is a serious result. It is also not enough. The same review flagged methodological heterogeneity, scarcity of external validation, and a near absence of calibration reporting.

Calibration matters because a model can rank patients in roughly the right order while still misstate how much risk any one patient actually carries. External validation matters because a model that works in one institution may degrade in another. If those two checks are weak, a strong AUROC becomes less of a deployment green light and more of a warning that the buyer may be mistaking discrimination for readiness.

The broader governance reviews are converging on the same conclusion. Bias and Fairness Across the Healthcare AI Lifecycle argues that fairness cannot be guaranteed by one metric, one publication, regulatory clearance, or one-time validation. Beyond Model Development in Healthcare AI reaches the operational version of that point: the field keeps recommending monitoring, but methods, action thresholds, fairness surveillance, and corrective responses remain weakly standardized.

That is the real maturity gap. The language of responsible health AI is advancing faster than the machinery of responsible health AI.

Regulators and buyers are quietly moving in the same direction
#

The encouraging part is that the strongest governance institutions are no longer speaking as if average performance were enough.

The WHO’s Ethics and governance of artificial intelligence for health places ethics and human rights at the center of AI deployment, not at the end of it. The FDA’s January 2025 press announcement on comprehensive draft guidance for developers of AI-enabled medical devices says explicitly that the agency is addressing transparency and bias throughout the device lifecycle, including postmarket performance monitoring. Its related guidance page on AI-enabled device software lifecycle management reinforces that risk management does not stop at approval.

The Coalition for Health AI is moving the buyer side in a similar direction. CHAI’s October 2024 post on assurance lab certification and a health AI “nutrition label” proposes model-card disclosures that include intended uses, targeted patient populations, key performance metrics, maintenance requirements, known risks, and known bias. That is a practical sign of where procurement is heading. Hospitals are being given better questions. They should use them.

What I would ask before approving a deployment
#

If I were reviewing a clinical AI product today, I would want five answers before I cared much about the average score.

  1. Which patient groups are represented in the training and test data, and which are thinly represented?
  2. What do the subgroup results look like by race or ethnicity, age, sex, language, site of care, and device or image source where relevant?
  3. What local validation has been run in the environment where the model will actually be used?
  4. What postdeployment metrics will be watched for drift, calibration error, fairness decay, and workflow harm?
  5. Who has the authority to pause, retrain, narrow, or retire the model when those signals move the wrong way?

Those are not idealistic add-ons. They are the minimum questions required if you want to know whether a clinical model is safe for more than the median case.

The uncomfortable truth
#

The average is often where an institution hides its uncertainty.

It lets a vendor compress a complex performance story into one number. It lets a health system say a model is evidence-based without specifying who might be poorly served. It lets a pilot feel mature before the harder questions of subgroup reliability, calibration, local fit, and postmarket monitoring have been answered.

Healthcare AI will not earn trust when it clears one summary metric. It will earn trust when vendors and hospitals can name exactly which patients, settings, and devices stress the model, and what they will do then.

Working on healthcare AI evaluation, procurement, or clinical governance and seeing a gap between the top-line score and the real deployment risk? I would like to hear what that gap looks like in practice.

Email me at sophia.patel@tlnw.uk

Vertical infographic comparing subgroup-performance warning signs in healthcare AI, including skin-tone AUROC gaps, care-setting drops, diabetes fairness tradeoffs, pulse-ox bias, and ICU-model validation gaps.
Healthcare AI can look strong overall while underperforming by skin tone, care setting, or subgroup.

References
#

AI Content Notice

This article was created using artificial intelligence technology. Whenever possible, we include references and sources to support the information presented. Readers are encouraged to consult these sources for further information. While we strive for accuracy and provide valuable insights, readers should independently verify information and use their own judgment when making business decisions. The content may not reflect real-time market conditions or personal circumstances.

Related Articles

Tags

#healthcare ai #algorithmic bias #subgroup validation #medical imaging #clinical governance