If you follow health technology at all, you have seen the headlines. An algorithm reads a scan as well as a radiologist. A model predicts a heart attack days before it happens. A tool spots sepsis hours ahead of the care team. The studies are real, the numbers are often impressive, and the excitement is understandable.

Then the tool arrives in an actual clinic, and something is off. The predictions are noisier than expected. The alerts fire on the wrong patients. The performance that looked so clean in the paper is harder to find on the ward.

This is not bad luck, and it is usually not fraud. It is a well documented pattern, and it has a name: the gap between how AI performs where it is built and how it performs where it is used. Understanding that gap is the single most useful thing a clinician, a researcher, or a health system leader can do before trusting any clinical AI. This piece opens our AI in Healthcare series by explaining why the gap exists. The next piece turns it into a checklist you can actually use.

Performance is not one number, it is a ladder

The first mistake is treating "accuracy" as a single fact about a model. It is not. A clinical AI tool passes through several very different tests, and each one is weaker evidence than the next.

Development performance. How well the model fits the data it learned from. This tells you almost nothing about the future, because a model can memorise quirks of its training set.
Internal validation. Performance on held-out data from the same source. Better, but the patients, scanners, and coding habits are still the same ones the model grew up with.
External validation. Performance at a different site, with different patients and equipment. This is where reality starts to bite.
Prospective evaluation. Performance when the model runs live, on patients who have not happened yet, inside a real workflow.
Post-market surveillance. Performance months and years later, as the population, the software around it, and clinical practice all drift.

Most published claims sit on the lowest two rungs. A systematic appraisal of hundreds of covid-19 prediction models in the BMJ found almost all of them at high risk of bias and poorly validated, which is a fair summary of the wider literature too. When you read that a model "achieved an AUC of 0.95", the honest follow-up question is: on which rung of the ladder, and how far is that from your patients?

A concrete example: the sepsis model that missed most of the sepsis

The clearest case study is one of the most widely deployed clinical AI tools in the world: a proprietary sepsis prediction model embedded in a major electronic health record and switched on at hundreds of hospitals.

When independent researchers at the University of Michigan validated it externally across more than 38,000 hospitalisations, the results were sobering. The model's area under the curve was 0.63, well below the discrimination the vendor had reported. At the alerting threshold in use, it identified only about a third of sepsis cases (sensitivity of 33 percent), and when it did fire, it was right only about one time in eight (positive predictive value of 12 percent). It also generated a large volume of alerts, adding work for nurses and physicians who then had to sort signal from noise.

Nothing about this makes the underlying idea worthless. It makes the point that a tool can be technically real, commercially successful, and widely installed, and still not deliver the benefit its marketing implies once it meets a new hospital's patients and workflows.

Why the gap opens

Four mechanisms account for most of the disappointment.

Dataset shift. A model learns the statistical world of its training data: the mix of patients, the scanner models, the way a particular hospital codes a diagnosis, even the time of day staff tend to order tests. Move it somewhere with a different mix and its assumptions quietly break. Clinicians now have a name for this failure mode, and a New England Journal of Medicine article laid out how ordinary changes, a new lab analyser, a software update, a shift in referral patterns, can degrade a model without anyone touching the model itself.

Optimism and leakage. During development it is disturbingly easy to let information leak from the future into the model, or to tune it until it fits the test set. Both inflate the reported numbers. Neither survives contact with genuinely new data.

Miscalibration. Discrimination (can the model rank sick above well?) is not the same as calibration (are its probabilities correct?). A model can rank patients well and still say "70 percent risk" for a group whose real risk is 20 percent. Since clinical decisions hang off thresholds, a miscalibrated model can be confidently, dangerously wrong even with a healthy AUC.

Low prevalence. Most serious events are rare, and rarity is unforgiving. If a condition affects 1 in 100 patients, even a fairly sensitive and specific model will produce mostly false alarms, because there are so many more well patients to misflag than sick ones to catch. This is arithmetic, not pessimism, and it is why positive predictive value at your prevalence matters far more than a headline AUC.

The alert-fatigue tax

The low-prevalence problem has a human cost. When a tool cries wolf often enough, people stop listening, and they are right to, because most of the alarms are noise. Studies of computerised alerts have long found that clinicians override the large majority of them. An AI that adds a fresh stream of low-value alerts does not just fail to help; it can crowd out attention from the alerts that matter and erode trust in the whole system.

A well designed tool respects this. It fires rarely, at the right person, at a moment when something can actually be done, and it makes the reason for the alert legible. Those are workflow decisions, not model-accuracy decisions, which is exactly why they get overlooked in a paper about accuracy.

What this means for you

If you are a clinician: treat "validated" as a question, not a reassurance. Ask which rung of the ladder the evidence sits on, and whether any of it comes from patients like yours. A model you cannot interrogate is a model you cannot fully trust.

If you build these models: the hard part is not the AUC, it is everything after it. Report calibration, not just discrimination. Validate somewhere you did not train. Assume dataset shift and plan to monitor for it.

If you lead a health system: the paper is the start of due diligence, not the end. Budget for local validation, integration, and monitoring, and be sceptical of any pilot that measures model output rather than patient or workflow outcomes.

None of this is a reason to dismiss clinical AI. Some of the most useful applications, which we cover later in this series, are already saving clinicians real time. It is a reason to judge each tool with the same rigour you would apply to a new drug or device. That is where we go next: the specific questions and metrics that separate a tool worth trialling from one worth skipping.

Browse the full AI in Healthcare series.

References

Wong A, Otles E, Donnelly JP, et al. External Validation of a Widely Implemented Proprietary Sepsis Prediction Model in Hospitalized Patients. JAMA Internal Medicine. 2021;181(8):1065-1070. https://jamanetwork.com/journals/jamainternalmedicine/fullarticle/2781307
Finlayson SG, Subbaswamy A, Singh K, et al. The Clinician and Dataset Shift in Artificial Intelligence. New England Journal of Medicine. 2021;385(3):283-286. https://www.nejm.org/doi/full/10.1056/NEJMc2104626
Wynants L, Van Calster B, Collins GS, et al. Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal. BMJ. 2020;369:m1328. https://ik.imagekit.io/assistencialabs/blog/content/369/bmj.m1328
Van Calster B, McLernon DJ, van Smeden M, et al. Calibration: the Achilles heel of predictive analytics. BMC Medicine. 2019;17:230. https://bmcmedicine.biomedcentral.com/articles/10.1186/s12916-019-1466-7
van der Sijs H, Aarts J, Vulto A, Berg M. Overriding of drug safety alerts in computerized physician order entry. Journal of the American Medical Informatics Association. 2006;13(2):138-147. https://academic.oup.com/jamia/article/13/2/138/734590