Suppose you have done the hard evaluation work from the previous piece and found a tool that genuinely holds up. You are not done. You are at the start of the part that most often goes wrong.

Deploying clinical AI is not a software install. It is a systems-integration, change-management, and monitoring program, and a model that looked excellent in evaluation can still cause harm if it is dropped into a workflow that was not redesigned to receive it. Here is how to do it safely.

Start in the shadows

Before an AI output changes a single decision, run it silently. In shadow mode, the model scores real patients in real time, but its outputs go to a log, not to the care team. You compare what it would have said against what actually happened and what clinicians actually did.

A shadow period of a few months does three things. It reveals the true alert volume you would have inflicted on staff. It shows whether the model's performance on your patients matches the evaluation you were sold. And it gives you a local baseline to monitor against later. Skipping this step is how organisations discover, live, that a tool fires ten times more often than expected.

Integration is most of the work

The unglamorous truth is that connecting a model to your electronic health record is usually harder, slower, and more expensive than the model itself. The data has to be mapped from your fields to the model's expected inputs. It has to arrive fast enough to be useful. Missing or late values have to be handled without silently corrupting the prediction. Standards such as FHIR and SMART on FHIR help, but they do not erase the work of reconciling how your clinic records reality with how the model expects it to be recorded.

Two design rules matter from day one. First, build a fail-safe: the ability to mute the AI, fall back to the previous workflow, and keep an audit trail of what it showed and when. Second, assume you will need to turn it off in a hurry one day, and make sure you can, without a vendor support ticket.

Design the alert, not just the model

A correct prediction delivered at the wrong moment, to the wrong person, in an unreadable form, is a failed intervention. The first piece covered alert fatigue; deployment is where you either respect it or create it.

Good deployment answers workflow questions the model cannot: Who receives this output? At what moment in their day? What, specifically, are they expected to do about it? What happens if they disagree? The most successful clinical AI is designed around a human in the loop who stays accountable for the decision, with the model as an assistant that makes its reasoning legible, not an oracle that issues verdicts.

Monitor for drift, forever

A clinical model is not a finished object. The population changes, a lab analyser is swapped, the software around it updates, clinical practice moves on, and performance drifts. This is the dataset-shift problem, and it does not announce itself.

So monitoring is not optional post-launch housekeeping; it is part of the tool. At a minimum, track the model's score distribution, its calibration, its performance across the subgroups you care about, and its real-world alert volume and override rate, on a defined cycle, with pre-agreed thresholds that trigger recalibration, retraining, or rollback. If no one owns that dashboard, no one owns the tool's safety.

Governance, and the rules that are catching up

Regulators have moved from treating AI as ordinary software to treating it as what it is: a medical technology whose performance can change over time. A few reference points worth knowing.

The FDA, with Health Canada and the UK's MHRA, published Good Machine Learning Practice guiding principles that stress data quality, reproducible development, human factors, and ongoing monitoring across the product's life. The agency has now authorised a large and fast-growing catalogue of AI-enabled medical devices, and has introduced mechanisms for vendors to pre-specify how a model may be updated without a fresh review each time. In Europe, the AI Act classifies AI that functions as a medical device, or as a safety component of one, as high-risk, with obligations for documentation, human oversight, and post-market monitoring. The World Health Organization has issued guidance on the ethics and governance of AI for health, and specific guidance on large multi-modal models.

You do not need to be a regulatory expert. You need a governance structure that behaves like these frameworks expect: a cross-functional review board (clinicians, data science, IT, legal, and ideally a patient voice) that approves deployments, sets the monitoring plan, and holds the authority to switch a tool off.

Two tales worth keeping in mind

The contrast between two well-known deployments captures the whole lesson.

IDx-DR is the encouraging one. It is an autonomous system that screens for diabetic retinopathy in primary care, and it became the first autonomous AI diagnostic the FDA authorised, in 2018, on the strength of a prospective study in real primary-care clinics. Its success was not magic; it was scope and evidence. The intended use was narrow and clearly defined, the trial measured performance where the tool would actually be used, and there was a clear escalation path for positive results.

IBM Watson for Oncology is the cautionary one. Marketed as a system that could recommend cancer treatments, it struggled to generalise across the very different practice patterns of real tumour boards, and reporting later surfaced internal concerns about unsafe or incorrect recommendations. High-profile partnerships were shelved after large sums were spent. The lesson was not that the idea was absurd, but that ambition outran evidence, the logic was opaque, and local customisation was underestimated.

A third case sits between them. When a leading diabetic-retinopathy model was deployed in clinics in Thailand, a human-centered evaluation found that a system with excellent laboratory accuracy still hit real-world friction: images were rejected for quality under clinic lighting, connectivity was slow, and nurses had to work around it. The model was good. The deployment context decided whether that goodness reached patients.

What this means for you

If you are a clinician: insist on a shadow period, local performance figures, and a clear escalation path before an AI recommendation enters your workflow. Know how to mute it.

If you build or integrate these tools: ship the monitoring, not just the model. Deliver drift detection, subgroup dashboards, and a documented rollback procedure as part of the product.

If you lead a health system: treat AI as an operational commitment with a running cost, not a one-time purchase. Fund the integration, the governance board, and the monitoring, and write vendor accountability for updates and drift into the contract.

Get this right and the payoff is real, especially for the applications where AI helps most today. That is where the series turns next: ambient documentation, the clearest near-term win for clinicians, and its limits.

Browse the full AI in Healthcare series.

References

US Food and Drug Administration, Health Canada, and MHRA. Good Machine Learning Practice for Medical Device Development: Guiding Principles. October 2021. https://www.fda.gov/medical-devices/software-medical-device-samd/good-machine-learning-practice-medical-device-development-guiding-principles
US Food and Drug Administration. Artificial Intelligence-Enabled Medical Devices (list). https://www.fda.gov/medical-devices/software-medical-device-samd/artificial-intelligence-enabled-medical-devices
Regulation (EU) 2024/1689 of the European Parliament and of the Council (the AI Act). https://eur-lex.europa.eu/eli/reg/2024/1689/oj
World Health Organization. Ethics and governance of artificial intelligence for health. 2021. https://www.who.int/publications/i/item/9789240029200
Abramoff MD, Lavin PT, Birch M, Shah N, Folk JC. Pivotal trial of an autonomous AI-based diagnostic system for detection of diabetic retinopathy in primary care offices. npj Digital Medicine. 2018;1:39. https://www.nature.com/articles/s41746-018-0040-6
Ross C, Swetlitz I. IBM's Watson supercomputer recommended "unsafe and incorrect" cancer treatments, internal documents show. STAT News. 25 July 2018. https://www.statnews.com/2018/07/25/ibm-watson-recommended-unsafe-incorrect-treatments/
Beede E, Baylor E, Hersch F, et al. A Human-Centered Evaluation of a Deep Learning System Deployed in Clinics for the Detection of Diabetic Retinopathy. CHI Conference on Human Factors in Computing Systems. 2020. https://dl.acm.org/doi/10.1145/3313831.3376718