The first piece in this series explained why clinical AI so often underperforms in the real world. This one is the practical companion: when a vendor or a colleague puts a tool in front of you, what should you actually ask before you let it near a patient?

You do not need to be a data scientist to do this well. You need a short list of the right questions, and the confidence to keep asking them until you get straight answers. Here is that list.

Ask what the model is really predicting

Every model predicts a label, and the label is often not the thing you care about. It is a stand-in, and the gap between the stand-in and reality is where harm hides.

The clearest illustration is one of the most cited papers in the field. Researchers examined a widely used algorithm that helped decide which patients received extra care management. It predicted healthcare cost as a proxy for healthcare need, which sounds reasonable until you remember that less money is spent on some groups of patients for reasons that have nothing to do with being healthier. Because of that, at the same risk score, Black patients were considerably sicker than White patients. Correcting the flaw would have raised the share of Black patients flagged for extra help from 17.7 percent to 46.5 percent. The maths was fine. The choice of label was not.

So the first question is simple: what exactly does this model output, how was that label defined, and where might it diverge from the clinical reality you actually want to act on? If the answer is vague, that is your answer.

Demand the right metrics, not the flattering one

AUC is the number vendors love because it is usually the highest one. It measures discrimination, the model's ability to rank a sicker patient above a healthier one. It is genuinely useful, and genuinely incomplete. Ask for four more things.

Calibration. Do the model's probabilities mean what they say? A tool that labels a group "80 percent risk" when their true risk is 30 percent will drive bad decisions no matter how well it ranks patients. Calibration has been called the Achilles heel of predictive analytics precisely because it is so often ignored. Ask for a calibration plot, and ask whether it was recalibrated for a population like yours.

Predictive value at your prevalence. Sensitivity and specificity are properties of the model. Positive and negative predictive value depend on how common the condition is in your setting. For a rare event, even a strong model can be wrong most of the times it fires. Ask for the expected positive predictive value and the number of alerts per thousand patients at the threshold you would actually run.

Net benefit. Decision curve analysis, introduced by Vickers and Elkin, asks a more honest question than accuracy: across the range of thresholds a clinician might reasonably use, does acting on the model beat the simple alternatives of treating everyone or treating no one? It folds the real costs of false positives and false negatives into one picture. If a vendor has never heard of it, that tells you something.

Number needed to alert. Borrowed from the logic of number-needed-to-treat: how many alerts must a clinician work through to change one patient's outcome for the better? A tool that needs a hundred interruptions to help once is a tool your team will soon ignore.

Insist on external and prospective evidence

As the first piece described, evidence sits on a ladder, and internal validation is near the bottom. Two questions cut through most of the noise.

First, has the model been validated externally, on patients, equipment, and workflows different from where it was built? A tool with only internal validation is a promising prototype, not a proven product.

Second, is there any prospective evidence, ideally measuring what happened to patients or to the workflow, rather than just the model's numbers? The strongest form is a proper trial. The field now has reporting standards built for exactly this: TRIPOD+AI for prediction-model studies, and the CONSORT-AI and SPIRIT-AI extensions for clinical trials and protocols of AI interventions. You do not have to memorise them. You can simply ask whether the study followed them, and treat a blank look as informative. Risk-of-bias tools such as PROBAST exist so that reviewers can grade this rigorously; the same instinct works at the bedside.

Check who it works for, and who it does not

An average is a place where no real patient lives. A model can post excellent overall numbers while quietly failing an age band, a skin tone, a body habitus, or the particular scanner your clinic runs. The care-management algorithm above passed every aggregate check and still encoded a large disparity.

Ask for performance broken down by the subgroups that matter in your population: age, sex, ethnicity where relevant, comorbidity, and device or data source. If those numbers do not exist, the honest interpretation is that no one has checked, which means you would be the ones to find out, on live patients.

A short list you can actually use

If you are a clinician, ask for: external validation on a population like yours; a calibration plot; the confusion matrix and positive predictive value at the operating threshold; subgroup performance; and a plain explanation of what drives each prediction and what you are meant to do when it fires.

If you are a researcher or data scientist, deliver: temporal and external validation; calibration and decision-curve analysis, not discrimination alone; pre-specified thresholds and evaluation plans; honest subgroup reporting; and code or a specification detailed enough to reproduce. Publish the negative results too.

If you are an administrator or leader, require: evidence of a local pilot with clinical or operational endpoints; realistic integration and monitoring costs in the business case, not just the licence fee; and contractual commitments for retraining, drift monitoring, and support before, not after, you sign.

Get straight answers to these and you can tell a tool worth trialling from one worth skipping, without a statistics degree. The next piece assumes you have found a tool worth trialling, and turns to the harder problem of deploying it safely.

Browse the full AI in Healthcare series.

References

Obermeyer Z, Powers B, Vogeli C, Mullainathan S. Dissecting racial bias in an algorithm used to manage the health of populations. Science. 2019;366(6464):447-453. https://www.science.org/doi/10.1126/science.aax2342
Collins GS, Moons KGM, Dhiman P, et al. TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods. BMJ. 2024;385:e078378. https://ik.imagekit.io/assistencialabs/blog/content/385/bmj-2023-078378
Liu X, Cruz Rivera S, Moher D, et al. Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension. Nature Medicine. 2020;26:1364-1374. https://www.nature.com/articles/s41591-020-1034-x
Vickers AJ, Elkin EB. Decision curve analysis: a novel method for evaluating prediction models. Medical Decision Making. 2006;26(6):565-574. https://journals.sagepub.com/doi/10.1177/0272989X06295361
Van Calster B, McLernon DJ, van Smeden M, et al. Calibration: the Achilles heel of predictive analytics. BMC Medicine. 2019;17:230. https://bmcmedicine.biomedcentral.com/articles/10.1186/s12916-019-1466-7