The previous piece argued that automation genuinely helps in specific places. This one is its necessary counterweight. Every tool that removes effort also moves effort somewhere else, and sometimes it moves risk with it. None of what follows is an argument against using clinical AI. It is an argument for using it with your eyes open, because the failure modes here are subtle, well documented, and easy to design around only if you know they exist.

Automation bias: the machine is confident, so we relax

The oldest of these risks has a name that predates the current AI wave. Automation bias is the tendency to over-rely on an automated system: to accept its suggestion when it is wrong, or to stop looking because the computer has apparently already looked.

A systematic review in JAMIA, drawing on 74 studies, laid out the pattern in healthcare specifically. Clinical decision support usually improves overall performance, which is exactly why it gets adopted. But the same review found that these systems also introduce new errors that clinicians fail to catch, because the presence of a confident automated recommendation changes how carefully people check. The mediators are predictable: heavier workload, time pressure, and greater trust in the system all make over-reliance more likely. So do the situations most clinics operate in every day.

The practical lesson is that a tool's average accuracy is not the whole story. What matters is how it fails, how often, and whether the humans around it are positioned to notice. The review's own list of mitigations is worth taping to a wall: train people on the tool's limits, keep clear human accountability for the decision, and design the interface to inform rather than to simply recommend.

Deskilling: use it or lose it, even for experts

The newer and more unsettling risk is deskilling: the possibility that leaning on AI erodes the underlying human skill, so that performance drops when the AI is absent.

For a long time this was a theoretical worry. In 2025 it got hard evidence, from an unlikely place. A multicentre study published in The Lancet Gastroenterology & Hepatology looked at experienced endoscopists, each with more than 2,000 procedures behind them, at centres that had adopted AI assistance for colonoscopy. It compared their detection performance on standard, non-AI colonoscopies before the centres adopted AI and after they had been using it routinely. The adenoma detection rate on those unassisted procedures fell from 28.4 percent before AI exposure to 22.4 percent after, an absolute drop of 6 percentage points and a relative decline of about a fifth.

Read that again, because it is genuinely surprising. These were seasoned experts, and regular exposure to AI assistance appears to have dulled their unaided detection. If it can happen to endoscopists with thousands of procedures, it is naive to assume it cannot happen with documentation, coding, triage, or any other skill we hand to a model. The mechanism is intuitive: a skill you stop exercising gets weaker. The implication for administrative AI is that convenience today can quietly buy dependence tomorrow, and that is a cost worth pricing in.

Hallucination: confident text that was never said

The specific failure mode of generative AI is that it can produce fluent, plausible text with no basis in the input. In a clinical context that is not a curiosity, it is a safety issue.

In October 2024, an Associated Press investigation examined a widely used AI transcription tool built on OpenAI's Whisper model and found that it fabricated content, inventing sentences, and in some cases medications or remarks, that were never spoken. Researchers cited in the report found invented text in a large share of the transcripts they examined. The report noted that a medical transcription product built on the technology had been used for an estimated seven million medical visits, and that in some deployments the original audio was discarded, leaving no way to check what the patient actually said against what the model wrote. OpenAI itself recommends against using the tool in high-risk decision-making contexts.

The takeaway is not "never use AI transcription". It is that a system which can confidently invent clinical content demands a verification step and a preserved source of truth. Erasing the original audio to save storage is exactly the wrong economy.

The verification tax

Which brings us to the cost that ties these risks together. When a model drafts your note, your reply, or your code, it does not remove the work of being correct; it moves that work from producing to checking. Economists would call it a shift in the task, not its elimination.

You can see this even in the friendliest data. In the Stanford draft-reply study from the previous piece, clinicians used the AI's draft as their starting point only about a fifth of the time, and the tool produced no measurable time saving, in part because a draft still has to be read and trusted before it is sent. That reading is the verification tax. It is usually smaller than the original effort, which is why these tools can still be worth it, but it is never zero, and if you forget to account for it you will overestimate the benefit and, worse, you may skip the checking altogether. Skipping the checking is where automation bias and hallucination meet, and it is where patients get hurt.

Designing around the risks

The good news is that all three risks respond to the same handful of design and governance choices:

Keep a human genuinely in the loop, with the time and the mandate to override the tool, not just click past it.
Preserve the source data, the audio, the original message, the raw result, so output can always be checked against reality.
Monitor performance after go-live, not just at the pilot. Models drift, populations change, and skills erode slowly enough to miss without measurement.
Protect the underlying skill. If a tool assists a clinical judgement, make sure clinicians still practise that judgement unaided often enough to keep it.
Assign clear accountability. The clinician who signs the note owns it, AI draft or not. That principle, more than any technical control, is what keeps attention where it belongs.

None of this is exotic. It is the same posture medicine already takes towards any powerful new tool: adopt it for its benefits, respect its failure modes, and never let convenience quietly become dependence. The final piece turns all of this, the burden, the economics, the honest wins, and these risks, into a practical plan.

Browse the full Admin Tax series.

References

Goddard K, Roudsari A, Wyatt JC. Automation bias: a systematic review of frequency, effect mediators, and mitigators. Journal of the American Medical Informatics Association. 2012;19(1):121-127. https://academic.oup.com/jamia/article/19/1/121/732254
Budzyń K, Romańczyk M, Kitala D, et al. Endoscopist deskilling risk after exposure to artificial intelligence in colonoscopy: a multicentre, observational study. The Lancet Gastroenterology & Hepatology. 2025;10(10):896-903. https://www.thelancet.com/journals/langas/article/PIIS2468-1253(25)00133-5/abstract
Burke G, Schellmann H. Researchers say an AI-powered transcription tool used in hospitals invents things no one ever said. Associated Press. October 26, 2024. https://www.seattletimes.com/nation-world/nation/researchers-say-an-ai-powered-transcription-tool-used-in-hospitals-invents-things-no-one-ever-said/
Garcia P, Ma SP, Shah S, et al. Artificial Intelligence-Generated Draft Replies to Patient Inbox Messages. JAMA Network Open. 2024;7(3):e243201. https://jamanetwork.com/journals/jamanetworkopen/fullarticle/2816494