In healthcare, an AI tool that is wrong 5% of the time can harm thousands of patients. That is why medicine has some of the most developed thinking on how to evaluate and govern AI. Understanding these practices is useful well beyond hospitals.

How medical AI is regulated

When software is intended to diagnose, treat, prevent or monitor disease, it is usually regulated as a medical device — often called software as a medical device (SaMD).

  • In the United States, the FDA reviews AI devices through pathways that depend on risk, and has published guidance on AI and machine learning, including plans for how manufacturers can pre-specify certain future model updates.
  • In the European Union, AI medical devices fall under the Medical Device Regulation and are also classed as high-risk AI under the AI Act.
  • Other countries have their own frameworks, increasingly coordinated through international regulators’ forums.

Tools for administrative tasks, such as scheduling or billing, generally aren’t medical devices — but can still affect patients.

Why medical AI fails in practice

Dataset shift. A model trained on one hospital’s patients, scanners and workflows can perform worse elsewhere. A well-known 2021 study in JAMA Internal Medicine externally validated a widely used proprietary sepsis prediction model and found substantially poorer performance than its developer had reported, missing many sepsis cases while generating many alerts.

Label problems. If training labels come from past clinical decisions, the model may learn past mistakes or biases.

Workflow mismatch. An accurate model that produces too many alerts causes alarm fatigue and gets ignored.

Shortcut learning. Models sometimes learn spurious cues — such as markings on images or the type of scanner — instead of real disease signs.

Drift. Changes in patient populations, treatments or equipment degrade performance over time.

The evaluation checklist

Health organizations increasingly evaluate AI tools with questions like these:

  1. Intended use: what exactly does the tool do, for which patients, in which setting?
  2. Evidence: was it validated on external data similar to our population? Prospectively?
  3. Performance by subgroup: does accuracy hold across age, sex, ethnicity and other relevant groups?
  4. Workflow fit: who sees the output, when, and what do they do with it?
  5. Human oversight: can clinicians override it? Are they trained to understand its limits?
  6. Monitoring: how will performance be tracked after deployment?
  7. Data protection: where is patient data processed and stored?
  8. Accountability: who is responsible for decisions and errors?
  9. Patient communication: are patients informed when AI is used in their care?

Generative AI raises new questions

Language-model tools — scribes, chatbots, summarizers — produce free text rather than fixed outputs, making evaluation harder. Organizations test them on samples of real cases, track error types such as omissions and invented details, and keep clinicians responsible for final content.

The principle

The WHO’s guidance on AI for health summarizes the consensus: protect human autonomy, promote safety and the public interest, ensure transparency, foster accountability, ensure equity, and make AI responsive and sustainable.

A worked example: deploying an imaging triage tool

Consider a hospital evaluating an AI tool that flags suspected intracranial hemorrhage on head CT scans:

  1. Check the authorization. Confirm the tool is cleared for this exact use, scanner types and patient population.
  2. Run a silent trial. For several weeks, the tool runs in the background while radiologists work as usual. The team compares its flags with final reports, measuring missed bleeds and false alarms on local data.
  3. Look at subgroups. Does performance hold for children, older patients, post-surgical scans and different scanners?
  4. Design the workflow. Decide how flags appear, who receives them, and what radiologists should do when they disagree with the tool.
  5. Go live with monitoring. Track performance monthly, review discordant cases, and set thresholds that trigger a pause.

This “silent mode” step is one of the most useful practices in clinical AI: it reveals problems with local data before any patient is affected.

What to watch next

  • Regulatory guidance for generative AI used in clinical workflows.
  • Requirements for post-market monitoring of AI devices.
  • Hospital AI governance committees and shared evaluation resources.
  • Patient disclosure rules for AI use in care.

Key takeaways

  • Clinical AI is usually regulated as a medical device; in the EU it is also high-risk AI.
  • Models often perform worse in new hospitals because of dataset shift.
  • An external validation of a widely used sepsis model found much weaker performance than reported.
  • Evaluation covers evidence, subgroup performance, workflow fit, oversight and monitoring.
  • Generative AI tools require new evaluation methods focused on errors and omissions.

Frequently asked questions

Are AI scribes regulated as medical devices?

Generally not when they only document conversations for clinician review, though rules vary. Data protection and professional responsibilities still apply.

What is external validation?

Testing a model on data from different institutions or populations than those used to develop it, to check that it generalizes.

Should patients be told when AI is used?

Many ethics frameworks recommend transparency, and some jurisdictions require it for certain uses.