The Clinician's Veto: Navigating Trust, Liability, and Uncertainty in Autonomous AI Prescribing
AI systems are now authorized to prescribe medications independently in the U.S. This shift raises critical questions about trust, liability, and how to handle uncertainty in medical decisions. Researchers highlight gaps in current regulations that could impact patient safety.
Researchers from arXiv have published a study examining the transition of AI systems from advisory roles to autonomous prescribing in healthcare. The study focuses on recent developments like the United States bill H.R. 238 and Utah's prescription-renewal pilot, which allow AI to prescribe medications without human oversight. These advancements, while promising, bring up significant concerns about trust, liability, and the handling of uncertainty in medical decisions.
For everyday people, this means that AI could soon be making critical healthcare decisions without direct human intervention. While this could lead to faster and more efficient care, it also raises questions about who is responsible if something goes wrong. The study highlights that current regulations do not adequately address how AI should communicate its confidence in prescriptions or differentiate between genuine clinical ambiguity (aleatoric uncertainty) and model ignorance (epistemic uncertainty).
The paper argues that regulatory guidelines should require calibrated per-prediction confidence thresholds for action-gated decisions, rather than relying solely on aggregate model performance metrics. This would help ensure that AI systems only act when they are sufficiently certain, and that clinicians and patients understand the level of uncertainty involved in each prescription.