I recently wrote about over-delegation in AI systems, which is the tendency to trust AI outputs a little more each time they’re right until oversight disappears. That problem is about to get bigger as healthcare organizations adopt agentic AI.

Agentic AI is software that can plan a task, carry it out, and move on to the next one without waiting for someone to click “approve.” In healthcare, this might look like a tool that both flags a scheduling conflict and reschedules the patient, or instead of alerting a human that there’s a low supply of a critical medication, the tool places the order itself. Given the right permissions, it can approve vendor payments on its own and perform any number of other tasks with autonomy.

Healthcare organizations are adopting agentic AI quickly. In a recent survey of 250 U.S. health system leaders, 28% said they already have agentic AI in production, and another 44% are piloting it. 88% expect AI agents to operate with at least some degree of autonomy.

But governance hasn’t kept pace. The same survey found that 72% of organizations have deployed AI tools or agents without formal IT approval, and only 17% believe existing identity approaches are sufficient.

We don’t have to guess what happens when AI goes wrong in healthcare organizations. The Centers for Medicare & Medicaid Services’ AI prior authorization program pilot resulted in “sweeping issues such as delayed approvals, miscommunication and provider frustration” according to Fierce Healthcare. It’s what happens when there’s a lack of governance.

Where to Start With Agentic AI Governance in Healthcare

Three things matter the most right off the bat:

Sort decisions for agents by what a mistake would cost.
Not every task an agent handles carries the same risk. Rescheduling a routine appointment and denying a claims appeal are not the same category of decision. Before you let an agent act independently, ask what happens if it’s wrong, who finds out, and how fast it can be undone. High-cost, hard-to-reverse decisions like ones that touch payments, patient care, or compliance should keep a human in the loop for now.

Build the audit trail before the agent goes live.
Gartner’s advice to finance leaders piloting AI agents applies to healthcare: treat the first deployment as a governance pilot, not a results pilot. That means achieving complete traceability of what the agent planned, accessed, and produced for every run — plus a failure log that documents issues and how they were fixed. If you can’t reconstruct what the agent did, it’s not ready to act without supervision.

Assign ownership.
A designated person needs to own each agent or agent type. Handing this responsibility to IT in a broad sense is not the way to do it. A named person should be accountable for what an agent is allowed to touch and how its performance gets reviewed. Shadow AI became a problem because organizations adopted tools informally with no one specifically responsible for them. Agentic AI without specific ownership is the same risk.

Do Agentic AI Right, and the ROI Will Come
Healthcare leaders need to think of AI governance as an unskippable step in their AI adoption process. If you’re looking for the ROI that comes with automation and agentic autonomy, the only way to truly see that return is by establishing a foundation rooted in healthy oversight. Without such a foundation, you’re looking at wasted money on tools, compliance risks, and even security risks that could be massively expensive to remediate.

Implementing a smart AI governance framework can be overwhelming for many healthcare leaders and IT departments. GuideIT has 30+ years of experience helping healthcare organizations adopt emerging technologies, and AI is no different. We help you develop policies, vet tools, train staff, and more so you can take full advantage of the exciting opportunities agentic AI offers. Reach out to ask any questions of our AI experts.