AI use in healthcare is moving faster than security review. By 2024, the FDA had cleared 950+ AI-enabled medical devices, while staff across hospitals and clinics were already using AI tools for notes, intake, billing, and messaging.

If I had to sum up the article in one line, it’s this: AI now needs the same risk control mindset as EHRs, medical devices, and any system that touches PHI.

Here’s the short version:

  • The main risks are shadow AI, PHI leakage, prompt injection, access sprawl, and vendor supply-chain gaps.
  • The fix isn’t one tool. It’s a clear governance process for intake, review, approval, logging, vendor checks, and reassessment.
  • HIPAA pressure is growing. Proposed Security Rule updates from December 2024 point to written risk analysis and a documented asset inventory at least every 12 months.
  • Vendors add risk too. AI features can appear inside software you already use, including EHR, imaging, and revenue cycle platforms.
  • The first move is simple: build an AI inventory, tier tools by risk, lock down access, require BAAs for PHI use, and keep humans in the loop for care-related output.

In other words, AI governance is now part of cyber risk management in healthcare. It’s how I would track what tools exist, what data they touch, who approved them, and what changed after launch.

The article makes one point very clear: if an AI tool can touch PHI or shape care, it needs review before use, not after a problem shows up.

Healthcare AI Governance - Risks, Compliance, and Frameworks Explained

The main AI-driven risks health systems must address

AI brings a cluster of risks across daily work, user access, and third-party tools. And a lot of them won't set off the usual security alarms. The three issues that need attention first are shadow AI, PHI leakage, and vendor or supply chain exposure.

Model misuse, shadow AI, and prompt injection in workflows that handle PHI

When clinicians or admin staff can't get an approved tool that fits the way they work, they often go around the system and pick one themselves. That's where shadow AI starts.

A coder drafting a denial appeal in ChatGPT, a nurse using an AI scribe that IT hasn't reviewed, or a billing specialist using a public LLM for revenue cycle tasks can all send PHI outside the health system's control. The tricky part is that old-school DLP and SIEM tools often don't catch it, because the traffic can look like normal web use [5].

Prompt injection adds another layer of risk. It can push an AI tool to reveal data or produce unsafe output inside a PHI-related workflow. One example is false instructions or bad data being inserted into an AI-assisted radiology workflow, which can lead to incorrect output [3].

This is why approval and intake controls matter before any AI tool touches a clinical or admin workflow.

Data leakage, unauthorized access, and privilege creep

PHI doesn't always leave a health system in ways that look like old-style data theft. Stored prompts, AI-generated outputs, and unauthorized data flows to third parties can all expose sensitive data in ways teams may not spot right away [1][3].

Privilege creep makes the problem worse over time. AI tools connected across EHRs, billing platforms, and scheduling systems can slowly pile up permissions that no one meant to grant. If something breaks, the damage can spread much farther [3].

That makes a few controls non-negotiable:

  • Access limits to keep permissions tight
  • Audit logging to track who did what and when

Third-party AI vendor and fourth-party supply chain risk

Many health systems are already using AI features they never directly bought. EHR updates, revenue cycle platforms, and imaging software have started adding generative AI features that weren't there when the first contracts were signed [5][2].

That sounds small on paper, but it's a big deal in practice. These add-ons often skip a new security review.

And even if a vendor has been checked, the risk doesn't end with that company. AI vendors may depend on outside LLM providers or subcontractors. That creates fourth-party dependencies, which can make data handling and retention hard to see clearly [5][1].

Old vendor reviews often miss AI-specific issues like model poisoning, data leakage, and evasion attacks, especially when the AI comes through downstream providers [1]. A proper AI vendor review needs plain answers to three questions:

  • What data is being used?
  • Where does it go?
  • Who can change the model?

These risks point straight to the controls health systems need next: intake review, access limits, logging, and vendor oversight.

Governance controls that reduce breach risk

AI Governance Lifecycle for Healthcare: From Intake to Reassessment

AI Governance Lifecycle for Healthcare: From Intake to Reassessment

AI governance needs a lifecycle process: approve, control, monitor, and retire tools that touch ePHI. The next job is turning that lifecycle into day-to-day controls.

Set policy and intake controls before approving AI tools

An AI use policy is the starting point. It should spell out which tools are approved, what PHI handling is off-limits, and what should trigger a formal review. That includes any tool that accesses ePHI, influences a clinical decision, or connects to an EHR. Staff also need a clear path for escalation, so they know how to ask for review before they start using a new AI tool.

Policy on its own isn't enough. Intake controls need to sit right next to it. Before any AI tool goes live, it should pass through a documented request and review process. In plain terms, that means recording the intended use case, the data the tool will access, and who will use it.

Apply model risk assessments, access restrictions, and audit logging

Not every AI tool creates the same level of risk. Risk tiering helps teams spend more time on the tools that can do the most harm. High-risk tools, especially those that access ePHI or affect patient care, need deeper validation before launch. That should include a review of intended use and clinical risk [2].

On the technical side, two controls are non-negotiable: least-privilege access and audit logging. Role-based access limits help contain ePHI exposure. Mandatory multi-factor authentication (MFA) for any system touching patient data adds another layer of protection. Logging should show what the AI tool did, what data it accessed, and when. That record matters when teams need to track output changes over time.

Once access and logging are in place, vendor review becomes the next checkpoint.

Strengthen vendor due diligence and human oversight

Every AI vendor that touches ePHI must sign a Business Associate Agreement (BAA). If a vendor refuses to sign a BAA, keep it out of PHI workflows. Beyond the BAA, vendor review should look at training data, change control, and fourth-party dependencies [4].

For adaptive AI tools that learn over time, require a documented change-control plan for approved model updates. That plan should define what changes are allowed after launch and how those changes will be tracked [2]. Human review is still part of the process. Even automated AI systems need defined review points, especially when outputs affect clinical decisions.

These controls work best when risk, compliance, IT, and clinical teams follow one shared process.

How to run AI governance across risk, compliance, and IT teams

Build a cross-functional model for AI decisions

AI governance needs a standing operating model, not a one-time policy. That model should spell out who reviews AI, who approves it, and who checks it again over time.

The governance group should include cybersecurity, compliance, legal, clinical engineering, and supply chain. Each team needs clear ownership for procurement, approvals, monitoring, and retirement.

Each function has a different job. CISOs and security teams handle day-to-day risk and compliance controls. Compliance teams focus on AI-specific BAA terms. Legal and supply chain write and enforce AI contract terms during procurement. Clinical engineering oversees AI-enabled medical devices and remote patient monitoring tools. Taken together, these teams cover the full AI lifecycle - from procurement through decommissioning [4].

One simple place to start is a shared glossary of AI risk terms. If clinical, compliance, and security teams each use the word “risk” in a different way, reviews slow down fast. A shared vocabulary helps teams move decisions forward and cuts down on confusion during escalations.

Standardize intake, review, exception, and reassessment workflows

Once ownership is clear, the process should work the same way for every request. The NIST AI Risk Management Framework (AI RMF) gives teams a practical structure built around four functions: Govern, Map, Measure, and Manage [2]. In a healthcare setting, that turns into a repeatable workflow.

Workflow Stage What Happens NIST AI RMF Function
Intake & Classification New AI request submitted; use case, ePHI access, and risk profile documented Map
Risk Assessment Tool tiered by HIPAA sensitivity and clinical risk; BAA status confirmed Map / Measure
Approval or Exception Committee reviews findings and documents the decision Govern
Reassessment Inventory and risk analysis reviewed at least annually; performance drift and vendor changes checked Measure / Manage

Exceptions should be documented on their own and given an end date. If a team needs to use an AI tool before it clears the full review process, that exception should be logged. Otherwise, it’s easy for short-term workarounds to turn into long-term blind spots.

Use centralized risk operations to maintain visibility and follow-through

Governance falls apart when findings, tasks, and approvals are scattered across spreadsheets, email threads, and department folders. That kind of setup makes follow-through hard and leaves teams guessing about status, ownership, and next steps.

Censinet RiskOps™ gives teams a central hub where AI-related policies, vendor assessments, remediation tasks, approvals, and monitoring data live together. With central tracking, approvals, exceptions, remediation, and reassessment stay in one audit trail. That same audit trail supports ongoing monitoring and makes reassessment faster when tools, vendors, or workflows change.

Conclusion: A practical path to safer AI adoption in healthcare

Health systems are past the point of debating whether they'll use AI. The issue now is how they'll govern it. And those controls only hold up when AI is tracked, tiered, and reviewed as part of the security program. In plain terms, AI governance needs to work like a core cyber control, not some side policy that sits on a shelf.

Start with high-risk workflows and scale governance from there

Begin with a full AI inventory. Use network discovery and team surveys to find every AI tool in use, including features tucked inside EHRs, revenue cycle platforms, and imaging software that staff may not even realize are AI. [5][2]

Then focus on the workflows where failure would bring the most patient and breach risk. Use the inventory and risk tiers to zero in on the areas most exposed to PHI, clinical harm, or vendor dependence. Clinical AI that shapes diagnosis needs tighter review than admin tools that draft emails. [2] Tackle the highest-risk workflows first, then expand from there.

Make AI governance part of your ongoing cyber risk management program

AI governance isn't a one-and-done project. Models drift as patient groups and data patterns change. Vendors may update algorithms without notice. New tools pop up in departments faster than procurement can review them. That's why continuous monitoring matters more than a one-time assessment.

Fold AI oversight into the same programs that already handle cybersecurity, privacy, and compliance, with clear ownership, measurable controls, and a set reassessment cadence. Recent HSCC guidance backs full-lifecycle governance from procurement through decommissioning. That puts AI governance inside healthcare cyber risk management, not off to the side.

FAQs

How do we find shadow AI in use?

Move from manual vendor reviews to proactive, continuous discovery. Inventory every AI application, agent, and conversation. Then use visibility tools to spot tools that slipped past normal procurement. It also helps to ask users to self-report unauthorized applications so you can baseline current usage.

Vendors can add AI features to approved software without saying much about it. Because of that, continuously monitor vendor data access patterns for behavior changes. Keep your technology asset inventory up to date at least once a year to help surface hidden deployments that handle protected health information.

Which AI tools need the highest level of review?

In healthcare, AI tools should be grouped by safety impact. High- or critical-risk tools need the most careful review.

That includes tools that handle PHI, guide treatment decisions, or sit inside clinical workflows. In plain terms, these are enterprise technology risks. They need the same level of care you’d expect for electronic health records, medical devices, and cloud platforms.

What should we ask AI vendors before sharing PHI?

Before sharing PHI with an AI vendor, run a risk analysis for that exact deployment. Make sure the vendor will sign a BAA that covers AI inference and bars any use of your data to train or fine-tune models.

Also check a few basics:

  • Encryption, access controls, and audit logging
  • Whether data is used for model training, and only on an opt-in basis
  • Exportable audit logs, breach notification timelines, and model documentation, including training sources and bias mitigation

Related Blog Posts