Most health systems are using AI before they have clear control over it. In 2026, 75% of U.S. health systems have deployed or plan to deploy at least one AI tool, but only 18% have mature AI governance.

That gap means one thing: AI risk is showing up faster than most teams can track it. I see the main problem as simple: many organizations can buy or turn on AI, but they still can’t clearly say who owns it, who reviews it, and who shuts it off when something goes wrong.

Here’s the short version:

  • AI risk is not covered by standard IT review alone.
  • Shadow AI is already in clinical and admin workflows.
  • Vendor tools can gain AI features after approval.
  • Most teams lack a full AI inventory and named owners.
  • Governance has to cover intake, review, monitoring, and retirement.
  • High-risk tools need tighter review, human checks, and traceability.

A few numbers make the issue hard to ignore:

  • 86% of healthcare IT leaders report shadow IT
  • More than 50% of organizations cannot detect when vendors add AI to approved products
  • 70% have AI governance committees, but only 30% keep an enterprise-wide AI inventory
  • 64% are testing or using agentic AI, while only 8% have rules built for it

What should leaders do first?

  1. Build one AI inventory
  2. Assign decision rights
  3. Set AI use and PHI rules
  4. Score tools by risk
  5. Monitor model changes, drift, and incidents
  6. Keep clear records for review and audit

If I had to sum up the article in one line, it would be this: governed AI is tracked AI, owned AI, and reviewable AI.

Healthcare AI Governance Gap: Key Statistics at a Glance

Healthcare AI Governance Gap: Key Statistics at a Glance

Where AI governance breaks down in healthcare

Shadow AI, weak vetting, and unclear accountability

The first governance failure usually isn't a cyberattack. It's much more ordinary: a clinician using an unapproved generative AI tool to draft a clinical note, or a staff member pasting PHI into a public chatbot to move a task along. That means unapproved use, activity no one can see, and no clear owner.

86% of healthcare IT executives report instances of shadow IT within their systems [3], and AI has made the problem much worse. Only 18% of healthcare professionals are aware of formal generative AI-use policies [5]. When people don't know the rules, they fill in the gaps on their own. That's how PHI ends up in places it never should have gone.

The accountability issue gets even messier inside clinical workflows. Ambient documentation tools can help, but if a generated note overstates medical necessity, the organization may face False Claims Act exposure. Billing and documentation tools need review before claims go out. If they don't, they can create upcoding risk. Recent DOJ recoveries and large reimbursement settlements show what unsupported coding can cost [1].

The problem doesn't stop there. 38% of organizations share AI risk across groups without clear escalation paths or have no defined owner at all [2]. So when a clinician spots a decision support tool giving inconsistent outputs, there may be no named person to contact. That's governance immaturity in plain terms: no approved intake, no owner, and no review loop.

The next breakdown is visibility. In many cases, organizations can't track these tools once they become part of daily work.

Weak data controls and fragmented monitoring

Even when an AI tool goes through a formal procurement process, the data controls around it are often incomplete. One major blind spot is vendor-embedded AI: over 50% of healthcare organizations cannot detect when a vendor quietly adds AI capabilities to an already-approved product [2]. Many organizations also don't have a full map of where PHI goes once it enters an AI system, whether vendor models are trained on that data, or whether deidentified data could still be reidentified through manipulated prompts [5].

The monitoring gap is just as serious. Silent vendor updates, workflow workarounds, and model drift can all change how a tool behaves after deployment. But in most organizations, there's no way to catch those shifts.

70% of healthcare organizations have established AI governance committees, yet only 30% maintain an enterprise-wide AI inventory [2]. As Ed Gaudet, CEO of Censinet, puts it:

"Healthcare has built the governance scaffolding for AI, but the operational muscle - inventory, asset management, detection methods, and clear accountability - is not keeping pace with adoption." [2]

That visibility gap leaves leaders making governance calls about tools they don't even know are in use. 64% of healthcare organizations are already experimenting with or deploying agentic AI - autonomous systems that can trigger workflows and access databases on their own - but only 8% have established specific governance for them [2]. These systems don't fit neatly into standard identity and access management frameworks, and most organizations haven't updated their controls to deal with them.

Without inventory and continuous review, leaders can't govern tools that keep changing after approval.

Why legacy governance falls short for AI

Standard IT governance was built for software that behaves the same way every time it runs. AI doesn't. Legacy IT governance is a point-in-time process: assess the tool during procurement, assign role-based access, and move on. AI governance has to be continuous, model-aware, and tied to named owners. It also has to account for model versions, training data provenance, silent capability updates, and autonomous agents operating outside standard access controls.

That's the core gap. AI needs a governance structure built for continuous intake, review, and oversight.

How Health Systems Can Safely Adopt AI: A Proven 5‑Pillar Governance Framework

A practical governance structure to close the gap

The gap closes when AI governance becomes part of day-to-day operations. In plain terms, governance has to approve, monitor, and retire AI tools, not just comment on them.

Build an AI governance committee with clear decision rights

The problem usually isn't that a committee doesn't exist. It's that the committee has no teeth. If a group can review an AI tool but can't stop it from going live, that's not governance. That's paperwork. The committee needs formal authority to approve, modify, or decommission any AI tool, and that authority should live in written policy.

For that to work, the committee should connect to an enterprise-wide AI inventory and model registry. Leaders need one place to see what's in use, who owns it, and where it's running. Membership should include clinical leadership, the CISO, compliance, privacy, legal, IT, data management, and risk management. That mix matters because one clinical support tool can trigger patient safety issues, HIPAA exposure, and cybersecurity risk at the same time.

The committee also needs two named owners:

  • A clinical lead, usually the CMO or a designated Medical Director, who owns patient safety outcomes
  • A technical lead, usually the CIO or CISO, who owns performance, security, and infrastructure

Without both, safety and security decisions split apart.

C-suite backing matters too. If executive leaders don't back the committee, it won't have the standing to overrule a business unit that's in a hurry.

Set policies for acceptable use, data governance, and model lifecycle management

A committee can't do much without written rules. Decision rights matter, but leaders also need policies that spell out how AI can be used, how data is handled, and how models move from launch to retirement. The core areas are acceptable use, data governance, model lifecycle management, human oversight, and incident response.

Acceptable use policies should draw a hard line between administrative AI, like scheduling or billing automation, and clinical AI, like diagnosis support or documentation. Those uses do not carry the same level of risk, so they should not follow the same approval path. Generative AI tools, especially the ones staff may start using on their own, need clear guardrails for PHI handling.

Data governance should define PHI handling at every step. That includes role-based access controls, encryption rules, data use agreements with vendors, and standards for de-identification. Business Associate Agreements also need updates so they spell out AI processing terms and what happens if a vendor retrains a model using your data.

Model lifecycle management is where many organizations fall short. Every deployed model needs:

  • A version record
  • A named approver
  • A validation result
  • A set review schedule

For continuously learning models, the FDA's Algorithm Change Protocol (ACP) framework gives teams a solid way to define retraining and post-deployment evaluation [7]. Retirement matters just as much. If a tool has been replaced but never formally shut down, it still creates risk.

AI incident response playbooks should cover clinical adverse events linked to AI outputs, privacy breaches tied to AI-processed data, and cybersecurity incidents involving AI systems or agentic tools. Each playbook needs a clear escalation path and a defined emergency disablement procedure so a tool can be shut off fast if needed.

These policies should line up with HIPAA and, when it applies, FDA SaMD guidance.

Standardize AI intake, risk scoring, and ongoing oversight

Those rules only hold up if every AI use case goes through the same front door. That means every AI use case - whether it's a new vendor product, an internal model, or a new AI feature slipped into an existing product - should follow one intake process. That process should capture the tool's purpose, the data types it uses, how it handles PHI, which clinical workflows it touches, what systems it connects to, and who the vendor is.

Risk scoring should be tiered. Low-risk administrative automation tools with little patient safety impact and no direct PHI processing can move through a lighter review path. Clinical support tools and anything with autonomous decisioning need deeper review, including parallel testing before go-live. Duke University Health System's ABCDS (Algorithm-Based Clinical Decision Support) framework, which uses regulatory "stop points" to manage 52 active models, shows how structured intake and lifecycle management can work at scale [7].

Ongoing oversight is where many governance efforts start to drift after deployment. Clinical outputs, data controls, security posture, bias audits, performance monitoring, and incident response all need a named owner, a set review cadence, and pre-defined triggers for re-evaluation. Those triggers can include a new model version, a change in training data, a shift in clinical workflow, or a performance metric dropping below a set threshold.

"If you cannot trace a prediction back to a versioned model, dataset, and approver, you do not have governance - you have a memo." [6]

Every AI tool should be traceable, versioned, and assigned to an owner. That's what turns governance from a policy document into something teams can actually use.

How Censinet can put AI governance into practice

Censinet

The frameworks and policies from the previous section only matter if teams can carry them out every day. That takes one system for inventory, review, and escalation.

Use Censinet RiskOps™ as a central system for AI policy, inventory, and risk workflows

Censinet RiskOps

Censinet RiskOps™ acts as a single system of record for AI-related policies, use cases, vendor documentation, inventory, and risk workflows. That directly maps to the article’s main risk areas: unvetted tools are flagged at intake, ownership gets assigned and tracked, data controls are documented, and monitoring gaps show up in real time.

Instead of having clinical, security, compliance, and IT teams work in separate lanes, everyone works from the same source. Remediation tasks can be assigned, tracked, and escalated in one place. And dashboard visibility into oversight gaps gives leaders a plain view of what’s active, what’s under review, and what’s overdue.

Speed up AI-specific third-party risk reviews with Censinet AITM and Censinet AI™

Censinet AITM

Censinet AITM helps vendors complete security questionnaires in seconds. It automatically summarizes evidence and documentation, captures key product integration details, and identifies downstream vendor dependencies. That matters because 54% of healthcare organizations have no documented method for detecting when vendors embed AI into existing, previously approved products [2].

Censinet AI™ pushes that speed deeper into the review process. It supports evidence validation, policy drafting, and risk mitigation through automation with human review. It also routes assessment findings and related tasks to the right stakeholders for review and approval, including members of the AI governance committee.

The key point here is control. Risk teams keep it through configurable rules and review processes, so automation supports critical decision-making instead of taking it over.

Ad hoc governance vs. orchestrated governance with Censinet

The gap between manual AI governance and a coordinated system tends to show up fast - during audit season, after an incident, or the moment a regulator asks for traceability. Here’s where that difference is easiest to see:

Dimension Ad Hoc AI Governance Orchestrated Governance (Censinet)
Time to review Manual, spreadsheet-based; slow and point-in-time [4] Accelerated via Censinet AITM; automated evidence validation [2]
Policy enforcement Inconsistent; policies often sit on a shelf [4] Centralized in RiskOps™; policies embedded in active workflows [2]
Traceability 40-point gap between committees and actual inventory [2] Real-time dashboard visibility into oversight gaps [2]
Audit readiness Reactive; difficult to produce traceability for regulators [1] Continuous evidence collection; centralized logs aligned to HIPAA and NIST [2]
Team burden High; siloed efforts across IT, security, and clinical teams [2] Reduced through coordinated remediation and AI-assisted task routing [2]

That’s the shift in plain English: less chasing, fewer blind spots, and a much clearer path from policy to day-to-day action.

AI governance maturity levels and next steps for healthcare leaders

5 maturity levels: ad hoc, emerging, defined, integrated, optimized

The maturity ladder below turns common governance gaps into a simple way to set priorities.

Maturity Level Key Characteristics
1. Ad Hoc Shadow AI, no inventory, reactive to vendor claims - no formal oversight or risk classification.
2. Emerging Basic policies exist, but intake, enforcement, and inventory rely on individual relationships instead of formal controls.
3. Defined Formal committee, documented approval process, and risk-based classification - but no quantitative monitoring or feedback loops.
4. Integrated Cross-functional oversight is in place, with centralized dashboards and structured vendor vetting.
5. Optimized Continuous monitoring, automated audit trails, and real-time detection of model drift or bias. Continuous improvement and expansion to new use cases.

Use this model to find your current state and decide which control to add next.

How to move up one maturity level without slowing innovation

Once leaders know where they stand, the next move is simple: go up one level at a time with a small set of controls. Start with the AI use cases that carry the most risk, then follow this sequence:

1. Days 1–30

Build a centralized AI inventory that includes pilot tools and unsanctioned shadow deployments. Assign formal decision rights to a multidisciplinary committee.

2. Days 31–60

Set approval thresholds and require documented human review for any AI output that affects care, reimbursement, or legal exposure.

3. Days 61–90

Add AI-specific questions to third-party risk reviews and update Business Associate Agreements (BAAs) to address how PHI is used for model training.

4. Days 91–180

Centralize logs, set recurring validation checks, and monitor for drift.

For high-risk tools, run them in parallel before go-live. That gives teams a clean way to compare AI recommendations against clinician decisions before any full integration.

"An AI governance committee without an AI inventory is a board meeting without an agenda." - Patrick Spencer, Kiteworks [2]

Conclusion: Governed AI is safer AI

Governed AI is safer AI because it is inventoried, assigned, and monitored. Closing the gap means putting formal governance structures in place, using risk-based intake workflows, and giving teams centralized oversight they can use every day. Censinet RiskOps™ and Censinet AI™ are built to put those controls into practice across the enterprise - to centralize inventory, review, and monitoring.

FAQs

How do we know our AI governance is immature?

Signs of immature AI governance are pretty easy to spot once you know what to look for.

A big one is the lack of an enterprise-wide AI inventory. If no one has a clear list of the AI tools in use, that’s a red flag. The same goes for unclear ownership and poor visibility into data use. If you can’t tell which tools are running, what data they can access, or who signed off on them, then oversight is mostly performative.

The problem doesn’t stop at go-live, either. It often shows up in weak monitoring for performance, drift, or bias. You may also see missing audit trails, no override logs, and review processes that don’t line up with actual clinical workflows.

Which AI tools should be reviewed first?

Start with a broad discovery process to identify every AI application, agent, and conversation in use. That includes formal deployments, pilot programs, and unsanctioned shadow AI. Then sort each use case by risk level.

Put immediate oversight in place for high-risk uses tied to diagnosis, clinical treatment, billing, coding, consent, and patient-facing recommendations. Any output that could materially affect a patient or a treatment decision should require documented human review.

Who should own AI decisions in a health system?

AI decisions in a health system should follow dual accountability.

That means one clinical leader, such as the Chief Medical Officer or Medical Director, is responsible for clinical safety. At the same time, one technical leader, such as the Chief Information Officer or Chief Digital Officer, is responsible for system performance and data security.

Each AI tool should also have a named owner. That person is on point for procurement, validation, monitoring, and incident response.

On top of that, a multidisciplinary AI advisory board should provide oversight. This helps keep decisions grounded in both patient care and system needs.

Related Blog Posts