AI in healthcare is now a control problem, not a pilot project. If I work in a health system, clinic, or provider group, the message is plain: I need an AI inventory, named owners, vendor rules, audit logs, and routine checks now.
Here’s the short version:
- AI is already inside daily work like notes, coding, billing, and patient messages.
- Risk is now tied to PHI, billing errors, and staff use of unsanctioned tools.
- NIST AI RMF, ONC HTI-1, HIPAA, and state laws are setting the bar at the same time.
- March 1, 2026 was a key HTI-1 date for certified health IT updates.
- Texas TRAIGA took effect on January 1, 2026, with patient AI disclosure rules.
- HIPAA penalties can reach $2.134 million per identical violation, per year, for willful neglect.
- 52% of organizations have AI agents or machine identities with critical excess permissions, versus 37% for human users.
What I take from this is simple: policy alone is not enough. I need to know:
- Who approved each AI tool
- What patient data it touches
- Whether the vendor trains on that data
- How outputs are checked
- When a tool must be paused or shut off
The article’s core point is clear: healthcare groups have to turn national rules into day-to-day controls across buying, setup, use, logging, and review. That is now the standard.
Governance, Compliance, and Risk Management for Healthcare AI Agents
sbb-itb-535baee
What National Standards Now Require for AI Governance
These standards now set the controls healthcare teams need to buy, deploy, monitor, and document AI. Each one maps to a different control layer: governance, disclosure, and privacy/security.
NIST AI RMF as the Working Model for AI Risk Management
The NIST AI RMF's four functions - GOVERN, MAP, MEASURE, and MANAGE - often get treated like a paperwork exercise. That's the wrong read. Each function points to a concrete operating control:
| NIST Function | What It Requires |
|---|---|
| GOVERN | Name an executive who owns AI governance and set up a cross-functional committee - Legal, Clinical, IT, and Privacy - with actual escalation power. |
| MAP | Keep a complete AI inventory, grouped by model type and risk. |
| MEASURE | Set validation standards for accuracy, hallucination risk, and model drift; run bias testing and red-team reviews on a set schedule. |
| MANAGE | Create incident response playbooks for AI-specific failures and define clear stop criteria for pausing or shutting down a system. |
Many organizations get stuck at GOVERN. A committee with no power to stop a system isn't governance. The control model should match the impact of the decision and how easy it is to reverse.
That model gets more concrete under HTI-1, which turns transparency into a buying and documentation rule.
ONC HTI-1 and the New Bar for Transparency and Intervention Risk Management
HTI-1 requires source attributes and public intervention risk summaries for Predictive Decision Support Interventions (DSIs). [2][3] More specifically, developers must provide access to 31 structured source attributes and publish public Intervention Risk Management (IRM) summaries that explain how risks are reviewed, reduced, and governed. [2][3]
Procurement teams should ask for these materials before any contract is signed. Internal oversight teams also need immutable audit trails for every AI-generated clinical recommendation, including clinician overrides and the reason for them. [3] That point matters more than it may seem at first glance. If no one can trace what the system suggested, what the clinician changed, and why, oversight starts to fall apart.
One design issue stands out: human review only works when the interface shows the source material next to the output. When that context is buried, clinicians are more likely to click through and approve by habit. That undercuts oversight completely. [4]
Those disclosure controls still sit inside HIPAA's privacy and security rules, which decide how AI can interact with electronic protected health information (ePHI).
HIPAA, HHS/OCR, and Federal and State Expectations for AI Tools
HIPAA's current privacy and security rules apply in full to AI systems that touch ePHI. Every AI vendor that handles PHI needs a BAA tied to the actual workflow. [4] Audit logs also need to cover all PHI interactions across the full seven-year retention window, including ambient documentation sessions, coding suggestions, and patient-facing communications. Teams should also record whether each code was AI-proposed or human-amended. That detail matters under False Claims Act scrutiny. [4]
On top of that, organizations should require:
- Strict retention and deletion controls
- No vendor model training on patient data
- Section 1557 compliance for accessible, non-discriminatory communications
These are not side issues. They shape how AI tools can be used in day-to-day care and how much risk a health system takes on. [4][2][7]
Where Healthcare Organizations Still Have AI Governance Gaps
Standards set the bar. Day-to-day work has to clear it.
That’s where many healthcare organizations still stumble. Governance only works when decision rights are clear, procurement has guardrails, and teams keep watching systems after launch. National standards now expect AI governance to hold up in production, not just sit in a policy binder.
Fragmented Ownership, Incomplete Inventories, and Weak Accountability
A common failure looks simple: there’s a policy, but no named owner and no one with the power to stop a deployment.
When that happens, accountability gets scattered. The AI inventory ends up partial, outdated, or both. One team assumes another team is tracking risk, and the result is a mess no one fully owns.
Administrative AI often slips in through normal SaaS purchasing paths, which can bypass BAA review and AI-specific risk assessment [4]. That makes administrative AI the blind spot.
Vendor Oversight and Documentation Controls Are Often Underdeveloped
Even when procurement teams do talk with vendors, the paperwork often falls short. Generic BAAs may not address GenAI data flows in a meaningful way. On paper, the contract may look fine. In practice, the data handling can tell a different story [4].
Another weak point shows up before launch. Many organizations put GenAI tools into use without a defined validation gate before go-live. That’s a problem because procurement controls can’t sit off to the side as a separate compliance task. They need to be built into governance from the start.
Monitoring and Internal Controls Are Not Keeping Pace With Production AI Use
Post-deployment controls often trail behind how people use AI in the real world. Human review can turn into rubber-stamping when the interface shows a recommendation but hides the source material behind it. At that point, the reviewer may be approving output without seeing enough to judge it.
Teams need continuous monitoring for drift, bias, and performance drops in specific patient groups across the full lifecycle [5].
Some use cases need tighter checks than others. Coding assistants, for example, need baseline checks for upcoding drift and False Claims Act exposure [4].
Cybersecurity risk is growing at the same time. Fifty-two percent of organizations possess AI agents and other machine identities with critical excessive permissions, compared with 37% for human users [1]. That gives attackers more room to work and adds exposure to prompt injection, model poisoning, system prompt leakage, and excessive agency - when AI systems take unauthorized actions [7].
Taken together, these gaps point to the same issue: procurement, monitoring, and incident response can’t run as separate tracks. They need one governance model.
How to Redesign AI Governance Around National Standards
AI Governance Maturity Levels in Healthcare: Minimal vs. Intermediate vs. Advanced
You close these gaps by using existing standards in day-to-day work. The fix is pretty simple in concept: turn those standards into clear ownership, tighter vendor controls, and steady monitoring.
Build a Standards-Aligned Governance Model for AI Across the Lifecycle
Before any AI tool goes live, spell out who can approve it, pause it, and shut it down. Those three decision rights should be written down and tied to named individuals. That’s the base layer. If that part is fuzzy, the rest of the governance model can fall apart fast.
Next, set up a cross-functional AI governance group. Give named owners responsibility for specific risk categories, not just for tools. That distinction matters. For instance, one person may own false negative rates in a clinical decision support model, while another is responsible for bias monitoring for a patient group that could be affected. That kind of precision is what the NIST AI RMF MAP function calls for [5].
The shift from informal governance to a standards-aligned model looks like this:
| Governance Area | Before NIST AI RMF / ONC HTI-1 | After Adoption |
|---|---|---|
| Decision rights | Informal, project-by-project | Written authority matrix with escalation paths [5] |
| AI inventory | Partial spreadsheets, siloed by department | Centralized registry with named risk owners [5] |
| Risk classification | Binary (clinical vs. non-clinical) | Tiered by use case, error consequence, and affected population [5] |
| Transparency | Vendor summaries | Standardized source, training, subgroup performance, and known limits [3] |
| Monitoring | Pre-deployment validation only | Continuous drift and subgroup performance tracking [5] |
Higher-risk use cases like ambient documentation, claims and coding, and patient communications need tighter approval gates. They also need explicit residual risk decisions before go-live.
Once decision rights are in place, procurement becomes the main point of control.
Update Procurement and Vendor Risk Management for AI-Specific Due Diligence
Generic vendor questionnaires don’t work well for AI. They skip over the issues that matter most. Contracts need to state the exact product tier, list the subprocessors that handle PHI, and make clear whether the vendor can use PHI for training or fine-tuning.
Beyond BAA scope, contracts should also require vendors to notify your organization about major model version changes or performance drift. AI systems can weaken as patient populations change. If the contract doesn’t require notice, your team may not know the model shifted until results start slipping [5] [3].
And contracts alone aren’t enough. Teams need a way to track model changes, drift, and exceptions after launch.
| Maturity Level | Characteristics |
|---|---|
| Minimal | Generic BAA templates; one-time risk assessments; reliance on vendor self-attestation; no AI-specific policy [4] [1] |
| Intermediate | Workflow-specific BAAs; AI inventory established; manual disclosure workflows; clinician approval required for clinical decisions [4] [3] |
| Advanced | PHI segregation controls; continuous drift monitoring; automated HTI-1 disclosure; immutable WORM logging; AI-specific insurance and indemnification [6] [8] [3] |
Centralize Monitoring, Documentation, and Internal Controls With Censinet
Centralizing these controls helps close the gap between policy and daily execution. Monitoring is what keeps production AI in line with NIST, HTI-1, and HIPAA expectations. Spread-out governance sounds fine on paper, but it doesn’t scale well.
Censinet RiskOps™ brings AI inventories, risk assessments, remediation workflows, and cybersecurity benchmarking into one place, so compliance, privacy, and security teams can work from a single source of truth. Censinet AI™ speeds up assessments by letting vendors complete security questionnaires in seconds. It also summarizes evidence and documentation, captures integration details and fourth-party risks, and generates risk summary reports from the full assessment record. Human review still stays in place at key steps.
Censinet AI also supports routing and orchestration. It sends findings, approvals, and corrective actions to the right stakeholders.
| Standard | Requirement | Internal Control |
|---|---|---|
| NIST AI RMF – GOVERN & MAP | Named accountability owners per risk category | Centralized AI inventory with owner assignments and escalation paths [5] |
| ONC HTI-1 | Predictive DSI transparency at point of care | Automated disclosure of source data, training data, subgroup performance, and known limits [3] |
| HIPAA Security Rule | Technical safeguards | Mandatory AES-256 encryption and MFA for all AI systems and API keys [8] |
| HIPAA Privacy Rule | Minimum Necessary standard | Prompt engineering that restricts PHI sent to LLMs to only the fields required for the task [8] |
| HHS/OCR | Audit controls | Tamper-evident logging of all AI inference calls with six-year retention [8] |
Conclusion: What Healthcare Leaders Should Do Now
The move from AI experimentation to day-to-day control is done. AI governance is no longer a “later” project. National standards now treat it as an operating control, and penalties for willful neglect can reach up to $2.134 million per identical violation, per year [4].
So the next step is simple: move from policy to ownership.
Start with a full AI inventory, including shadow AI. Put a named owner on each tool. Require workflow-specific BAAs, no-training clauses, and human review for outputs tied to clinical or financial decisions.
AI readiness means safe, compliant, repeatable use.
Once ownership and vendor controls are set, governance can’t stay stuck in a one-time review. It needs to become continuous monitoring. Review models every quarter against defined thresholds, and keep tamper-evident audit logs for the full required retention period [5] [8].
Operational governance is what separates safe scale from unmanaged exposure. The aim is controlled adoption: governance strong enough to cut risk, while still flexible enough to support AI use across clinical and administrative workflows. National standards are pushing healthcare organizations toward governed, repeatable, auditable AI operations - and that’s the bar to meet now.
FAQs
How do we start an AI inventory?
Start with a shared, organization-wide definition of what counts as an AI tool. That definition should cover predictive models, generative AI, and AI features built into platforms you already use.
Next, ask department heads to identify AI tools in their teams, including third-party applications. Use the HHS FY25 AI Use Case Inventory as a template so everyone works from the same format.
This shouldn’t be a one-and-done audit. Treat it as a continuous process tied to procurement and contract reviews, so new tools get flagged before they quietly slip into day-to-day work.
Which AI tools need the strictest controls?
The strictest controls apply to AI used in clinical decision-making, patient-facing interactions, or the handling of PHI. That covers tools used for diagnostics, care plan generation, and clinical documentation, including ambient note-takers.
In plain terms, the closer an AI system gets to patient care, the tighter the rules get.
These tools need human-in-the-loop oversight, strong BAAs, and documented risk analyses to support HIPAA compliance and meet changing federal standards. Autonomous AI agents and mental health chatbots also face heavier scrutiny because their clinical impact can be serious, and so can the liability.
What should we require from AI vendors?
Require more than generic agreements. Put clear AI transparency and accountability terms in the contract itself, not in sales decks or website copy.
Include terms for:
- BAAs that spell out AI-related data use and any subprocessors
- Model documentation covering intended use, training data, known limits, and failure modes
- Validation data broken out by relevant demographic subgroups
- Update notice before model changes
- Access for monitoring
- The right to run in-house evaluation
- The right to roll back or suspend use without penalty if performance falls short
That matters because polished marketing language is cheap. Contract language is what gives you something to point to when a model drifts, a vendor changes a subprocessor, or results start slipping for one patient group but not another.
You want the paper trail to match the pitch. If a vendor says the model is safe, tested, and fit for a given use case, the contract should say what was tested, where it tends to fail, who had access to the data, and what happens if later versions don’t perform the same way.
And don’t settle for top-line validation claims. Ask for subgroup data that reflects the people you serve. A system that looks fine on average can still miss the mark in ways that only show up when you break results out by age, sex, race, ethnicity, language, disability, or other relevant groups.
Also, lock in your options if things go sideways. You should be able to review updates, watch performance, test the system yourself, and pause or roll back use without getting hit with a penalty. If the model underperforms, you need a clear exit ramp - not a fight over fine print.
