The QA and VV Gap. Verifying AI Vendors Before They Touch Patient Data.

A SOC 2 report, a demo, and a security questionnaire do not prove an AI tool is safe for patient data. If you approve an AI vendor before checking intended use, PHI flow, model testing, subgroup results, logging, deletion rules, and BAA terms, you can miss risks that only show up in care settings.

I see the article making one clear point: PHI access should come last, not first. Before any patient record reaches an AI vendor, you should confirm:

• What the tool is for and what it is not for
• What PHI it receives, stores, logs, or sends to others
• How the model was tested, including local and subgroup results
• Whether PHI can be used for training
• How long data stays and how deletion is proved
• Whether a signed BAA is in place before any pilot
• Who monitors the model after go-live

A few facts stand out. Business associate involvement in large healthcare breaches rose from 20% in 2009–2017 to 43% in the first half of 2026. And vendor incidents exposed about 75% of people affected by major health data breaches in 2024. That tells me weak vendor review is no longer a side issue.

The article also makes a sharp distinction between basic QA and true AI verification and validation. Basic QA asks whether a vendor has baseline security controls. AI V&V asks whether the tool works safely for your use case, your patients, and your workflow. That includes things like hallucination rates, subgroup gaps, alert burden, drift checks, audit trails, and clinician review steps.

Here’s the short version of the review model:

• Gate 1: Validation
  • intended use
  • testing method
  • subgroup results
  • local fit checks
• Gate 2: PHI handling and contracts
  • data-flow maps
  • subprocessors
  • retention and deletion
  • BAA and data-use limits
• Gate 3: Approval
  • security
  • privacy
  • compliance
  • procurement
  • clinical sign-off

I’d sum it up like this: a secure vendor can still be unsafe for patient care. The right question is not “Did they pass security review?” It’s “Did we verify that this AI tool can touch patient data without creating new clinical, privacy, or compliance risk?”

Vendor Due Diligence What’s Really Under the Hood of ‘Powered by AI’

sbb-itb-535baee

The Problem: Where AI Vendor Reviews Break Down

Most healthcare AI reviews still end with a questionnaire, a SOC 2 report, and a demo. That may work for general software checks. It does not tell you whether a tool is safe for clinical notes, risk scoring, or patient encounter audio. And it does not give providers the proof they need before any PHI starts moving through the system.

Approval Shortcuts That Create Risk

One of the biggest mistakes is treating HIPAA compliant as approval. HIPAA compliant is a vendor claim, not a certification.^[3][8][10] That label says nothing about what happens to PHI during inference, whether prompts are logged, or whether patient data flows back into model training.

The next common mistake is leaning on SOC 2 alone. A SOC 2 audit documentation checklist looks at general security controls. It does not cover hallucinations, subgroup performance, or how PHI may sit in logs, backups, or derived data.

Other shortcuts can cause problems later, and they often slip by at the review stage. Teams may skip subgroup performance and bias review and approve a tool on one top-line accuracy number. That can hide harm. A readmission model, for example, may still work against patients from lower-income ZIP codes if the model leans on proxies for socioeconomic status. Teams also miss vendor subprocessors all the time - the transcription engines, LLM hosting platforms, and logging services behind the main product. Each one may touch PHI, yet many reviews never map them out.^[4][7]

Approving a pilot before a BAA is signed is a hard stop. Any vendor that creates, receives, maintains, or transmits PHI is a business associate under HIPAA and must sign a BAA before PHI flows.^[1][9][10] A pilot without a BAA is a compliance failure and a breach risk. If any of these shortcuts show up, the review should stop.

Red Flags That Should Pause or Stop Approval

These are not small issues. They are warning signs that should halt approval, not invite a few extra follow-up emails.

The breach data makes this hard to ignore. Business associate involvement in large healthcare breaches rose from 20% in 2009–2017 to 43% in the first half of 2026.^[5] Vendor incidents now hit a huge share of people affected by health data breaches - about 75% of all individuals impacted by major health data breaches in 2024 were exposed through vendor incidents.^[6] That’s the price of waving through weak reviews.

Basic QA Artifacts vs. True AI V&V Evidence

The gap here is not about having more paperwork. It’s about having proof that the tool is fit for use.

Ambient scribes make this even messier. These tools do not just produce a final note. They can also create audio recordings, intermediate transcripts, and derived metadata. All of that counts as ePHI under HIPAA.^[2] So if a review looks only at the EHR note, it misses most of the data the tool creates and stores. That expands the PHI footprint far beyond what many teams account for.

The next step is setting a pre-PHI verification framework that closes these gaps.

The Solution: A Pre-PHI Verification Framework for AI Vendors

To close the QA/V&V gap, put three pre-PHI gates in place: validation evidence, PHI handling and contracts, and formal approval checkpoints. No tool should go live until it clears all three.

Validation Evidence: Intended Use, Performance, and Local Fit

Before any PHI moves through a tool, require a written intended-use statement. That statement should spell out the clinical use, who will use it, the scope, and which decisions it may inform.

Then ask for the validation record behind the tool. That means documentation on the training data scope, whether the tool was externally validated, how it was evaluated, and which performance metrics were used for the actual use case - accuracy, sensitivity, specificity, false positive rates, false negative rates, and calibration when it applies.^[13][15]

Aggregate accuracy by itself doesn't tell you enough. A model can look fine overall and still do a poor job for part of your patient population. That's why you should require subgroup performance tables broken out by age, sex, race/ethnicity, disease severity, site, or device configuration. The goal is simple: confirm that the tool does not perform much worse for certain groups.^[13][15]

You also need to check whether the tool fits your own workflows and patient population before a broad rollout. Ask for retrospective testing against your historical data. Then use silent-mode testing that runs in parallel without changing care. This gives you a safer way to see whether outputs stay stable, useful, and safe in your local setting.^[13][15]

Once the model lines up with the intended use, the next step is to make sure the data path and the contract line up with that same scope.

PHI Handling, Security Controls, and Contract Terms

Start by mapping the full PHI path. Review encryption in transit and at rest, role-based access controls, audit logging, and clear timelines for retention and deletion.^[11][12][14] Ask for architecture diagrams and data-flow maps that show where PHI is captured, processed, stored, and sent. That includes cloud regions and every named subprocessor.^[11][12][14]

A signed BAA is required, but it doesn't settle the whole issue. You still need to review the data-use language and make sure it matches the approved scope rather than the vendor's default terms. Why? Because standard vendor terms may still keep broad rights to use customer data for model training, product improvement, analytics, or prompt retention.

The BAA and any related data-use agreement should clearly prohibit training on PHI, define permitted uses and disclosures, set breach notification timelines, and require the return or destruction of PHI when the contract ends.^[11][12][8]

Approval Checkpoints and Go/No-Go Criteria

A defensible go/no-go call needs cross-functional sign-off. In plain English, one group shouldn't make this call alone. Turn the evidence into a formal checklist so that security, privacy, compliance, procurement, and clinical review all come together in one decision.

A true go decision means the basics are already in place: a signed BAA, a completed security review, acceptable validation results, documented safety guardrails, approved data flows, and named owners for post-deployment monitoring.^{[11][12][13][15]} If any of those pieces are missing, the answer should be no-go - or a conditional go backed by a written remediation plan and a hard deadline.

For higher-risk tools, documented local-fit testing should be required before live use.

Applying the Framework to Ambient Scribes, Assistants, and Analytics Tools

Use the same three gates for every AI tool, but ask for different proof based on the use case. The framework does not change. The evidence does. An ambient scribe and a sepsis model should not be judged with the same checklist, even if both handle PHI. Reviewers need a clear sense of which questions fit which tool.

Ambient Documentation and Clinical Assistants

Ambient scribes become business associates under HIPAA as soon as they handle PHI for a covered entity, so a signed BAA is required before use.^[16][17][18] What sets ambient scribes apart is the PHI they leave behind outside the final note. In addition to the finished documentation, they may create audio files, draft transcripts, and prompt/response logs. Under HIPAA, all of that counts as ePHI.^[2] So the review has to go far beyond a vendor’s normal QA process.

Ask the vendor to spell out how recording starts, how it stops, and who can control it. That includes limits on where recording can happen, such as exam rooms only and not public spaces, plus UI safeguards and role-based permissions. The system also needs to apply one-party or two-party consent rules based on location or policy profile, and it should log every consent event for audit review. Get the retention schedule in writing for raw audio, draft transcripts, and prompt/response logs. Then confirm the vendor can enforce deletion at the system level and produce audit records showing that deletion took place.

No AI-generated draft should go into the medical record without clinician review and sign-off.^[17][18][19] That means there should be a clinician approval step before note insertion, along with a review queue for low-confidence text. Also confirm that the tool cannot place orders, change medications, or edit problem lists unless a clinician takes a separate action.

Analytics tools call for a different kind of review. The focus shifts away from recording controls and toward data quality, model behavior, and workflow effects.

Analytics and Decision-Support Models

For analytics and decision-support tools, start with data integrity and workflow impact. Ask for a data lineage summary that shows which source systems fed the model, what date range those systems cover, and how missing values, outliers, and implausible entries were handled. Label quality matters just as much. Ask who created the outcome labels, how inter-rater reliability was measured, and whether clinical experts reviewed the labeling method.

Subgroup performance is a common weak spot from vendors. A model can post strong top-line metrics and still do much worse for certain patient groups. Ask for stratified results by race/ethnicity, sex, age band, language, and insurance type. If the vendor cannot produce those tables, that gap needs to be fixed before deployment.

Alert burden also needs direct review. In plain terms, this is the number of alerts clinicians have to deal with. High false positive rates can lead to alert fatigue and create patient safety risk.^[21][22][23] Ask for past or simulated data on alert volume per 100 admissions or per clinician shift, along with override rates. Make sure thresholds are configurable so your team can tune sensitivity for your own patient population.

The main question is not accuracy by itself. It is whether the outputs can be trusted in local workflows and whether bad predictions can cause downstream clinical harm. AI outputs should be labeled clearly, show key drivers when possible, and let clinicians dismiss or override them.^[20][22][23] The system should also log who viewed, accepted, changed, or rejected each output.

Use the table to match each AI use case with the proof that must be in place before PHI access starts.

AI Use Case vs. Required Verification Artifact

The table below maps each AI use case to the minimum review artifacts before PHI access.

Turning Vendor Review Into Continuous AI Governance

Passing an initial review does not mean a vendor stays approved. Governance has to keep going. It can't be a one-time gate.

Vendors can change data-use terms and other AI terms over time. Because of that, review has to continue after deployment.

Monitoring AI Vendors After Deployment

PHI access should depend on current evidence, not only what a vendor shared at onboarding. Once a tool goes live, the job shifts from approval to active oversight.

Build clear checkpoints into the vendor management cycle. Require advance notice of any change to data use, the model, or logging, and treat each one as a trigger for reassessment. That way, reassessment becomes part of the approval itself, not some rare exception.

On the provider side, schedule periodic reviews of:

• audit trails
• incident reports
• updated validation evidence

The initial validation package should serve as the baseline for later reassessment. For high-risk clinical AI, require per-inference audit trails so you can see what the system sent and received during use. If a vendor cannot produce current evidence when reassessment comes up, suspend PHI access until the gap is closed.

Only 16% of health systems currently have an enterprise-wide AI governance strategy, and 86% of healthcare IT executives report shadow AI use inside their organizations ^[24]. Without a structured reassessment cycle, approved tools and unapproved ones can start to look the same from a risk standpoint.

Using Censinet to Standardize AI Vendor Review

The hard part usually isn't finding evidence. It's keeping one record that every team can rely on.

This is where many programs start to wobble. Security, compliance, legal, procurement, and clinical teams often track different files, different versions, and different decisions. Censinet centralizes evidence collection, BAA checks, and risk documentation in one record of truth ^[24].

Conclusion: What a Defensible Go/No-Go Decision Requires

A defensible approval means having documented evidence that the tool passed pre-PHI review and remains under monitoring.

Start with a live AI tool inventory that lists:

• the tool
• the vendor
• the intended use
• the PHI touched
• the owner

A defensible approval requires proof that the tool cleared the pre-PHI gates and is still under active review.

The strongest AI approvals in healthcare are evidence-based, cross-functional, and built to be revisited. QA matters, but V&V and ongoing monitoring are what make PHI access defensible. No AI vendor should access patient data without clearing the required verification steps, and no approval should be treated as permanent.

FAQs

What’s the difference between basic vendor QA and AI V&V?

Basic vendor QA covers standard security and compliance checks, like encryption, SOC 2 reports, and access controls. Those checks matter. But they don't deal with AI-specific risks such as model drift, bias, or algorithmic errors.

AI V&V goes a step further. It asks for proof that the model performs well in clinical settings, including validation studies, bias testing, intended use, limitations, and ongoing monitoring.

Why should PHI access only follow AI vendor verification?

Access to PHI sets off HIPAA rules, including a signed Business Associate Agreement (BAA). If you share PHI with a vendor before that’s in place, you’re looking at a HIPAA violation.

Verification also gives you a way to check whether the vendor’s claims hold up against AI-specific risks. That includes training on patient data, data leakage, unauthorized subprocessor access, and biased or unvalidated outputs.

What documents should we require before an AI vendor touches patient data?

Before any AI vendor touches patient data, require a few non-negotiables.

You need a signed HIPAA BAA. You also need model documentation, plus clinical validation results that show the model’s known limits. That part matters. If a tool works well in one setting but falls short in another, you want that spelled out upfront.

Ask for training and testing documentation too. That should cover data provenance, demographic mix, and how PHI/PII was handled during development and evaluation. In plain English: where the data came from, who was represented in it, and what was done to protect sensitive information.

The vendor should also provide a PHI/PII data-flow diagram. That gives you a clear map of where data goes, who can access it, and where it’s stored or processed.

On top of that, require security and privacy evidence, along with contract terms that spell out:

• how PHI can be used
• retention and deletion rules
• audit rights
• incident timelines
• model update terms

This isn’t paperwork for the sake of paperwork. It’s the baseline for knowing what the vendor is doing with patient data, what risks come with the model, and what happens if something goes wrong.

Related Blog Posts

• Healthcare AI Data Governance: Privacy, Security, and Vendor Management Best Practices
• The Third-Party AI Problem: Vendor Risk in an Algorithm-Driven World
• The AI-Ready Organization: Cultural and Technical Prerequisites for Success
• The New Front Line in Healthcare Cybersecurity Is AI Governance