AI is transforming clinical decision-making but introduces new risks that traditional security methods can't fully address. These risks include adversarial inputs, data leaks, model degradation, and supply chain vulnerabilities, all of which can directly impact patient safety. Regulatory bodies like the FDA now require AI-specific threat modeling to ensure compliance and protect sensitive health data.
Key takeaways:
- Adversarial Inputs: Attackers can manipulate data to trick AI models, leading to incorrect diagnoses.
- Data Leaks: AI models can inadvertently expose patient information through outputs.
- Model Drift: AI performance can silently degrade over time, risking patient outcomes.
- Third-party vendor security risks: Dependencies on external libraries and models increase vulnerabilities.
- Mitigation Strategies: Include input validation, continuous monitoring, human oversight, and strong access controls.
To manage these risks, structured threat modeling frameworks like STRIDE can identify vulnerabilities, while tools like data flow diagrams and attack trees help map and address potential abuse paths. Healthcare organizations must integrate AI-specific security measures into their development processes and reassess risks after updates to maintain safety and compliance.
Threat Modeling with AI: Turning Every Developer into a Threat Modeler
sbb-itb-535baee
Core AI Risks in Clinical Applications
Traditional frameworks often fall short when addressing AI's unpredictability. As Microsoft Security explains, "Traditional threat modeling evolved around deterministic software: known code paths, predictable inputs and outputs, and relatively stable failure modes. AI systems... break many of those assumptions." [1] This disconnect creates significant blind spots in securing clinical applications, exposing vulnerabilities that not only affect individual models but also increase the overall risks within clinical systems.
Adversarial Inputs and Model Manipulation
AI systems can be attacked simply through manipulated inputs. Evasion attacks, for example, involve subtle changes to data - like an X-ray or lab result - that trick an AI model into producing an incorrect diagnosis. A 2019 study published in Science by researchers from Harvard Medical School and MIT demonstrated this risk. They showed that adding minimal noise to a fundoscopy image could change a model's diagnosis from "no diabetic retinopathy" to "proliferative diabetic retinopathy" with 100% confidence. Lead researcher Samuel Finlayson warned:
"The ease with which adversarial examples can be generated in the medical domain suggests that we must be extremely cautious about deploying these models in high-stakes clinical settings without robust defenses." – Samuel Finlayson, Harvard Medical School
Another growing issue is prompt injection, where attackers craft inputs to bypass safety measures in clinical large language models. This can result in the extraction of sensitive patient data or the generation of harmful medical advice. Even more concerning is data poisoning, where attackers introduce a tiny percentage of malicious data - sometimes as little as 0.01% - into the training set. This can create hidden backdoors in the model, directly compromising clinical decision-making.
Data Leakage Risks
AI models trained on patient data can inadvertently expose sensitive information. Through techniques like model inversion, attackers can reconstruct private training data from the model's outputs without needing access to the database itself. This means patient health information could leak directly from the model, bypassing traditional security measures. Inference APIs and logging systems further increase the risk, creating significant compliance challenges for healthcare organizations governed by HIPAA. Unfortunately, conventional data security controls often aren't equipped to handle these unique vulnerabilities.
Model Hallucinations in Clinical Workflows
Large language models can produce outputs that sound confident and precise but are entirely incorrect. In clinical settings, such hallucinations - like a fabricated drug dosage or a non-existent contraindication - can lead to serious patient safety incidents. Worse, these errors might not be flagged in system logs, leaving clinicians to act on incorrect information before the mistake is identified. Traditional threat models often overlook this type of risk, focusing more on unauthorized access than on outputs that, while delivered as intended, can be dangerously misleading.
Model Drift and Performance Degradation
AI models aren't static - they degrade over time, and this can happen quietly in clinical environments. Studies show that clinical models may experience noticeable performance declines in as little as 3 to 6 months due to shifts in patient demographics or hospital protocols. Two key issues arise here: data drift, where new inputs differ from the original training data (e.g., changes in imaging hardware), and concept drift, where updated clinical guidelines render the model's training obsolete. The FDA's Digital Health Center of Excellence has emphasized, "AI models in healthcare are not 'set it and forget it.' They require a lifecycle of continuous monitoring and local validation to ensure they remain safe and effective." Without active monitoring, these silent failures can jeopardize patient safety even when the system appears to function normally.
Third-Party and Supply Chain Vulnerabilities
Many clinical AI tools depend on third-party components, such as foundation models, open-source libraries, pre-trained elements, and external data feeds. Each of these dependencies introduces potential vulnerabilities. For instance, a flaw in an upstream library, a compromised model weight file, or a poorly secured data provider can expose the healthcare system to risks that may not be immediately apparent. Traditional healthcare third-party risk assessment questions often fail to address AI-specific concerns like the integrity of training data, security of model updates, or the safety of inference APIs. Closing these gaps is essential for reducing exposure to supply chain risks.
How AI Expands the Attack Surface in Clinical Systems
AI vs. Traditional Clinical Systems: Expanded Attack Surface & Key Risks
Integrating AI into clinical systems introduces new vulnerabilities that go beyond those seen in traditional, rule-based systems. Unlike deterministic systems - where specific inputs predictably lead to specific outputs - AI systems operate in a more fluid and adaptive manner. As Microsoft Security explains, "AI systems treat conversation and instruction as part of a single input stream, where text - including adversarial text - can be interpreted as executable intent." [1] This shift fundamentally changes the attack surface, as illustrated in the table below.
| Category | Traditional Clinical Systems | AI-Integrated Clinical Systems |
|---|---|---|
| System Behavior | Predictable; follows fixed rules and logic [1] | Dynamic and evolving; less predictable [1] |
| Attack Surfaces | Vulnerabilities in software, networks, and hardware [1] | Adds risks from adversarial data and model manipulation [1] |
| Data Sensitivity | Focused on securing stored and transmitted data [1] | Requires safeguarding data quality, as flawed inputs directly affect AI decisions [1] |
| Mitigation Focus | Aims for complete risk elimination via patching and secure coding [1] | Emphasizes managing residual risks and limiting damage through layered protections [1] |
Vulnerable Input Channels
AI introduces new risks in how data enters the system. Every input channel - whether text prompts, document uploads, EHR integrations, or audio and image data - becomes a potential attack vector. For example, adversarial prompt injections can hide malicious instructions within clinical documents. When an AI processes such a document, it might prioritize the hidden instruction over the clinician's intended command, leading to issues like data breaches or harmful outputs. Traditional web application firewalls are not equipped to detect these sophisticated attacks [1].
Risks in AI Model Outputs
AI-generated outputs can heavily influence clinical decisions, sometimes without adequate scrutiny. This overreliance on AI recommendations can delay critical treatments. A notable example is the Epic Systems Sepsis Model, which was found by University of Michigan researchers to miss 67% of sepsis cases, as reported in JAMA Internal Medicine in June 2021. Such failures often go unnoticed, making it harder to identify and correct errors in time. In clinical environments, these silent errors can have serious consequences.
Data Pipelines and Logging Risks
The data pipelines that feed AI models are another area of concern. These systems often log complete input prompts and responses for debugging purposes. If these logs are not properly secured or anonymized, they can expose sensitive patient health information (PHI). Beyond logging, the training pipeline itself is vulnerable. An attacker could inject compromised data into the fine-tuning process, subtly altering the model's behavior for specific patient scenarios. Microsoft Security highlights that "AI systems often fail in the gaps between components where control is implicit." [1]
Access Control and Privilege Boundaries
AI tools in clinical settings can go beyond answering questions - they can call APIs, update records, and initiate workflows. If access controls are weak, a single compromised feature could lead to cascading effects across interconnected systems. For instance, an AI model with excessive privileges could be manipulated to alter medication orders or access unauthorized patient records. Implementing strict least-privilege principles for every API and tool an AI system interacts with is critical to mitigating these risks [1].
Third-Party Dependencies
Many clinical AI systems depend on external resources like foundation models, cloud-based inference APIs, open-source libraries, and third-party data feeds. Each of these dependencies introduces potential vulnerabilities. For example, if an update server for a third-party model is compromised, attackers could inject malicious code or tampered model weights into the system. The FDA's 2026 Premarket Cybersecurity Guidance now mandates Software Bill of Materials (SBOM) traceability and documented threat models to address these risks throughout a product's lifecycle [2]. Without clearly mapping these trust boundaries, healthcare organizations risk overlooking critical vulnerabilities that standard vendor assessments might miss. Addressing these challenges requires advanced threat modeling techniques, which will be explored further in subsequent sections.
Threat Modeling Techniques for Clinical AI
Addressing AI-related threats in clinical settings demands structured approaches. These methods are essential for connecting identified risks with practical solutions, offering security and clinical teams a repeatable way to pinpoint, classify, and prioritize threats before they escalate into incidents.
Using STRIDE for Threat Categorization
The STRIDE framework can be adapted for clinical AI by focusing on probabilistic failure modes instead of deterministic flaws [4][5]. Each STRIDE category aligns with specific AI-related risks:
| STRIDE Category | AI Adaptation | Clinical Example |
|---|---|---|
| Spoofing | Model Impersonation | A fake tool impersonates a hospital's diagnostic system to capture clinician queries [4] |
| Tampering | Data/Model Poisoning | Backdoor triggers are added to EHR data, causing models to misclassify high-risk patients [4] |
| Repudiation | Provenance Loss | Missing versioning in data pipelines prevents auditing changes to lab unit mappings [3] |
| Information Disclosure | Model Inversion | Repeated oncology model queries reconstruct private genomic data [4] |
| Denial of Service | Resource Exhaustion | "Sponge Examples" overload an inference server during triage [4] |
| Elevation of Privilege | Alignment Bypass | Prompt injection circumvents safety filters, enabling restricted medical advice [4] |
As the STRIDE-AI Framework authors emphasize, "A secure code base does not guarantee a secure model if the training data is poisoned or if the model is susceptible to adversarial perturbations." [4] This is particularly concerning in clinical environments, where tampered models can subtly influence care. Alarmingly, 61% of organizations deploying AI lack a dedicated security strategy [4], making structured frameworks like STRIDE indispensable for risk management.
This categorization forms the basis for mapping data flows and identifying potential abuse paths.
Data Flow Diagrams for Mapping PHI
Data flow diagrams (DFDs) visually represent how patient data moves through an AI system, covering everything from EHR ingestion and processing to feature stores, model training, and inference outputs. Trust boundaries - where data transitions between on-premises EHR systems and cloud analytics or between user sessions and workload identities - should be clearly marked [3][6].
"If you can't clearly draw the boundary, you can't clearly secure it. Trust boundaries are where identity and authorization decisions become concrete." - Code Labs Academy [3]
Once the diagram is complete, STRIDE can be applied to each interaction point. This approach uncovers machine learning-specific risks - like membership inference and training-time poisoning - that traditional security reviews might miss. By combining STRIDE with DFDs, teams can better visualize where AI-related vulnerabilities are likely to emerge.
Attack Trees for Abuse Path Analysis
Building on STRIDE and DFDs, attack trees map out adversarial steps in clinical AI systems. These diagrams detail how an attacker might exploit the system, whether by bypassing safety measures or extracting PHI through repeated queries. Attack trees are particularly useful in identifying threats that traditional security assumptions may overlook [4].
In clinical environments, risk scoring must consider more than just data loss. As Code Labs Academy notes, "In hospitals, impact often includes patient harm, clinician workload spikes, and loss of trust that can stall adoption." [3] Teams can further refine attack tree analysis by integrating the MITRE ATLAS framework to create attacker scenarios that align with clinical workflows [3].
Risk-Based Prioritization
Not all threats are equally severe. Risk can be scored using the formula R = L × I, where likelihood (L) reflects how easily an attack can be executed, and impact (I) accounts for clinical consequences [4]. For example, automated prompt injection might score higher on likelihood than resource-intensive weight poisoning. Impact scores should consider outcomes ranging from minor quality issues to severe PHI breaches or safety failures.
| Risk Score | Severity | Response |
|---|---|---|
| ≥ 20 | Critical | Immediate mitigation required |
| 12–19 | High | Significant risk to patient data or model integrity |
| 6–11 | Medium | Residual risk after initial mitigations |
| ≤ 5 | Low | Minimal impact on clinical workflows |
Source: STRIDE-AI Risk Scoring Matrix [4]
Structured threat modeling delivers tangible benefits. For instance, a black-box evaluation of a retrieval-augmented generation (RAG) chatbot using STRIDE-AI cut adversarial attack success rates from 80% to 15% through targeted mitigations [4]. This highlights the importance of prioritizing risks alongside identifying them.
How to Reduce AI Risks in Clinical Applications
Addressing AI risks in clinical environments requires a combination of technical measures, structured workflows, and active monitoring. No single solution can eliminate these risks, but layering safeguards creates a more resilient system.
Input Validation and Prompt Filtering
AI systems depend heavily on clean and accurate input data. In clinical settings, this means validating and normalizing all incoming data - whether it comes from electronic health record (EHR) systems, vital sign monitors, or manual entries by clinicians. These steps ensure that data aligns with expected formats before reaching the AI model.
To prevent misuse, rate limiting should be applied to input channels. This stops attackers from flooding the system with crafted inputs aimed at reverse-engineering the AI's decision-making process or overwhelming its resources. Tailored input filters for each system component add another layer of protection. For anomalies that are flagged, fallback manual reviews must be in place to maintain system integrity. These foundational controls are essential for building trust in AI systems.
Human Oversight of AI Outputs
Even with robust input controls, human oversight is critical for maintaining safety in clinical decision-making. AI-generated recommendations, especially those involving high-stakes scenarios like sepsis alerts, medication dosing, or diagnostics, should always go through a human checkpoint. This ensures clinicians can catch potential errors, such as hallucinated or biased outputs, that technical measures might miss.
However, this doesn’t mean every AI output requires review. Instead, workflows should be designed so that only critical recommendations need explicit clinician approval before any action is taken. This balances efficiency with safety.
Data Minimization and PHI Controls
Reducing the amount of data processed by AI systems minimizes risks. By limiting inputs to only the essential features and applying strict field-level controls for protected health information (PHI), the exposure to potential attacks is significantly lowered. This approach also reduces the risk of advanced threats like model inversion attacks, which attempt to reconstruct sensitive patient data.
Authentication and Role-Based Access
Strong identity and access controls are a must. Role-based access ensures that only authorized users and systems can interact with AI models, access training data, or make configuration changes. For added security, service-to-service communication should use mutual TLS (mTLS), which helps prevent interception or downgrade attacks on data being transmitted.
Monitoring for Model Drift and Bias
AI models in healthcare aren’t static - they can degrade over time as patient demographics shift, medical protocols evolve, or the data they rely on changes. Continuous monitoring of performance metrics, output trends, and subgroup performance is essential to detect these changes early. Setting clear thresholds for re-evaluation or retraining ensures that any issues are addressed before they can impact patient care. Significant updates to models should also undergo fresh threat assessments and validation.
Reviewing Vendors and Dependencies
Managing third-party AI risk through vendor assessments is another critical step in reducing risks. Beyond standard evaluations, it’s important to verify how third-party models were trained, what data was used, and how updates are managed. Vendors should maintain documented security practices, and all model files, rule sets, and configurations must be signed and verified before deployment. This prevents unauthorized changes, such as replacement or rollback attacks.
| Risk Area | Primary Threat | Recommended Control |
|---|---|---|
| Model Inputs | Manipulation or crafted payloads | Schema validation, rate limiting, input normalization, integrity checks |
| Clinical Outputs | Manipulation or leakage | Guardrail checks, safe explainability, human review workflows |
| Model Artifacts | Unauthorized replacement | Artifact signing, immutable registries, promotion gates |
| PHI in Transit | Interception or downgrade | TLS 1.2+/1.3 and mTLS for service-to-service calls |
Putting AI Threat Modeling into Practice in Healthcare
Embedding AI Threats into Secure Development Processes
Managing AI risks effectively starts with addressing them early - ideally during the design phase of the Software Development Life Cycle (SDLC). This "shift-left" strategy ensures that AI-specific threat modeling is integrated well before any code reaches a clinical setting.
To do this, traditional STRIDE analysis must be expanded to include AI-specific threats. For example, Inference Attacks align with Information Disclosure, while Adversarial Evasion, where crafted inputs distort clinical outputs, falls under Tampering. The real value of this early integration is that it embeds security considerations into the design, making it easier to address vulnerabilities before they become deeply rooted in the system's architecture.
Collaboration across disciplines is crucial for success. Clinical informaticists, security engineers, and compliance officers all need to contribute during design reviews. Without clinical insights, security teams might overlook how a flawed AI recommendation could disrupt care workflows. Similarly, clinical teams may not spot instances where a model's behavior has been maliciously altered without input from security experts.
Building on this early-stage collaboration, ongoing risk reassessment ensures AI systems remain secure as they evolve.
Reassessing Risks After Updates
AI systems are constantly changing, and each update - whether to the model, dataset, or broader system - can introduce new vulnerabilities. Structured reassessment after updates is essential to identify and address these risks.
This process must examine the entire system loop, not just the model itself. Updates to components like API queries, tool integrations, or credential management can create security gaps across interconnected layers. As the Microsoft AI Red Team has pointed out:
"The non-deterministic nature of AI models means manual testing cannot scale. You need automation and statistical evaluation to cover the variation space." [5]
Additionally, changes in the supply chain, such as the introduction of a compromised dataset, can jeopardize the integrity of downstream models. Regular audits of these supply chain elements are critical to maintaining security. This ongoing vigilance ensures threat modeling remains a core part of managing clinical risks.
Connecting AI Threat Modeling to GRC Workflows
After identifying and reassessing risks, integrating AI threat modeling into governance, risk, and compliance (GRC) workflows strengthens overall security. AI-specific threats shouldn't exist in isolation - they must align with broader GRC processes to provide a unified view of risks.
This integration allows security teams to map AI threats directly to regulatory requirements. For example, risks like model inversion or adversarial manipulation can be tied to HIPAA, the HITECH Act, and the NIST AI Risk Management Framework (AI RMF). Connecting these dots makes it easier to demonstrate compliance and prioritize mitigation efforts based on both regulatory exposure and technical impact.
Using Censinet RiskOps™ for AI Risk Management
Platforms like Censinet RiskOps™ help healthcare organizations centralize and streamline AI risk management across the entire lifecycle. Acting as "air traffic control" for AI governance, the platform routes key findings and tasks to the appropriate stakeholders, such as members of an AI governance committee, for review and action.
A centralized AI risk dashboard allows organizations to track and manage critical risks identified during threat modeling. Tools like Censinet AI™ also incorporate human-in-the-loop automation, streamlining processes like evidence validation and risk mitigation. This approach speeds up workflows while ensuring that human judgment remains central to critical, high-stakes decisions.
Conclusion: Addressing AI Risks Through Clinical Application Threat Modeling
AI has reshaped how we think about risks in clinical applications. Unlike traditional software, which operates with fixed, predictable logic, AI systems rely on probabilities and the data they’re trained on. This means their behavior can change over time - whether due to shifts in patient demographics, retraining of models, or new system integrations. Because of this, security reviews must be an ongoing process.
As highlighted earlier, risks like adversarial inputs, data leakage, hallucinations, model drift, and supply chain vulnerabilities don’t exist in isolation. They often overlap, leading to serious clinical consequences such as misdiagnoses, inappropriate treatments, breaches of patient health information (PHI), and disruptions in care. Threat modeling provides a structured approach to tackling these interconnected risks, focusing on their real impact on patient safety. The stakes are high: in 2024, the average cost of a healthcare data breach in the U.S. hit $10.93 million, and between 2018 and 2023, breaches affecting 500 or more individuals rose by more than 93%, according to HHS OCR data.
The link between patient safety and cybersecurity is undeniable. A compromised AI model in radiology or an inaccurate discharge summary generated by the system doesn’t just pose a technical problem - it’s a clinical failure. That’s why AI threat modeling needs the same level of attention as other critical healthcare safety practices, like monitoring medication safety or assessing risks in medical devices.
To move forward, AI threat modeling should be integrated into the software development lifecycle (SDLC) from the very beginning. Reassessments should be triggered whenever there are changes to models or integrations, and human oversight must remain in place for decisions with high clinical stakes. Additionally, outputs from threat modeling should feed into governance, risk, and compliance (GRC) systems, as well as vendor risk management workflows. Tools like Censinet RiskOps™ can support healthcare organizations by centralizing AI risk findings, simplifying third-party assessments for AI-enabled vendors, and ensuring that the right actions are taken by the right stakeholders at the right time. These measures align with the secure development practices discussed earlier.
FAQs
What are the biggest AI-specific threats to patient safety in clinical apps?
AI systems in healthcare face unique risks that could directly impact patient safety. One such risk is data poisoning - when malicious actors introduce harmful data into training sets. This can lead to AI models making incorrect diagnoses or recommending improper treatments.
Another significant issue is adversarial inputs, where small, deliberate alterations to medical images or data can confuse AI systems, reducing their accuracy and potentially causing harm. Similarly, prompt injection attacks manipulate the outputs of AI systems, steering them toward unsafe or incorrect conclusions.
Lastly, model inversion attacks pose a major privacy concern. These attacks can extract sensitive patient information from AI models, jeopardizing both confidentiality and safety. Addressing these challenges requires implementing strong, targeted security measures designed specifically for AI in clinical settings.
How can we detect and respond to model drift before outcomes are affected?
Detecting and addressing model drift demands constant vigilance and periodic evaluations. Set up automated alerts to monitor critical metrics such as accuracy, bias, and data drift - these can help spot issues early. It's also essential to keep a clean validation set handy for comparing the model's performance against baseline expectations. To stay ahead, regularly retrain models, refresh threat models, and incorporate methods like anomaly detection and input validation. These steps can help mitigate risks to patient safety and maintain the system's reliability.
What should we require from AI vendors to reduce supply chain risk and meet FDA expectations?
To minimize supply chain risks and align with FDA requirements, it's crucial to set clear expectations for AI vendors. Here’s what to require:
- Detailed Software Bill of Materials (SBOM): Ensure vendors provide a comprehensive list of all software components, including third-party and open-source elements.
- Adherence to Cybersecurity Standards: Vendors should comply with FDA cybersecurity guidelines and standards like ISO 13485.
- Continuous Post-Market Monitoring: Ongoing surveillance of AI products is essential to identify and address potential issues after deployment.
- Threat Modeling Across the Product Lifecycle: Require vendors to assess and mitigate risks at every stage of development and usage.
- Vulnerability Management and Updates: Insist on clear, proactive processes for identifying, managing, and patching vulnerabilities.
These measures not only reduce risks but also help maintain compliance and build trust in AI-driven solutions.
