Healthcare Organizations Adopting Artificial Intelligence

Regulatory Compliance, Patient Privacy (PHI), and Workflows — a white paper framework for evaluating which AI tools a HIPAA covered entity or business associate may deploy lawfully, under what conditions, and with what contractual and clinical safeguards.

This white paper is provided for informational and educational purposes only and is not intended to constitute legal, tax, accounting, financial, or other professional advice. Readers should not act or refrain from acting based on any information here without first seeking advice from qualified legal counsel. Receipt of this document does not create an attorney-client relationship, consultant-client relationship, or any other professional relationship.

Opening Statement

This analysis evaluates the landscape of AI tools available to healthcare organizations — providers, health systems, digital-health companies, payers, and their vendors — to determine which tools can be deployed lawfully, under what conditions, and with what safeguards. Its objective is to define the functional and contractual requirements an AI software vendor must satisfy before it touches Protected Health Information (PHI) or influences a clinical or coverage decision.

The analysis proceeds from a premise the research confirms: no AI tool, regardless of its technical architecture, eliminates a healthcare organization's compliance obligations. Vendor technology can reduce risk, simplify compliance, and strengthen an organization's defensibility — but the obligations to apply the minimum-necessary standard, to obtain any patient authorization required by law, to keep a licensed professional accountable for AI output that affects care, and to verify AI-generated content before it reaches a patient or a claim decision are obligations of the covered entity, not of vendor selection.

At the same time, vendor selection is far from neutral. Under HIPAA, disclosing PHI to a vendor that has not signed a Business Associate Agreement (BAA) is itself a violation — regardless of intent and regardless of whether a breach ever occurs. The Office for Civil Rights (OCR) has repeatedly imposed six- and seven-figure penalties on organizations for exactly this gap. Separately, the 2023–2026 wave of class actions over AI-driven claim denials — Estate of Lokken v. UnitedHealth Group and Kisting-Leung v. Cigna — has established that deploying AI to supplant clinical judgment carries direct liability. Choosing the wrong tool, or deploying the right tool without a proper BAA, matters.

This document serves as a practical resource for defining vendor decision requirements within the specific regulatory framework that governs U.S. healthcare organizations (with California noted where state law adds obligations), identifying the questions that must be answered before any tool is deployed with patient data, and providing the contractual and procedural templates needed to operationalize compliance once a vendor is selected.

The healthcare counterpart to the legal DPA

Where a law firm relies on a Data Processing Addendum (DPA) to bind an AI vendor, a healthcare organization relies on the Business Associate Agreement (BAA). The BAA is the direct analog — but it is stronger: it is a statutory requirement under 45 CFR §§ 164.502(e) and 164.504(e), not merely a best practice. Any AI vendor that creates, receives, maintains, or transmits PHI on the organization's behalf is a business associate by law, and PHI may not be disclosed to it until a compliant BAA is signed.

Governing Regulatory Framework

The analysis is conducted against the following federal and (where noted) California standards, each of which imposes specific obligations on the use of AI tools in healthcare:

Authority	Relevance to AI Vendor Selection
HIPAA Privacy Rule (45 CFR Part 164, Subpart E)	Governs use and disclosure of PHI. A covered entity may not disclose PHI to an AI vendor without a signed BAA. The minimum-necessary standard limits what PHI the tool may access.
HIPAA Security Rule (45 CFR §§ 164.308–316)	Requires administrative, physical, and technical safeguards for electronic PHI — risk analysis, encryption, access controls, audit logging. OCR's 2025 proposed update expressly folds AI systems into the required risk analysis.
HIPAA Breach Notification Rule (45 CFR §§ 164.400–414)	A business associate must report security incidents and breaches to the covered entity; the covered entity must notify individuals (generally within 60 days). BAA must fix the notification window.
HITECH Act / Omnibus Rule (2013)	Makes business associates directly liable under HIPAA and requires the same obligations to flow down to subcontractors (sub-processors, including any third-party LLM provider).
Business Associate Agreement (45 CFR §§ 164.502(e), 164.504(e))	The mandatory written contract required before any PHI is disclosed to a vendor. The healthcare equivalent of a DPA — and a legal precondition to deployment, not an option.
FDA — AI-Enabled Device / SaMD (GMLP, PCCP, TPLC guidance)	If the AI tool performs a device function (informing diagnosis or treatment), FDA clearance/authorization and total-product-lifecycle controls may apply. A Predetermined Change Control Plan governs adaptive models.
21st Century Cures Act — CDS criteria	Determines whether Clinical Decision Support software is FDA-regulated. Tools that let a provider independently review the basis of a recommendation may fall outside device regulation.
CMS — Medicare Advantage rule (2024)	Coverage and medical-necessity determinations may not be based solely on an algorithm; an individualized determination grounded in the patient's circumstances is required.
CA AB 3030 — GenAI in Health Care (2025)	Generative-AI patient communications about clinical information must carry an AI disclaimer and human-contact instructions — unless a licensed provider reviews the message first.
CA SB 1120 — Physicians Make Decisions Act (2025)	In utilization review, AI may not make the final medical-necessity determination; a licensed physician or competent professional must. Oversight by DMHC/CDI.
CA CMIA — Confidentiality of Medical Information Act	State medical-confidentiality law requiring patient consent before disclosure, with statutory damages. Reaches some digital-health apps not covered by HIPAA; the AG has warned against AI "dark patterns" to obtain consent.
Section 1557 / civil-rights standards	AI outputs must not produce discriminatory results across protected classes; bias testing and representative data are expected.
NIST AI Risk Management Framework	A voluntary but increasingly regulator-referenced framework for AI validity, reliability, security, explainability, and fairness — used alongside HIPAA.

Analysis Objectives

This analysis is structured around six objectives, each addressing a distinct compliance or operational dimension of AI tool adoption at a healthcare organization.

Establish the Regulatory Baseline. Identify the specific federal and state rules that govern the use of AI tools with PHI and in clinical decision-making, and map each rule to the concrete vendor-selection and workflow decisions it requires.
Assess Vendor Compliance Posture Against HIPAA's Safeguard Standard. Evaluate each vendor against HIPAA Security Rule safeguards and the minimum-necessary standard: encryption and access controls, no use of PHI for model training, sub-processor controls, audit logging, and independent certifications (SOC 2 Type II, HITRUST CSF).
Evaluate PHI-Disclosure and BAA Risk for Each Vendor. Determine whether the vendor is a business associate, whether a HIPAA-compliant BAA is in place before any PHI is disclosed, and whether the deployment would survive an OCR audit. Sharing PHI without a BAA is a per-se violation regardless of the vendor's technical security.
Assess Clinical Workflow and Human-Oversight Suitability. For tools that influence diagnosis, treatment, clinical documentation, or coverage, confirm that a licensed professional remains the accountable decision-maker (SB 1120, CMS MA rule) and that patients are informed of AI use where required (AB 3030).
Identify the Minimum Contractual Requirements for Compliant Deployment. Define the BAA provisions — including AI-specific enhancements such as the training prohibition and sub-processor flow-down — that a healthcare organization must obtain from any selected vendor before deploying the tool with PHI.
Recommend Practical Workflow Protocols for Clinical Adoption. Translate compliance requirements into actionable protocols: minimum-necessary data scoping, verification of AI output, documentation of the vendor assessment, and clinician sign-off — creating the easiest defensible path to compliant AI adoption.

Scope and Assumptions

This analysis assumes that the organization is a HIPAA covered entity (or a business associate acting on one's behalf) and that it has obtained, or will obtain prior to deployment, any patient authorization required under applicable state law (for example, under California's CMIA), and that it will provide any patient-facing AI disclosures required by law (for example, AB 3030 disclaimers on generative-AI clinical communications). These baseline protections are treated as satisfied throughout the vendor analysis. The remaining analysis focuses on vendor architecture, contractual protections (the BAA), and clinical workflow as the variables that determine compliance.

The Litigation Landscape Defining Healthcare AI

Two lines of legal exposure now define AI adoption in healthcare. Both bear directly on vendor selection.

1. Improper disclosure of PHI to a vendor. Under HIPAA, disclosing PHI to a vendor that has not signed a BAA is itself a violation — the enforcement risk does not depend on a breach occurring. OCR's enforcement history makes the cost concrete: North Memorial Health Care paid $1.55 million and Raleigh Orthopaedic Clinic paid $750,000, in each case for giving a vendor access to patient data with no BAA in place. As recently as March 2026, OCR settled with a software vendor following a breach affecting roughly 15 million individuals, and the 2026 NYC Health + Hospitals incident — traced to a third-party vendor's weaker controls — illustrates that a covered entity remains liable for a supplier's security posture. The BAA, and the vendor risk management around it, is precisely what is supposed to govern that relationship.

2. AI supplanting clinical or coverage judgment. In Estate of Lokken v. UnitedHealth Group (D. Minn.) and Kisting-Leung v. Cigna (E.D. Cal.), plaintiffs allege that AI models — nH Predict and PxDx — were used to deny medically necessary care without individualized physician review, overriding treating clinicians. The Cigna complaint alleges the tool processed claims at roughly 1.2 seconds each; the UnitedHealth complaint alleges the model's determinations were reversed on roughly 90% of appeals. Both cases survived early motions, and in March 2026 a federal court ordered broad discovery into UnitedHealth's AI-driven claims processes — a signal that the inner workings of these tools are themselves discoverable. Separately, a state attorney general has pursued a healthcare-AI vendor over allegedly false and misleading claims about the accuracy and safety of its products deployed in hospitals.

The takeaway: the vendor relationship (the BAA) and the human-oversight architecture are the two variables that determine compliance. A tool that is technically secure but deployed to replace a clinician's judgment is a liability; a clinically sound tool deployed without a BAA is a regulatory violation.

The Business Associate Framework — The Key Legal Mechanism

Where attorney-client confidentiality turns on evidentiary privilege, healthcare confidentiality turns on the Business Associate framework. Any vendor that creates, receives, maintains, or transmits PHI on the covered entity's behalf is, by definition, a business associate under 45 CFR 160.103 — even a cloud vendor that cannot read the PHI because it is encrypted. Three elements must hold for a compliant deployment:

A signed BAA is in place before any PHI is disclosed to the vendor.
The vendor implements the HIPAA Security Rule safeguards and honors the minimum-necessary standard.
The same obligations flow down to every sub-processor the vendor uses, including any third-party LLM provider.

Without the BAA, the disclosure is unlawful regardless of the vendor's technical security — the structural parallel to sharing confidential information with an AI platform that lacks contractual confidentiality protections. A vendor that declines to sign a BAA when PHI is involved is not a viable option.

Executing the Framework — What to Verify

Confidentiality & safeguards. The vendor should preserve PHI in an isolated, encrypted environment, restrict access on a minimum-necessary basis, and neither it nor its sub-processors should use PHI to train, fine-tune, or improve any model. These commitments must appear in the signed BAA, not only in marketing materials.

Security certifications. SOC 2 Type II confirms the vendor's security controls operated effectively over an independent audit period; HITRUST CSF certification maps controls specifically to HIPAA. SOC 3 is the public-facing version available without an NDA. Request current reports, the sub-processor list, and any penetration-test attestations.

Human oversight preserved. For any tool that influences diagnosis, treatment, documentation, or coverage, confirm the workflow keeps a licensed professional as the accountable decision-maker — and that vendor accuracy and safety claims are independently validated rather than accepted at face value.

Mapping Vendor Claims to a HIPAA Safeguard Test

HIPAA / Safeguard Criterion	Expected Vendor Claim	Your Verification Step
PHI used for model training?	No — contractually prohibited, including for sub-processors and any LLM provider	Confirm the prohibition in the signed BAA and request sub-processors' zero-retention / no-training commitments
Data retention period	Defined and minimized (ideally session-only)	Confirm the maximum retention in the BAA; require deletion-on-demand with written certification
Sub-processor / LLM sharing	Complete list published; HIPAA obligations flow down	Review the sub-processor list for gaps; verify BAAs exist down the chain
Security certifications	SOC 2 Type II + HITRUST CSF	Strong — request current certificates and the SOC 2 report
Human oversight of clinical / coverage output	Licensed professional remains the decision-maker	Confirm the workflow and, where required, AB 3030 patient disclosures and SB 1120 physician review

The Provider's Non-Delegable Obligations

The obligations under HIPAA and state clinical-oversight law are not vendor-specific — they are professional and organizational obligations that travel with the covered entity and the licensed professional. Even if a vendor has perfect technical protections, the organization still must:

Apply the minimum-necessary standard — the AI tool accesses and uses only the PHI strictly required for its purpose.
Obtain any patient authorization required by state law before disclosing medical information (for example, under the CMIA), without relying on manipulative interfaces to secure consent.
Keep a licensed professional accountable for AI output that affects care or coverage (SB 1120, the CMS Medicare Advantage rule), and disclose AI use to patients where required (AB 3030).
Document the vendor assessment and the basis for reliance on the tool.

No tool — no matter how secure — substitutes for these steps. The tool can make compliance easier and more defensible, but it cannot eliminate the requirement. With minimum-necessary scoping, a signed BAA, and preserved human oversight in place, a healthcare organization can route actual patient data through a compliant AI tool.

Why a BAA?

The Business Associate Agreement obligation is not about what technology processes the data — it is about the legal relationship between the covered entity and the vendor. Three frameworks independently require it, regardless of the model's architecture:

HIPAA Privacy Rule (45 CFR 164.502(e)). A covered entity may not disclose PHI to a business associate without first obtaining satisfactory assurances, in a written contract, that the vendor will safeguard the PHI. The BAA is that contract.

HIPAA Security Rule (45 CFR 164.308(b), 164.314(a)). Those assurances must be documented in a written agreement addressing the safeguarding of electronic PHI — encryption, access controls, incident reporting, and sub-processor obligations.

State medical-confidentiality law (e.g., CA CMIA). Independent consent and confidentiality obligations attach to medical information; a written vendor agreement operationalizes them and can reach vendors outside HIPAA's scope.

And under the HITECH Act and 2013 Omnibus Rule, the BAA is also what makes the vendor directly liable for HIPAA compliance. Verbal or implied assurances do not satisfy HIPAA. In short: the BAA is the healthcare counterpart to the legal-sector DPA — but it is a statutory mandate, and operating without a required BAA is a violation whether or not a breach ever occurs.

BAA Elements

Core elements of a HIPAA-compliant Business Associate Agreement for an AI vendor. Items 1–13 track the HIPAA Rules; the AI-specific language is called out where it belongs. The HHS sample BAA is a starting point, not a ready-to-sign agreement — it must be tailored to the actual service, the PHI involved, and AI-specific data handling.

Parties and Relationship Definition. Identifies the covered entity and the business associate, confirms the BAA is incorporated into the underlying services agreement, and states which activities the vendor is engaged to perform with PHI.
Definitions. Adopts the HIPAA Rules' defined terms (PHI, ePHI, Breach, Security Incident, Subcontractor, Minimum Necessary, Designated Record Set) and adds AI-specific terms: Model Training, Sub-Processor / Sub-LLM, Retention Period. Vague definitions are where compliance gaps hide.
Permitted Uses and Prohibited Processing. Specifies the exact purposes for which the vendor may process PHI and explicitly prohibits everything else — above all, using PHI to train, fine-tune, evaluate, or improve any AI model without the covered entity's written authorization. This is the single most critical clause for a healthcare AI vendor.
Minimum Necessary. Commits the vendor to accessing and using only the minimum PHI necessary to perform the service, consistent with 45 CFR 164.502(b).
Safeguards (Security Rule). Requires implementation of the HIPAA Security Rule: encryption (AES-256 at rest, TLS in transit), access controls, audit logging, and tenant/database-level isolation of each customer's PHI.
Sub-Processor / Sub-LLM Controls. Requires a complete sub-processor list, advance notice before adding sub-processors, the covered entity's right to object, and flow-down of the same HIPAA obligations to every sub-processor — expressly including any third-party LLM provider.
Breach and Security Incident Notification. Requires the vendor to report security incidents and breaches of unsecured PHI within a defined window (many organizations require 24–72 hours; OCR's 2025 proposal contemplates 24-hour notice of contingency-plan activation), with content requirements: nature of the incident, PHI categories affected, and remediation.
Data Retention, Return, and Destruction. States the maximum retention period, the covered entity's right to demand deletion of a specific patient's or matter's data, the deletion timeframe with written certification, and return or destruction of PHI on termination.
Individual Rights Support. Obligates the vendor to support the covered entity's duties to provide individuals access to and amendment of PHI, and an accounting of disclosures, for PHI in a designated record set.
Audit Rights and Logs. Grants the covered entity the right to audit — or commission an audit of — vendor compliance with reasonable notice, and requires the vendor to maintain and produce audit logs of all access to PHI.
Representations and Warranties. Vendor warrants authority to enter the BAA, current SOC 2 Type II / HITRUST certification, accuracy of the sub-processor list, that no PHI has been or will be used as training data without written authorization, and — if the tool performs a device function — its FDA clearance/authorization or documented CDS-exclusion status.
Term, Termination, and Survival. Ties the BAA to the services-agreement term and specifies which obligations survive termination (prohibited processing, security, audit, and the deletion/return duties).
Governing Law. Specifies the governing state and jurisdiction; for California organizations this should be explicit given CMIA and California's AI-in-healthcare statutes.
Exhibit A — Sub-Processor List. A living exhibit naming each authorized sub-processor (including any LLM provider), its role, data location, and confirmation that training on PHI is contractually prohibited. Updated with notice whenever the list changes.

Notice of Privacy Practices & Patient-Facing Disclosures

Parallel to a law firm's engagement-letter update, a healthcare organization should align its patient-facing documents with its AI use:

Notice of Privacy Practices. Review the NPP to ensure the described uses and disclosures of PHI for treatment, payment, and operations accurately reflect AI-assisted workflows and vendor relationships.

Generative-AI communication disclaimers (AB 3030). Where generative AI produces a patient communication about clinical information, include a prominent AI disclaimer and clear instructions for reaching a human provider — unless a licensed provider reviews the message before it is sent.

Patient authorization where required. Where state law (e.g., the CMIA) requires patient consent to disclose medical information, obtain it through clear, non-manipulative means before routing data to an AI vendor.

Software AI Vendor Requirements

The following are the minimum functional and contractual requirements an AI software vendor must satisfy before deployment with PHI or in a clinical/coverage workflow.

Executed HIPAA-Compliant BAA (before any PHI). A signed, tailored BAA must be in place before the vendor receives any PHI. It must include the AI-specific training prohibition and sub-processor flow-down. A vendor that declines to sign a BAA when PHI is involved is disqualified — this is a legal precondition, not a negotiating point.
Absolute Training Prohibition. PHI — including prompts, uploaded records, and AI-generated output derived from PHI — is contractually prohibited from use in training, fine-tuning, evaluating, or improving any AI model. The prohibition must extend to all sub-processors, including any third-party LLM provider, and must appear in the signed BAA, not just marketing materials.
Minimum-Necessary Access, Retention & Deletion. The vendor accesses only the minimum PHI necessary for the service. Retention is defined and minimized — session-only or zero-retention where technically feasible; otherwise the maximum period is stated in the BAA, with the organization's right to demand deletion of a specific patient's data within a defined window (e.g., 24–72 hours) and written certification of deletion.
Security Safeguards & Database-Level Tenant Isolation. Encryption at rest (AES-256) and in transit (TLS), role-based access controls, and audit logging, with each customer's — and each patient's — PHI isolated at the database level, not merely separated logically. Cross-customer contamination is a material privacy and security risk. SOC 2 Type II and HITRUST CSF certification expected.
Preserved Human Oversight & Validated Performance. For any tool that influences diagnosis, treatment, documentation, or coverage, the workflow must keep a licensed professional as the accountable decision-maker (SB 1120, CMS Medicare Advantage rule), disclose AI use to patients where required (AB 3030), and rest on vendor accuracy, bias, and safety claims that the organization has independently validated rather than accepted at face value.
Regulatory (FDA) Status Clarity. If the tool performs a device function — informing diagnosis or treatment beyond transparent clinical decision support — confirm FDA clearance/authorization or a documented Cures Act CDS exclusion, and, for adaptive models, a Predetermined Change Control Plan governing how the algorithm may change after deployment.

Appendix

Vendor Compliance Scorecard

A reusable template for evaluating a specific AI vendor. Score each requirement as Met, Partial, or Not Met, and record the evidence obtained. A single "Not Met" on items 1–4 should generally halt deployment until resolved.

Requirement	Evidence to Request
Executed HIPAA-compliant BAA in place before PHI disclosure	Signed BAA; AI training-prohibition clause; sub-processor flow-down
No PHI used to train / fine-tune / improve any model (incl. LLM)	BAA clause; sub-processor no-training / zero-retention commitments
Minimum-necessary access; defined retention; deletion-on-demand	BAA retention & deletion clause; certification-of-deletion process
Encryption (AES-256 / TLS), access controls, audit logging	SOC 2 Type II report; architecture description
Database-level tenant / patient isolation	Architecture attestation (not just access-control policy)
Independent certifications	Current SOC 2 Type II + HITRUST CSF certificates; SOC 3
Complete, published sub-processor / LLM list with flow-down	Sub-processor list; downstream BAAs
Breach / incident notification within a defined window	BAA notification clause with timeline and content requirements
Human oversight preserved for clinical / coverage decisions	Workflow documentation; SB 1120 physician-review; AB 3030 disclosures
Validated accuracy / bias / safety claims	Validation studies; bias testing; performance-monitoring plan
FDA status clear (clearance, CDS exclusion, or PCCP)	FDA clearance letter, CDS-exclusion analysis, or PCCP

Red Flags

Vendor will not sign a BAA, or offers only a generic click-through with no AI-specific terms.
Training prohibition appears only in marketing or a privacy policy — not in the signed BAA.
No sub-processor list, or an LLM provider in the chain without a flow-down BAA.
Retention and deletion are undefined, or deletion cannot be certified in writing.
Marketing claims of accuracy or safety with no validation data, bias testing, or performance-monitoring plan.
A tool positioned to make — rather than support — clinical or coverage decisions with no licensed-professional review in the workflow.

Assuring Compliance Without SOC 2

What HIPAA actually mandates is not SOC 2 — it's implementing the Security Rule safeguards and documenting a formal risk analysis. There's no such thing as "HIPAA certified"; you're compliant or not, and you prove it with policies, risk assessments, BAAs, and evidence. Do and document that work, and a small company is compliant. Cheaper ways to demonstrate the same posture to a buyer:

HIPAA Security Risk Assessment — required by law anyway (45 CFR 164.308(a)(1)). HHS/ONC publish a free SRA Tool. This is the non-negotiable foundation, not an option.
Written policies and procedures mapped to the Security Rule: access control, encryption, audit logging, incident response, contingency/backup, workforce training.
A third-party HIPAA gap assessment or attestation — a consultant reviews your controls and issues a HIPAA compliance report. A fraction of a SOC 2 audit's cost and healthcare-specific.
Independent penetration test + vulnerability scan — concrete outside evidence, usually a few thousand dollars.
Build on HIPAA-eligible cloud infrastructure (AWS/Azure/GCP) under a cloud BAA, so you inherit the physical/infrastructure controls and can point to the provider's certifications instead of proving them yourself.
Complete a standardized security questionnaire (SIG Lite, CAIQ, or the client's own) with evidence attached — often exactly what the buyer's security team actually needs.
Lighter attestations as stepping stones: HITRUST's tiered e1/i1 assessments are far cheaper than a full r2 or SOC 2 and are healthcare-specific; SOC 2 Type I is faster and cheaper than Type II, and many companies start there before pursuing Type II. Compliance-automation platforms (Vanta, Drata, Secureframe, Thoropass) have HIPAA modules that produce a monitored report at lower cost.

Underneath all of it, the technical baseline both HIPAA and any attestation will look for is the same: encryption at rest (AES-256) and in transit (TLS), MFA, least-privilege access, audit logging, and tested backups.

Bottom line: the mandatory items for the agency are the BAA (up and down the chain, including the LLM provider) plus the Security Rule controls and a documented risk analysis. To win the deal without SOC 2, package a HIPAA SRA + written policies + a third-party HIPAA attestation or pen test + a completed security questionnaire, running on HIPAA-eligible cloud infra — and offer a roadmap toward SOC 2 or HITRUST as you scale. Many healthcare buyers accept exactly that from smaller vendors, especially with a clear remediation timeline.

One thing worth flagging: the proposed 2025 HIPAA Security Rule update would convert several currently "addressable" safeguards into mandatory ones and add annual verification, so the security floor for even small vendors is rising — worth building toward now rather than retrofitting later.

This document is a generic compliance-planning reference, not legal advice. Regulatory requirements — particularly state AI-in-healthcare laws and the pending HIPAA Security Rule update — are evolving rapidly; confirm current requirements and consult qualified healthcare counsel before deploying any AI tool with PHI.

Download the full white paper