Building HIPAA-Compliant AI: Architecture, Data Handling and Vendor Choices

Healthcare organisations want AI for documentation, patient communication, coding, triage support and operations. Many of these uses involve protected health information (PHI). HIPAA does not prohibit AI; it requires that PHI be used and disclosed appropriately and protected with administrative, physical and technical safeguards — wherever it flows.

This guide is for technical teams designing AI features in the US healthcare context. It is not legal advice; involve your privacy officer and counsel. For grounding assistants in approved clinical content, see our guide to RAG in healthcare.

HIPAA in one page for AI teams

• Privacy Rule: governs uses and disclosures of PHI, including the minimum necessary standard and patients’ rights.
• Security Rule: requires administrative, physical and technical safeguards for electronic PHI, based on a documented risk analysis.
• Breach Notification Rule: requires notification of affected individuals without unreasonable delay and within 60 days of discovery, notification to HHS, and media notice for breaches affecting more than 500 residents of a state or jurisdiction.
• Business associates: vendors that handle PHI on your behalf must sign a business associate agreement (BAA) and comply with the Security Rule.

Where PHI flows in an AI system

Reference architecture

1. Identity and access: users authenticate with your identity provider; every request carries the user’s identity and role.
2. AI gateway: a single service holds model credentials, enforces policies, redacts where appropriate and logs metadata.
3. Retrieval with permissions: searches return only records the user is allowed to see, filtered before content reaches the model.
4. BAA-covered model endpoint: a model service covered by a BAA, configured for no training on your data and minimal retention.
5. Output checks: validate structure, detect PHI where it should not appear, and route sensitive outputs to human review.
6. Storage: encrypted databases and indexes in HIPAA-eligible services, with defined retention.
7. Monitoring: security logging, access reviews and alerting on unusual access patterns.

Keep PHI processing inside environments you control or that are covered by BAAs. Never send PHI to consumer AI tools or to APIs not covered by an agreement.

De-identification options

HHS guidance describes two methods for de-identifying PHI under the Privacy Rule:

• Safe Harbor: remove 18 specified identifiers — including names, most geographic subdivisions smaller than a state, dates other than year, contact details, record numbers and full-face photographs — and have no actual knowledge that the remaining information could identify an individual. Free-text notes need careful redaction.
• Expert Determination: a qualified expert applies statistical or scientific methods and documents that the risk of re-identification is very small in the intended context.

De-identified data is no longer PHI, which makes it suitable for many evaluation, analytics and development tasks. Be cautious: free text, rare conditions and linkable data can re-identify people, and language models can reproduce identifiers from training data.

Choosing vendors

Security safeguards that matter most

• Risk analysis covering the AI components, updated when architecture or vendors change.
• Encryption of PHI in transit and at rest, including indexes, caches and backups.
• Least-privilege access for users, services and AI tools, with regular reviews.
• Audit logs of who accessed what PHI through the AI system.
• Prompt-injection defences: treat retrieved documents and user input as untrusted, and limit what model outputs can trigger.
• Testing: security testing and red teaming of AI features before release.
• Incident response plans that include AI-specific scenarios, such as an assistant revealing another patient’s data.

The proposed Security Rule update

In January 2025 HHS published a proposed rule to strengthen the HIPAA Security Rule, including more specific requirements for risk analysis, asset inventories, encryption and multi-factor authentication. Final action has since been delayed — HHS’s regulatory agenda moved it to long-term actions, with a final rule not expected before mid-2027. Many of its proposals already reflect good practice, so designing AI systems to meet them now reduces future rework.

Beyond HIPAA

• Clinical decision support: software that informs diagnosis or treatment may be regulated by the FDA; see the four non-device criteria in our RAG in healthcare guide.
• Non-discrimination: rules such as Section 1557 of the Affordable Care Act address discrimination in health programmes, including through tools.
• State laws: many states have health privacy and breach laws that add requirements.
• Other countries: health data in other jurisdictions falls under their own laws.

Worked example: a documentation assistant

A hypothetical clinic group wants an assistant that drafts visit summaries from clinicians’ notes for review.

1. Scope and risk: the assistant drafts; clinicians review and sign. No autonomous changes to the record. Privacy and clinical leads approve the use case.
2. Data flow: notes are sent from the EHR through the AI gateway to a BAA-covered model endpoint configured with no training and minimal retention. Only the current visit’s note and relevant problem list are sent — not the full record.
3. Logging: the gateway logs request metadata (user, patient identifier, time, model version, token counts) but stores prompt and response content only in the EHR, where the signed summary belongs.
4. Evaluation: a de-identified set of 200 historical notes is used to measure accuracy and omissions, reviewed by clinicians before pilot.
5. Access and audit: only clinicians with access to the patient can trigger the assistant, and each use is recorded in the audit log.
6. Incident readiness: the incident plan covers wrong-patient summaries and data exposure, with a kill switch to disable the feature.

The design keeps PHI in the EHR and covered services, limits what the model sees and gives the compliance team a clear picture of every data flow.

Common pitfalls

• Sending PHI to a model endpoint not covered by a BAA during prototyping.
• Logging full prompts and responses in tools without BAAs or retention limits.
• Building vector indexes of patient records without access filtering.
• Using real patient data in test sets shared broadly with developers.
• Assuming de-identified free text is safe without review.
• Treating compliance as a one-time sign-off rather than ongoing risk management.

Frequently asked questions

Can we use large language model APIs with PHI?

Yes, if the provider signs a BAA covering the specific service, data handling meets your requirements and your risk analysis supports it. Consumer chat tools are not appropriate for PHI.

Are embeddings PHI?

Treat embeddings derived from PHI as PHI. They can be linked back to source records and, in some cases, reveal content.

Do we need patient consent for AI use?

It depends on the use. Many treatment, payment and operations uses do not require separate authorisation under HIPAA, but other laws and your organisation’s policies may. Ask your privacy officer.

How do we test AI features without exposing PHI?

Use de-identified or synthetic data where possible, and restrict any real-data testing to controlled environments with access logging.