RAG in Healthcare: Use Cases, Architecture and Safeguards for Grounded AI Assistants

Healthcare organisations hold enormous amounts of knowledge: clinical guidelines, formularies, care pathways, policies, payer rules, research and patient records. Staff spend a great deal of time finding the right passage in the right version of the right document. Language models promise to help, but a model answering from its training data alone can be outdated, generic or simply wrong — unacceptable where decisions affect patients.

Retrieval-augmented generation (RAG) addresses part of that problem. Instead of answering from memory, the system first retrieves relevant passages from approved sources and then asks the model to answer using only those passages, with citations. This guide explains where RAG is useful in healthcare, where it is not, and what it takes to build a system clinicians, compliance teams and patients can trust. It is written for product and technology leaders; it is not clinical or legal advice.

How RAG works, in plain terms

A RAG system has two phases.

Preparing the knowledge (offline). Approved documents are collected, cleaned, split into passages, tagged with metadata such as source, version, date and access level, and converted into embeddings for search. Many systems also keep a keyword index, because exact terms such as drug names, codes and dosages matter.

Answering a question (at run time).

1. The user asks a question.
2. The system checks who the user is and what they are allowed to see.
3. It retrieves the most relevant permitted passages using semantic and keyword search, then re-ranks them.
4. It sends the question and passages to the model with instructions to answer only from the passages, cite them and say when the answer is not found.
5. It checks the response — for citations, unsupported claims and policy violations — before showing it, with links to the sources.

If you are deciding between RAG and training a model on your own data, our guide to RAG versus fine-tuning compares the two. In healthcare, RAG is usually the starting point because knowledge changes often and every answer must be traceable to a source.

Where RAG helps, and where it should not be used

The pattern is clear: start where answers support professionals with information they can verify, and where an error is caught before it affects care.

The regulatory line: decision support and medical devices

In the United States, the FDA’s final guidance on Clinical Decision Support Software, issued on 28 September 2022, explains when software that supports clinical decisions falls outside the definition of a medical device. A software function must meet all four criteria to be considered non-device clinical decision support:

1. It does not acquire, process or analyse medical images, signals from in vitro diagnostic devices, or patterns or signals from a signal acquisition system.
2. It displays, analyses or prints medical information about a patient or other medical information, such as peer-reviewed studies or clinical guidelines.
3. It supports or provides recommendations to a healthcare professional about the prevention, diagnosis or treatment of a disease or condition.
4. It enables that professional to independently review the basis for the recommendations, so they do not rely primarily on the software to make a diagnostic or treatment decision.

RAG’s design helps with the fourth criterion — citations let professionals see the basis of an answer — but citations alone do not make a product non-device. Tools aimed at patients, or that replace rather than support professional judgement, face a different analysis. Other countries have their own rules, such as the TGA in Australia, the MHRA in the UK and the EU Medical Device Regulation. Involve regulatory advisers early if the use case touches diagnosis or treatment.

Privacy and security: HIPAA and beyond

If the system processes protected health information for a US covered entity or business associate, the HIPAA Security Rule applies. It requires administrative, physical and technical safeguards, a documented risk analysis and risk management. Some implementation specifications are required and others are “addressable”, which means you must assess them and document what you did and why. Documentation must be retained for six years.

For a RAG system this translates into practical design decisions:

• Data minimisation. Index only what the use case needs. A policy assistant does not need patient records.
• Retrieval-time access control. Apply the user’s permissions as a filter during search, so restricted passages are never retrieved, rather than hiding them after generation.
• Vendor agreements. Use model and hosting providers that will sign a business associate agreement where required, and confirm data is not used to train their models.
• Encryption and isolation. Encrypt data in transit and at rest, including the vector index, and separate tenants or facilities where required.
• Audit logging. Record who asked what, which passages were retrieved and what was answered. Treat the logs themselves as sensitive.
• Retention. Define how long prompts, responses and logs are kept, and redact identifiers where they are not needed.

Outside the US, equivalent obligations come from laws such as Australia’s Privacy Act, the UK GDPR and the EU GDPR. The OWASP Top 10 for LLM Applications is a useful technical checklist, especially sensitive information disclosure (LLM02) and vector and embedding weaknesses (LLM08).

A reference architecture

Preparing clinical documents for retrieval

Most RAG quality problems start in ingestion, long before the model sees a question. Clinical content has features that generic pipelines handle badly:

• Tables and dosage schedules. Keep each table as one unit, or split by row while repeating the header, so a dose is never separated from the drug, route and population it applies to.
• Section hierarchy. Prefix each passage with its document title and section path — for example, “Sepsis pathway > Adults > Antibiotic timing” — so a retrieved passage carries its context.
• Abbreviations and synonyms. Maintain a local list (for example, MI and myocardial infarction, or brand and generic drug names) and use it to expand queries and keyword indexes.
• Versions and effective dates. Store approval date, review date and status. Exclude drafts and superseded versions from the index, not just from the display.
• Scanned documents. Run optical character recognition with quality checks; poor scans produce silent retrieval failures.
• Local variation. Where a network has site-specific procedures, tag passages by site so staff see the version that applies to them.

Worked example: a policy assistant for a hospital network

Consider a hypothetical network of four hospitals with around 2,500 approved policies and procedures spread across an intranet and shared drives. Nurses and junior doctors report spending several minutes finding the right procedure during a shift, and often ask colleagues instead.

Scope. The pilot answers questions about approved policies and procedures for clinical and administrative staff. It does not access patient records and does not answer questions outside the policy library; those receive a referral to the relevant department.

Design choices.

• Sources are limited to documents with an owner, an approval date and an “active” status in the document management system. A nightly sync adds, updates and retires passages automatically.
• Each passage carries site, department and sensitivity tags. A staff member at one hospital sees network-wide policies plus their own site’s procedures.
• Retrieval combines keyword and semantic search with an abbreviation list built with the nursing education team.
• Answers are short, quote the key step where helpful, and always show the policy title, section, version and approval date with a link to the full document.
• When the retrieved passages do not answer the question, the assistant says so and suggests the department that owns the topic.

Evaluation. Clinical educators wrote 200 questions from real enquiries, including 30 that should be refused because they ask for clinical advice or fall outside the library. The pilot proceeds to shadow use only when agreed thresholds for correct retrieval, groundedness, citation accuracy and correct refusal are met, and no answer contradicts an active policy.

What to measure in use. Time to find a procedure (sampled through short surveys), the share of answers users mark as helpful, reported errors and their causes, and the number of questions referred because no policy covers them — which often reveals gaps in the policy library itself.

Measuring value

What drives the cost of a healthcare RAG project

The model is rarely the largest cost. Budget for document clean-up and ownership (often underestimated), ingestion pipelines and connectors to document systems, access-control integration with identity systems, evaluation work with clinicians — whose time is scarce and expensive — security review, hosting with appropriate agreements, and ongoing monitoring and content maintenance. A narrow pilot on a well-maintained document set is far cheaper than one that first has to fix years of unmanaged content.

Common failure modes and how to reduce them

• Outdated sources. The system cites a superseded guideline. Use document lifecycle management and show the version and approval date with each citation.
• Retrieval misses. The right passage exists but is not found, often because of abbreviations or synonyms. Combine keyword and semantic search, add a synonym list and test with real staff questions.
• Unsupported statements. The model adds details not in the passages. Use strict instructions, automated groundedness checks and an “I could not find this in the approved sources” response.
• Context loss from chunking. A dosage table split in half loses meaning. Chunk by document structure, not fixed length.
• Over-trust. Users stop checking sources. Design the interface to put sources first, and train users on limitations.
• Data leakage. Retrieval returns a passage the user should not see. Enforce access control in the retrieval layer and test it explicitly.

How to evaluate before go-live

1. Build a question set with clinicians and domain experts: 100 to 300 real questions covering common, difficult and out-of-scope cases, each with the correct answer and source.
2. Measure retrieval: does the correct passage appear in the top results?
3. Measure answers: correctness, groundedness (every statement supported by a cited passage), citation accuracy and appropriate refusal for out-of-scope questions.
4. Test safety cases: prompt injection in documents, attempts to obtain restricted information and requests for clinical advice outside scope.
5. Agree thresholds with clinical governance before testing, for example a minimum groundedness rate and zero tolerance for certain error types.
6. Run a shadow phase where staff use the system alongside existing processes and report discrepancies.
7. Monitor continuously after launch, with periodic review of sampled answers and every user-reported error.

The NIST AI Risk Management Framework and its Generative AI Profile provide a structure — govern, map, measure and manage — that many healthcare organisations use to document this work.

A phased rollout plan

Frequently asked questions

Does RAG eliminate hallucinations?

No. It reduces them by grounding answers in retrieved sources, but models can still misread or add to the passages. Groundedness checks, citations and human review remain necessary.

Can we use patient records in a RAG system?

It is technically possible, but it raises the bar for privacy, access control, vendor agreements and audit. Many organisations start with policies and guidelines, then add record summarisation once controls are proven.

Is a RAG assistant a medical device?

It depends on its intended use and functions. Tools that support professionals with verifiable information may fall outside device regulation; tools that diagnose, recommend treatment to patients or analyse signals or images are more likely to be regulated. Get regulatory advice for your specific product and market.

Which documents should we start with?

Approved, frequently used, well-structured documents with clear owners — such as policies, procedures and coding guidance. Avoid sources with unclear versions or ownership.

How do we keep answers up to date?

Connect ingestion to your document management process so that approvals, updates and retirements flow through automatically, and show the version and date with each citation.