RAG or fine-tuning? Giving an AI model your company knowledge

PoliteTech Editorial · Generative AI and LLMs

Two different problems

Retrieval-augmented generation (RAG) gives a model the right information at the moment it answers, by searching your documents and inserting relevant passages into the prompt. Fine-tuning changes how a model behaves by training it on examples. One supplies knowledge; the other shapes style, format or specialised skills.

When RAG is the right choice

• Your information changes often: policies, prices, product details, documentation.
• Answers must cite sources so users can verify them.
• Different users may see different documents, so permissions matter.
• You want to start quickly without collecting training data.

When fine-tuning helps

• You need a consistent output format or tone across thousands of requests.
• The task is narrow and repeated, such as classifying tickets into your own categories.
• A smaller fine-tuned model can replace a larger one to reduce cost or latency.
• You have enough high-quality examples of correct outputs.

Where projects go wrong

The most common problem is poor retrieval rather than a weak model: documents split in the wrong places, missing metadata, outdated copies or no permission filtering. Another is skipping evaluation. Without a set of real questions and expected answers, it is impossible to tell whether a change made things better or worse.

Combining both

Mature systems often use both: RAG supplies current facts with citations, while a fine-tuned or carefully prompted model produces answers in the required structure. Start with RAG and good prompts, measure quality on your own questions and only add fine-tuning when the evaluation shows a clear gap.

A practical starting plan

• Choose one knowledge domain and collect 30–50 real questions with known answers.
• Index the relevant documents with metadata and access rules.
• Measure retrieval accuracy and answer faithfulness on your question set.
• Improve chunking, search and prompts before considering fine-tuning.
• Decide hosting based on data policy: hosted API, your cloud or private deployment.