A team asks an AI assistant about a policy that changed last week. The answer sounds fluent but describes last year’s rules. Should the team fine-tune the model, connect a document library, or improve the instructions? The right choice depends on what is failing. A knowledge-access problem and a behavior problem need different tests.
At a glance
- Retrieval-augmented generation, or RAG, supplies relevant material to a model when it answers.
- Fine-tuning changes model behavior through additional training; it is not a live document search.
- Neither approach makes an answer automatically accurate, current, or safe.
Start with the failure, not the technique
Write down three real questions your system gets wrong. Record the expected answer, the source that supports it, and the consequence of an error. If the correct information is missing from the supplied context, investigate retrieval. If the facts are available but the output repeatedly uses the wrong format or classification, investigate instructions, examples, and possibly fine-tuning.
NIST defines RAG as a generative system paired with a separate retrieval system or knowledge base. That separation matters: you can update a document collection without treating every edit as a new model-training project. However, finding a document and answering it correctly remain separate steps.
A comparison that starts with reader needs
| Question | Retrieval approach | Training approach |
|---|---|---|
| Where are changing facts kept? | In a maintained source collection | Not a substitute for a current source collection |
| What should you inspect? | Retrieved passages, dates, permissions, and citations | Training examples, labels, behavior, and evaluation results |
| What often fails? | The right passage is missed or misread | The learned pattern fails on a new type of input |
A small test before a large investment
Consider a fictional retailer with 20 delivery policies. Create 12 questions: four answered directly in the policies, four requiring two passages, and four whose answers are absent. Run the same questions against a basic prompt, then a retrieval prototype. Keep the questions and grading rules fixed. This is a worked example, not a reported product benchmark.
For each answer, check whether its supporting passage actually contains the claim. Score unsupported answers separately from incomplete answers. A system that says it cannot find a policy can be more useful than one that invents a confident rule. Add a recently revised document and check whether the system favors the current version.
Retrieval brings its own security questions
A retrieved page can contain instructions as well as facts. OWASP warns that indirect prompt injection can arrive through external material. A document telling an assistant to ignore its task should be treated as untrusted content, not as permission to send messages or disclose records. Limit tool permissions and review consequential actions at the application boundary.
Also test access controls. A helpful answer is still a failure if it reveals a document the user was never allowed to read. Separate public documentation from internal records, and confirm that retrieval respects the requesting user’s permissions.
A decision checklist
- Can a clearer instruction or a few examples solve the problem?
- Does the task depend on facts that change frequently?
- Can you identify and maintain authoritative documents?
- Do you have representative examples and a fair evaluation set?
- Can the system decline to answer when evidence is missing?
Choose the smallest approach that passes the test. Combining retrieval and training may eventually be appropriate, but complexity should earn its place through measured improvement. For checking the evidence behind individual answers, see our source-verification workflow.
Separate a retrieval failure from an answer failure
Keep two checks in your evaluation record: whether the required passage was retrieved, and whether the final answer used that passage correctly. If retrieval misses the policy, changing the answer format will not repair the missing evidence. If the passage is present but the answer misquotes it, retrieval success alone is insufficient.
Use a held-out set of questions that was not used to choose prompts or training examples. Read our guide to task-level AI evaluation when defining success, then use a claim-by-claim source ledger to inspect evidence. This is an evaluation design, not a claim that a particular model passes it.
Sources and limitations
- NIST: fine-tuning definition
- NIST: retrieval-augmented generation definition
- OWASP: prompt injection prevention
- NIST: ARIA evaluation planning manual
Reviewed October 17, 2026. This is a practical explanatory framework, not a hands-on comparison of commercial products. The test scenario is illustrative.
To diagnose retrieval failures before changing model training, see RAG Chunking: Preserve the Evidence Before You Tune the Answer and RAG Retrieval Evaluation: Check the Evidence Before Grading the Answer.