3 min read
RAG or fine-tuning? Start with what's missing
Most teams pick between retrieval and fine-tuning too early. The better first question is whether the model lacks knowledge or lacks behaviour, and often the answer is neither.
"Should we use RAG or fine-tune?" is one of the most common questions in applied AI, and it usually comes too early. It assumes the model needs changing. Often it doesn't. When it does, the two techniques fix different problems, so picking one is less a matter of taste than of working out what's actually wrong.
We use one question to sort it out. When the model gets something wrong, is it missing knowledge or missing behaviour?
Missing knowledge
The model doesn't know your refund policy, last week's price change or what's in a contract it has never seen. No amount of clever prompting fixes that, because the facts aren't in the model.
This is the problem retrieval solves. You keep your documents in a searchable store, find the passages relevant to each request and put them in the prompt. The model answers from text it can see.
Retrieval has three properties that matter more in practice than any benchmark:
- Updates are cheap. Change a document and the next answer uses it. Nothing gets retrained.
- Answers can cite their sources. When the model is wrong, you can see which passage it read, which turns "the AI is broken" into a fixable search problem.
- Access control is possible. You can filter what gets retrieved by who's asking. Facts baked into model weights can't be hidden from one user and shown to another.
Most failures in a retrieval system are search failures, not model failures. The right passage wasn't found, or it was split across two chunks, or a near-duplicate outranked it. If answers are poor, look at what was retrieved before blaming the model.
Missing behaviour
The model knows enough, but it doesn't act the way you need. It writes in the wrong register, ignores a required format, or handles a narrow task inconsistently even with good instructions.
This is where fine-tuning earns its cost. Training on a few hundred to a few thousand good examples teaches a pattern more reliably than describing it. It's also how you get a smaller, cheaper model to do one task as well as a larger general one, which matters when the task runs millions of times.
What fine-tuning is bad at is teaching facts. A model trained on your documents will reproduce their style and some of their content, but unreliably, with no way to tell which parts it remembered correctly and no simple way to update it when the documents change.
Often it's neither
Before either technique, try the boring options. They solve more problems than people expect.
- Put the documents in the prompt. Context windows are now large enough to hold a whole policy manual or a long contract. If your knowledge fits, you may not need a retrieval system at all, and prompt caching makes repeating the same large context much cheaper than it used to be.
- Write better instructions and examples. A clear task description with three or four worked examples fixes a surprising share of "behaviour" problems.
- Use a stronger model. Fine-tuning a weaker model to match a stronger one is sometimes worth it for cost. It's rarely worth it as the first step.
These options are fast to try and easy to undo. Both RAG and fine-tuning add infrastructure you'll maintain for as long as the feature exists.
A way to decide
| If the model is wrong because... | Try first | Then |
|---|---|---|
| It doesn't have the facts | The documents in the prompt | Retrieval, once they no longer fit |
| The facts change often | Retrieval | |
| It ignores format or tone | Better instructions and examples | Fine-tuning |
| It's too slow or costly at scale | A smaller model with good prompts | Fine-tuning the smaller model |
The combination is common in mature systems. Fine-tuning sets how the model behaves and retrieval supplies what it knows. But we'd treat that as an end state you grow into, not a starting architecture.
Measure before choosing
The most useful thing to build first isn't a pipeline. It's a small test set of real requests with answers you'd accept. Fifty is enough to start. Run every option against it. Without one, "RAG or fine-tuning" is a debate about preferences. With one, it's a comparison of numbers, and the boring option often wins.
More notes