When should a company use RAG instead of fine-tuning?
Posted by Applied AI Commons team (community team) on
Teams adding language models to their products often face an early architecture decision: retrieve relevant documents at query time, adjust the model itself through fine-tuning, or combine both. Each approach has different implications for accuracy, freshness, cost and maintenance.
Members might answer from several angles: - How often the underlying knowledge changes - Whether the goal is new facts or a consistent style and format - What evaluation looks like for each approach - The ongoing effort to keep retrieval data or tuned models current
What questions do you ask when deciding between retrieval, fine-tuning or a mix of the two, and what tends to tip the decision?
5 replies
Applied AI Commons team (starter reply from the community team),
A few questions that often clarify the decision: - How often does the underlying knowledge change? Frequent change favours retrieval. - Do answers need to cite sources? Retrieval makes that much easier. - Is the problem missing facts, or inconsistent style and format? Fine-tuning is better suited to the latter. - Do different users need access to different documents? Retrieval can respect permissions at query time. If most answers point towards facts and freshness, retrieval is usually the simpler starting point.
Applied AI Commons team (starter reply from the community team),
Evaluation looks different for each approach. For retrieval, it helps to measure two things separately: whether the right documents were found, and whether the answer used them faithfully. Many failures blamed on the model are actually retrieval failures. For fine-tuning, evaluation usually focuses on whether the model follows the target format or behaviour consistently, and whether it lost general ability in the process. In both cases, a fixed test set built from real questions is more useful than ad hoc checking.
Applied AI Commons team (starter reply from the community team),
Ongoing maintenance is easy to underestimate. Retrieval systems need documents kept current, duplicates and outdated versions removed, and chunking and indexing revisited as content changes. Fine-tuned models need retraining when requirements shift or a newer base model becomes available, which can mean rebuilding training data and repeating evaluation. Estimating this ongoing effort, not just the initial build, often changes which option looks cheaper over time.
Applied AI Commons team (starter reply from the community team),
A practical order that many teams follow: start with careful prompting, then add retrieval if the model needs facts it does not have, and consider fine-tuning only once there is clear evidence that prompting and retrieval cannot achieve the required consistency. Each step adds complexity, so it is worth confirming the simpler option has genuinely reached its limits. This also means you have a baseline to compare against if you do eventually fine-tune.
Applied AI Commons team (starter reply from the community team),
A common misconception is that fine-tuning is a reliable way to teach a model new facts. In practice, it tends to be better at shaping behaviour, tone and format than at storing knowledge that can be recalled accurately, and it offers no easy way to show where an answer came from. Combining both, such as a lightly tuned model that uses retrieved documents well, can make sense at scale. What factors have tipped the decision for members, and have you ever switched approaches midway?