Fine-tuning vs RAG is the wrong question. For most teams, the answer is a ladder, not a binary: try prompting and context first, then RAG, then tool use, and reach for fine-tuning last. Each rung is cheaper and faster to change than the one above it, and the two cheapest rungs fix most of the problems teams pay to fine-tune away.
And the rung most teams jump to first can backfire: fine-tuning a model on genuinely new facts has been shown to increase its hallucinations (Gekhman et al., 2024). The two interventions that resolve most real failures sit lower and cost less: better prompts and better retrieval. Climb in order, stop at the first rung that works, and you save weeks of effort and a recurring training bill.
Fine-tuning vs RAG: use a ladder, not a binary
Treat optimization as four rungs you climb in order: prompt and context engineering, then RAG, then tool use, then fine-tuning. OpenAI's accuracy guide frames model failures as either a context problem or a behavior problem, and it tells you to start with prompt engineering because that is often the only method needed for tasks like summarization, translation, and code generation. RAG fixes missing or stale knowledge. Fine-tuning fixes inconsistent format, tone, or reasoning. They solve different problems, so treating them as rivals hides the real decision.
The reason to go in order is economic. Editing a prompt takes minutes. Adding documents to a retrieval index takes hours. Wiring up a tool takes a day or two. Fine-tuning takes a labeled dataset, a training run, and retraining whenever the underlying facts or base model change. Climbing costs more at every step, so the rule is simple: climb only when the rung below has provably failed a real eval. Most teams skip three rungs and pay for the most expensive one first.
The try-in-this-order rule: prompting and context, then RAG, then tool use, then fine-tuning. Stop at the first rung that hits your accuracy bar. Only climb when you have a failing eval that the current rung cannot fix.
Step 1: prompting and context engineering, the cheapest fix
Start here, because it is free and instant. Most accuracy problems are not knowledge problems or behavior problems. They are instruction problems: a vague task description, a missing output schema, no examples, or relevant context that you simply forgot to put in the window. Tightening the prompt and supplying the right context in the request often closes the gap with zero infrastructure.
The shift from one-off clever wording to deliberately assembling everything a model needs in its window is why people now talk about context engineering rather than prompt tricks, which our piece on how context engineering won covers. Few-shot examples, a fixed output format, a system message that pins the role, and the right reference text pasted inline resolve a large share of failures before you build anything at all.
- Add 2 to 5 worked examples (few-shot) so the model sees the pattern you want.
- Specify the exact output format: JSON keys, headings, length limits.
- Paste the relevant reference text directly into the prompt when it is small enough to fit.
- Pin the role and constraints in a system message instead of repeating them every turn.
- Re-run your eval after each change so you know which edit moved the number.
If the facts the model needs are small and stable enough to paste into every request, you do not need RAG yet. RAG is what you reach for when that context gets too big, too private, or changes too often to inline.
Step 2: RAG, when you need fresh or private facts
Climb to RAG when the model needs knowledge it does not have: information published after its training cutoff, or proprietary documents that were never in its training set. Lewis and colleagues introduced retrieval-augmented generation in 2020. It pairs the model's internal parametric memory with a non-parametric memory: a searchable index, originally a dense vector index over Wikipedia. At query time you retrieve the relevant passages and feed them into the prompt, so the model answers from fresh, cited evidence instead of from memory alone.
RAG's big advantage is update cost. New facts arrive, you add documents to the index, and the system reflects them immediately. No retraining. That makes it the right tool for a knowledge base that changes weekly, a product catalog, support tickets, or any corpus you cannot or will not bake into model weights. It also grounds answers in retrievable sources, which is one of the more reliable ways to cut fabrication, as our honest guide to hallucination mitigation covers.
RAG is not free either. Quality depends on retrieval quality, and retrieval quality depends heavily on how you split your documents. Chunks that are too large dilute relevance; chunks that are too small lose context. Getting this right is its own project, which is why we wrote a dedicated RAG chunking strategy recipe. Bad chunking is the most common reason a RAG system underperforms while the team blames the model.
Step 3: tool use, when the model needs to act or compute
Use tool use when the model needs to do something rather than know something: run a calculation, query a live database, send an email, or hit an external API. Function calling lets the model emit a structured JSON object naming a function and its arguments. Critically, the model does not execute anything. It produces the call; your code runs it and returns the result. That separation is the whole design, and we break it down in our explainer on LLM function calling.
This rung sits above RAG because retrieval is really a special case of tool use: a read-only lookup. Tool use generalizes it to live, dynamic, and write actions. When a question needs current data the model cannot have memorized, like an account balance or today's inventory, a tool fetches it on demand. When the task needs exact arithmetic or a deterministic operation, a tool guarantees correctness instead of trusting the model to compute it. Neither fine-tuning nor RAG can substitute for an action that has to actually happen in another system. A model that knows your refund policy cold still cannot issue the refund.
- Live data that changes by the second: prices, balances, availability, status.
- Deterministic computation: math, date arithmetic, unit conversion, code execution.
- Write actions: create a ticket, send a message, update a record.
- Multi-step workflows that chain several systems together.
Step 4: fine-tuning, the rung most teams reach for too early
Fine-tune when prompting cannot reliably produce the behavior you need, not when you need new facts. OpenAI's guidance is explicit: reach for fine-tuning when the model produces inconsistent formatting, when the tone or style is wrong, or when it does not follow your reasoning steps consistently. Fine-tuning bakes a behavior into the weights so you no longer have to spell it out in every prompt. It is for how the model responds, not what it knows.
Good fine-tuning targets are narrow and stylistic: a fixed JSON schema the model keeps drifting from, a house writing voice, a classification convention, a domain's structural patterns. These are things you can demonstrate with examples but struggle to fully describe in a prompt. If you can express the requirement clearly in words and the model follows it, you do not need to fine-tune. Climb this rung only after prompting has demonstrably plateaued on a behavioral target.
| Rung | Fixes | Change cost | Retrain on knowledge change? |
|---|---|---|---|
| Prompt and context | Vague instructions, missing inline context | Minutes, no infra | No |
| RAG | Missing, stale, or private facts | Hours, index plus retriever | No, just update the index |
| Tool use | Live data, computation, taking actions | Days, wire up functions | No |
| Fine-tuning | Inconsistent format, tone, reasoning style | Weeks, labeled data plus training | Yes, retrain when behavior or base model changes |
The real cost of fine-tuning: data and knowledge drift
Fine-tuning's true price is not the training run. It is the labeled dataset you must build and the retraining treadmill you sign up for. You need enough high-quality, consistent examples to teach the target behavior, and for fine-tuning, quality of training data matters more than quantity. Producing that data is work that recurs every time the target shifts.
Knowledge drift is the trap. If you fine-tune facts into a model and those facts change, the model is now confidently wrong, and the only fix is to retrain. RAG sidesteps this entirely because you just update the index. The treadmill compounds when you upgrade the base model: a new version means re-running your whole fine-tuning pipeline on the new base. RAG and tool use ride along on model upgrades for free; fine-tuning does not.
What fine-tuning does not fix (and people expect it to)
Fine-tuning is a poor way to add new knowledge, and trying to use it that way can make accuracy worse. Gekhman and colleagues studied exactly this and found that large language models mostly acquire factual knowledge during pretraining, while fine-tuning teaches them to use that knowledge more efficiently. When the fine-tuning set contains genuinely new facts, the model learns those examples slowly, and as it does, its tendency to hallucinate rises in proportion. So fine-tuning to teach facts can degrade the very reliability you wanted.
- It does not reliably inject new facts; that is RAG's job, and forcing it can increase hallucination.
- It does not give the model live or changing data; that is tool use.
- It does not replace good prompting; if the prompt is vague, a fine-tuned model is still guessing.
- It does not remember your last conversation; weights are static between training runs.
Fine-tuning sets a model's default behavior once, but gives the model no memory of specific past interactions with a specific user. A fine-tuned support model still forgets what a customer told it yesterday. Persisting that kind of state is a separate concern from any of the four rungs, and it is where an external memory layer like MemX fits: it stores durable, per-user context across sessions and feeds it back at query time, so you get continuity without retraining anything. It complements the ladder rather than replacing a rung.
How to decide: RAG, fine-tuning, or prompting?
Walk the rungs against a real eval, and stop at the first one that passes. The rule is mechanical, which is the point: it removes the temptation to jump straight to the most expensive option.
- Is the prompt vague, underspecified, or missing context you could just paste in? Fix the prompt and context first.
- Does the model lack facts that are private, fresh, or simply not in its training data? Add RAG.
- Does the task need live data, exact computation, or an action in another system? Add tool use.
- Is the format, tone, or reasoning style still inconsistent after all of the above? Now fine-tune on a clean, narrow dataset.
- Do you need per-user continuity across sessions? Add a persistent memory layer, separate from the four rungs.
In production, these stack. A mature system often runs a tight prompt, RAG for knowledge, tools for actions, and a small fine-tune for house style, all at once. OpenAI notes that teams whose evals show both context and behavior gaps may end up combining RAG and fine-tuning. That is the destination, not the starting point. You arrive there by climbing one rung at a time and keeping only the rungs that earned their place against your eval.
01Is RAG or fine-tuning better?
Neither is universally better; they solve different problems. RAG supplies missing, fresh, or private facts at query time. Fine-tuning bakes in a consistent format, tone, or reasoning style. Most teams should try prompting and RAG before fine-tuning, and mature systems often combine RAG and fine-tuning together.
02When should you fine-tune instead of using RAG?
Fine-tune for behavior, not knowledge: when the model gives inconsistent formatting, the wrong tone, or skips your reasoning steps even with good prompts. Use RAG when the gap is missing facts. Fine-tuning facts is unreliable and can raise hallucinations, so keep facts in retrieval.
03Can fine-tuning add new knowledge to an LLM?
Poorly. Research by Gekhman et al. found LLMs mostly learn facts during pretraining, and a model learns genuinely new facts slowly during fine-tuning, which increases hallucination. Fine-tuning is better at making existing knowledge usable. For new or changing facts, use RAG and update the index instead.
04Why is fine-tuning expensive to maintain?
The recurring cost is data and retraining, not the single training run. You must build a clean labeled dataset, and you must retrain whenever the target behavior changes, the underlying facts drift, or the base model is upgraded. RAG and tool use ride along on model upgrades without retraining.
05What should you try before RAG or fine-tuning?
Prompt and context engineering. Tighten the instructions, add a fixed output format, include a few worked examples, and paste relevant context into the request. OpenAI notes this is often the only method needed for tasks like summarization, translation, and code generation, and it costs minutes with no infrastructure.
