Use RAG (retrieval-augmented generation) when the model needs facts it doesn't have, such as your documents, recent data or anything that changes, because it looks them up at answer time and can cite them. Use fine-tuning when the model already has the knowledge but needs to behave differently every time, such as a fixed output format, a house style or a narrow classification task, because it bakes examples into the model's weights.
Shorthand: RAG changes what the model knows for this request; fine-tuning changes how it responds to every request. A sensible order is a good prompt first, then RAG, and fine-tuning only if a measured gap remains.
What RAG is
RAG retrieves relevant text from your own data and puts it into the prompt before the model answers. The term comes from Lewis et al. (2020), who combined a model's "parametric memory" (what it learned in training) with a "non-parametric memory" it can search. Their abstract names the problems this addresses: "providing provenance for their decisions and updating their world knowledge remain open research problems" for models that rely on their parameters alone.
A typical pipeline: split documents into chunks, embed each chunk, store the vectors in a vector database, then at question time embed the question, retrieve the closest chunks and pass them to the model with an instruction to answer from them. OpenAI's embeddings guide gives the motivation: supplying facts through search instead of putting all your information in the context window, which raises token costs. See what a vector database is for the retrieval half.
What fine-tuning is
Fine-tuning continues training a pre-trained model on your own examples, so the new behaviour is stored in its weights. In supervised fine-tuning you provide pairs of input and ideal output. OpenAI's guide lists the jobs it fits best:
- Classification
- Nuanced translation
- Generating content in a specific format
- Correcting instruction-following failures
Its stated benefits over prompting alone include fitting more examples than a single prompt can hold, shorter prompts that save "token costs at scale and can be lower latency", and training "a smaller, cheaper, faster model to excel at a particular task".
For open models, parameter-efficient methods make fine-tuning much cheaper than retraining everything. LoRA trains small low-rank matrices beside frozen weights; its authors report reducing trainable parameters "by 10,000 times" and GPU memory by 3 times compared with fully fine-tuning GPT-3 175B, with "no additional inference latency".
Fine-tuning vs RAG compared
| RAG | Fine-tuning | |
|---|---|---|
| Changes | What the model sees in this request | The model's weights |
| Best for | Facts, documents, changing or private knowledge | Format, tone, task behaviour, consistent structure |
| Updating knowledge | Re-index the changed documents | Train again |
| Can cite sources | Yes, the retrieved chunks are known | No |
| Up-front cost | Build an ingestion and retrieval pipeline | Collect and label examples, run training, evaluate |
| Per-request cost | Higher: retrieved text adds input tokens, plus a retrieval step | Can be lower: shorter prompts, possibly a smaller model |
| Main failure mode | Retrieves the wrong chunks, so the model answers from the wrong text | Learns the wrong pattern from inconsistent examples |
| Data needed to start | Your documents as they are | Curated input/output examples (OpenAI minimum: 10) |
When to use RAG
- Answering questions over your documentation, policies, tickets or codebase.
- Anything that changes faster than you'd retrain: prices, stock, schedules, news.
- When answers must show their sources, for checking or compliance.
- When different users may see different data. Retrieval can filter by permissions per request; a fine-tuned model can't unlearn something for one user.
RAG also reduces, but doesn't eliminate, made-up answers: the model can still misread or ignore the retrieved text. Why AI hallucinates covers that side.
When to use fine-tuning
- Every output must follow a strict structure, and prompting alone keeps drifting from it.
- A classification or extraction task with many labelled examples.
- A consistent voice or style that's hard to describe but easy to demonstrate.
- Moving a task from a large, expensive model to a smaller one that's been trained on it.
- Very long prompts full of examples, repeated on every call, that you'd like to shrink.
Fine-tuning needs an evaluation set first. OpenAI's guide says you "need a reliable way to determine whether your fine-tuned model is performing better than a base model", and suggests starting with about 50 well-crafted examples; if those have no effect, rethink the task or the prompt before adding more data.
A practical availability note (October 2026)
Check that the option exists for the model you use. As of October 2026, OpenAI's documentation says it "is winding down the fine-tuning platform": it is closed to new users, existing eligible customers can create training jobs until January 6, 2027 (or until the base model shuts down, if sooner), and fine-tuned models stay available for inference until their base models are deprecated. Other providers and open-weight models have their own options and timelines, so check the current docs for the model you plan to use before designing around fine-tuning.
Using both together
The two solve different problems, so they combine well. One way to combine them is RAG for the facts plus a fine-tuned model that has learned how to use retrieved context: answering only from the provided passages, citing them in your format, and saying "not found" when the passages don't contain the answer. Build the RAG system first; it is easier to change, and its failure cases tell you whether fine-tuning would help at all.
How to decide, step by step
- Write 20 to 50 realistic test questions with the answers you expect. This is your evaluation set for every option below.
- Try a strong prompt with a few examples. OpenAI notes that prompt engineering "may be all you need". See how to write a good prompt.
- If failures are about missing or outdated facts, add RAG.
- If failures are about format, consistency or style, and you have good examples, try fine-tuning.
- Re-run the same evaluation after each change and keep only what measurably helps.
Frequently asked questions
- Can fine-tuning teach a model new facts?
- Facts that appear in training examples can influence answers, but it is an unreliable way to add knowledge: you can't easily check what was learned, you can't cite where an answer came from, and updating a fact means training again. For knowledge that changes or must be traceable, retrieval is the better fit. Fine-tuning is strongest for teaching format, style and task behaviour.
- Is RAG cheaper than fine-tuning?
- RAG has no training cost, but every request carries the retrieved text in the prompt, so you pay for those extra input tokens on every call. Fine-tuning costs money and effort up front and can then allow shorter prompts. Which is cheaper overall depends on request volume and how much context each request needs; estimate both with your own numbers.
- How much data do I need to fine-tune?
- For OpenAI's supervised fine-tuning, the minimum is 10 examples, and OpenAI suggests starting with about 50 well-crafted examples and evaluating before adding more. The quality and consistency of the examples matters more than the count. Other providers and open-source tools set their own limits.
- Should I try prompt engineering first?
- Yes. OpenAI's own guidance is that prompt engineering 'may be all you need' for many use cases. A clear prompt with a few examples is the quickest thing to test, and the evaluation set you build for it is exactly what you will need later to judge whether RAG or fine-tuning actually helped.