Fine-tuning vs RAG vs prompting: when to use which
Start with prompting, always. Add RAG (retrieval-augmented generation) when the model is missing knowledge, such as your private documents or facts that change often. Use fine-tuning only when the model is missing behaviour, like a strict format or style, that good prompts and examples cannot fix. Let a small test set decide, not hype.
- 1
Three ways to make an AI fit your job
OpenAI explains the choice with an exam analogy. Prompting and RAG are like letting the model sit the exam with the textbook open: you put the right information in front of it, in the prompt. Fine-tuning is like months of classes beforehand: the model learns from many examples and its internal settings change.
In plain terms: prompting means writing better instructions and examples. RAG means automatically finding the relevant pages from your documents and adding them to the prompt. Fine-tuning means training an existing model further on your own examples, which changes its weights (the numbers it learned in training).
- Prompting: better instructions
- RAG: fetch your documents into the prompt
- Fine-tuning: retrain on your examples
Key ideaPrompting and RAG change what the model sees. Fine-tuning changes the model itself.
- 2
Always start with prompting and a test set
OpenAI calls prompt engineering “typically the best place to start”, and Google’s tuning guide says the same: start with prompting to find the best prompt. It costs nothing to set up and you see results immediately.
Before you go further, build a small test set: 20 to 50 real questions with the answers you expect. This is called an eval. It tells you whether a change actually helped. If your documents are small, you may not even need RAG: Anthropic notes that if your knowledge base is under about 200,000 tokens, you can simply put all of it in the prompt.
A tiny eval for a college helpdesk bot (illustrative) Q: What is the last date to pay semester fees? Expected: 15 July Q: Can I get a refund after the add/drop week? Expected: No, only within 7 days Q: Who do I contact for hostel allotment? Expected: Warden office, Block C ... Score each version of your system: how many of 30 does it get right? - 3
Use RAG when the model lacks knowledge
A model only knows what was in its training data. It has never seen your company’s HR policy, your coaching institute’s fee PDF or this week’s prices. Google describes RAG as combining search with an LLM, so answers are grounded in up-to-date information. Updating knowledge is cheap: change the document and the next answer uses it.
The catch is that RAG is only as good as its search step. Google warns that if the retrieved text is irrelevant, the answer can be “grounded but off-topic or incorrect”. Anthropic showed that better retrieval matters: its Contextual Retrieval method cut failed retrievals by 49%, and by 67% when combined with reranking. You will build this in the Embeddings and RAG lesson.
- Facts are private: internal docs, policies, product manuals
- Facts change often: prices, schedules, stock, news
- Users need citations: “show me where it says that”
- Too much material to fit in one prompt
- 4
Use fine-tuning when the model lacks behaviour
Fine-tuning shines when you need the same kind of output again and again: sorting support tickets, extracting fields from forms, or always answering in one strict format or brand voice. Google lists classification, sentiment analysis and entity extraction as good fits. It can also let a smaller, cheaper model do a job that needed a bigger one, with shorter prompts.
A real example: Anthropic fine-tuned its small Claude 3 Haiku model to moderate forum comments. Accuracy rose from 81.5% to 99.6%, and tokens per query dropped by 85%. The cost is data and upkeep. OpenAI suggests starting with 50 or more high-quality examples; Google says think 100 examples or more. And when your facts change, you have to retrain.
- Good for: consistent format, tone, classification, cheaper specialised models
- Not good for: adding fresh or frequently changing facts
- Needs: hundreds of clean examples, an eval, and time to retrain
- Check availability: OpenAI’s docs say it is winding down its fine-tuning platform and it is closed to new users (as of Nov 2026); Google still offers tuning for Gemini
- 5
Cost and effort, side by side
Here is the trade-off in one view. The order also matches how much work each one takes, which is why every vendor guide says to go left to right.
Prompting vs RAG vs fine-tuning (rough guide) Prompting RAG Fine-tuning Fixes instructions missing knowledge inconsistent behaviour Setup effort minutes days (search pipeline) weeks (data + training) Data needed a few examples your documents 100s of labelled examples Update facts edit the prompt edit the documents retrain the model Shows sources no yes no Extra cost none storage + search training + upkeep - 6
A decision flow you can reuse
Run your eval after every step and only move on if the score is still too low. Diagnose the failure first: wrong or missing facts point to RAG; right facts but wrong format, tone or reasoning pattern point to fine-tuning.
You can combine them. IBM’s example is a legal assistant that uses RAG for recent court decisions, prompting for document format and fine-tuning for firm-specific style. But more is not always better: in one OpenAI case study, adding RAG to a fine-tuned model lowered accuracy. Let the test set decide.
- Clear prompt + eval
- Good enough? Ship it
- Missing facts? Add RAG
- Wrong behaviour at scale? Fine-tune
- Re-run the eval
Key ideaKnowledge problem: RAG. Behaviour problem: fine-tuning. Everything else: a better prompt.
Straight from the source
The original idea, then the same idea taken apart, then how you use it today.
- 1 · Understand
- They paired a model’s built-in memory, the knowledge stored in its parameters, with an outside memory: a searchable index of Wikipedia. The system first retrieves relevant passages, then writes its answer using them, and this gave more specific and more factual answers than the model alone.
- 2 · Break it down
- Parametric memory: what the model learned in training. It is hard to inspect or update.
- Non-parametric memory: documents you can search, read and replace at any time.
- Updating knowledge is a swap: when they replaced a 2016 Wikipedia index with a 2018 one, the model answered questions about the newer world leaders, with no retraining.
- Because each answer is built from retrieved passages, the system can point to where a fact came from.
- 3 · Use it
- When your app must know your own or fast-changing facts, keep them in documents and retrieve them, instead of training them into the model. To change an answer, change the document.
- 1 · Understand
- Instead of retraining all of a model’s weights, LoRA freezes them and trains small extra matrices added to each layer. Compared with fully fine-tuning GPT-3 175B, it cut the number of trainable parameters by 10,000 times and GPU memory by 3 times, with no extra delay when the model answers.
- 2 · Break it down
- The big model stays frozen. Only a small add-on is trained.
- Their bet: the change a model needs for a new task is simple (“low rank”), so a small add-on can capture it.
- Because each add-on is small, one shared base model can be paired with many add-ons, one per task.
- 3 · Use it
- If prompting and RAG are not enough and you fine-tune an open model such as Llama, you will almost certainly meet LoRA. It needs far fewer trainable numbers and less GPU memory than full fine-tuning, which puts fine-tuning within reach of a small budget.
Words to know
- RAG (retrieval-augmented generation)
- Finding the most relevant parts of your documents and adding them to the prompt so the model answers from them.
- Fine-tuning
- Training an existing model further on your own input-output examples so its behaviour changes.
- Weights
- The millions or billions of numbers inside a model that store what it learned. Fine-tuning changes them; prompting and RAG do not.
- Eval (test set)
- A list of real questions with expected answers, used to score whether a change helped.
- Context window
- How much text a model can read at once. If your documents fit, you may not need RAG.
Watch
Hand-picked from the people who explain this best. The video loads only when you press play.
A clear whiteboard comparison of all three, including costs like compute and “catastrophic forgetting”, and how they combine.
Try it
Feel the difference in 10 minutes, no code
- 1Find a short document the model has never seen, such as your college’s attendance rules, a company leave policy or a product FAQ page.
- 2Prompting only: ask Claude or ChatGPT a specific question about it without pasting it. Note whether it guesses.
- 3“Manual RAG”: in a new chat, paste the document inside <document> tags and ask: “Answer only from the document and quote the line you used.” Compare.
- 4“Fine-tuning feel”: give 3 examples of answers in a strict format (for example, “Answer: … | Source line: …”) and ask 3 new questions. See how examples lock the format.
- 5Write one line: which problem did each step fix, knowledge or behaviour?
Check yourself
Pick an answer to see if you are right, and why.
Done? Keep going
Mark this lesson complete to keep your place. Progress is saved on this device, no login.
Questions people ask
Is RAG better than fine-tuning?
They solve different problems. RAG gives a model knowledge it lacks, like private or changing facts, and can show sources. Fine-tuning changes behaviour, like format or tone. Most teams should try prompting first, then RAG, and fine-tune last.
Can fine-tuning add new knowledge to an LLM?
It is a poor fit for that. Vendor guides present fine-tuning mainly for format, style and task behaviour, and every fact update would mean retraining. For new or changing facts, put the information in the prompt or use RAG.
How much data do you need to fine-tune an LLM?
OpenAI suggests starting with 50 or more high-quality examples, and Google’s guide says to think of 100 or more, often hundreds to thousands for supervised tuning. Quality matters more than quantity.
Sources and further reading
- OpenAI: Optimizing LLM accuracy
- OpenAI: Model optimization (evals, prompting, fine-tuning)
- Anthropic: Introducing Contextual Retrieval
- Anthropic: Fine-tune Claude 3 Haiku in Amazon Bedrock
- Anthropic: Prompt engineering overview
- Google Cloud: Introduction to tuning (Gemini)
- Google Cloud: What is retrieval-augmented generation?
- IBM Technology video transcript: RAG vs Fine-Tuning vs Prompt Engineering
- Lewis et al. (2020): Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
- Hu et al. (2021): LoRA: Low-Rank Adaptation of Large Language Models