What Is RAG in AI? Retrieval-Augmented Generation Explained

RAG (retrieval-augmented generation) is a technique where an AI system first searches a collection of documents for passages relevant to your question, then adds those passages to the prompt so the language model answers from them. It lets a chatbot answer questions about your own, private or recent information without retraining the model.

What problem RAG solves

A language model only knows what was in its training data, up to its training cutoff. It has never seen your company's HR policy, your product manual or yesterday's support tickets, and when asked about them it may guess. The 2020 paper that named the technique, by Patrick Lewis and colleagues, framed it as giving a model two kinds of memory: the knowledge stored in its parameters, and an external index it can look things up in.

Anthropic describes RAG in one sentence: a method that "retrieves relevant information from a knowledge base and appends it to the user's prompt". The model then answers with the right material in front of it, as if it were an open-book exam.

How RAG works, step by step

A RAG system has two phases: preparing the documents once (and whenever they change), and answering each question.

Preparation (indexing)

1. Collect the source documents: PDFs, wiki pages, tickets, database rows.
2. Chunk them into passages. Anthropic notes chunks are "usually no more than a few hundred tokens". Small enough to be specific, large enough to make sense alone.
3. Embed each chunk: run it through an embedding model, which turns it into a vector of numbers that represents its meaning (see what are embeddings).
4. Store the vectors with the chunk text in a vector index, along with metadata such as source, date and access permissions.

Answering a question

1. Embed the question with the same embedding model.
2. Retrieve the chunks whose vectors are closest to the question's, typically the top 5 to 20.
3. Augment the prompt: insert those chunks, with instructions such as "Answer using only the context below and cite the source."
4. Generate: the language model writes the answer from the supplied text.

A simplified version of the final prompt the model receives:

Answer the question using only the context. If the context doesn't contain
the answer, say so. Cite the document name for each fact.

Context:
[hr-handbook.pdf, p.12] Employees hired after 1 January 2025 accrue 2 days
of paid leave per month, starting from their first full month.
[remote-policy.md] Remote employees follow the same leave rules as office staff.

Question: How much paid leave do remote workers hired in 2026 get?

The user only typed the last line. Everything above it was added by the retrieval step.

Why retrieval is the hard part

The model can only be as good as the passages it's given. If retrieval returns the wrong chunks, the answer is wrong or empty however capable the model is. Microsoft's Azure AI Search documentation lists the practical challenges: users phrase questions differently from how documents are written (its example is someone asking about "PTO policy for remote workers" when the documents say "time off" and "telecommute"), content is spread across many systems, the model's context window limits how much can be passed, answers need to arrive within seconds, and users must only see documents they're allowed to see.

The common fixes:

• Hybrid search. Combine vector search (good at meaning) with keyword search such as BM25 (good at exact terms like product codes and error numbers). Both Microsoft and Anthropic recommend the combination.
• Reranking. Retrieve a larger candidate set, then use a second model to re-score how well each one answers the question, and keep the best few.
• Better chunks. A chunk that says "Revenue grew 3% over the previous quarter" is useless without knowing which company and quarter. Anthropic's "contextual retrieval" adds a short, generated description of where each chunk sits in its document before embedding it.

Anthropic published measurements for these techniques on its own test sets: contextual embeddings reduced the top-20 retrieval failure rate by 35%, adding contextual BM25 keyword search brought the reduction to 49%, and adding reranking brought it to 67%, cutting the failure rate from 5.7% to 1.9%. Those are its results on its datasets, not guarantees, but they show how much retrieval quality varies with design choices.

When you don't need RAG

If all your material fits in the model's context window, you can skip retrieval and paste the whole thing into the prompt. Anthropic's guidance is that for a knowledge base smaller than 200,000 tokens, about 500 pages, including all of it in the prompt can work; prompt caching makes repeating it cheaper. That's simpler and avoids retrieval failures entirely. RAG becomes necessary when the collection is too large for the context window, when including everything would cost too much per question, or when different users may only see different documents.

To estimate how many tokens your documents are, paste a sample into a token counter such as larely's token counter. What is a context window covers how much current models can take.

RAG vs fine-tuning vs long context

They combine: a fine-tuned model can still use RAG, and a RAG system can retrieve whole documents into a long context window instead of small chunks.

Where RAG still goes wrong

• Nothing relevant retrieved. The model may answer from its own memory anyway. Tell it explicitly to say when the context doesn't contain the answer.
• Outdated or conflicting documents. Retrieval finds text by similarity, not by truth. An old policy and its replacement look almost identical to a vector search. Remove superseded documents or store dates and filter on them.
• Answer split across many documents. Questions like "which customers complained about billing last quarter?" need aggregation, which top-k retrieval of a few chunks can't do. A database query is the right tool there.
• Permission leaks. If the index ignores access rights, the chatbot can quote documents the user shouldn't see. Filter by permission at retrieval time, not after generation.

RAG reduces made-up answers but doesn't eliminate them; the model can still misread or misattribute what it was given. Why AI hallucinates and how to reduce it covers the other techniques that pair well with it.

Frequently asked questions

Is RAG the same as fine-tuning?

No. Fine-tuning changes a model's weights by training it on more examples, which is slow to repeat and doesn't reliably add facts. RAG leaves the model unchanged and supplies relevant text in the prompt at question time, so updating knowledge means updating the documents. Fine-tuning suits style and format; RAG suits facts that change.

Does RAG stop AI hallucinations?

It reduces them for questions your documents answer, because the model works from supplied text instead of memory. It doesn't eliminate them: the model can still misread a passage, combine passages wrongly, or answer from memory when retrieval finds nothing useful. Instructing it to say when the context lacks an answer, and to cite sources, helps.

Do I need a vector database for RAG?

Not necessarily. Small collections can be searched in memory, and many existing databases and search engines now support vector search. A dedicated vector database helps once you have a large number of chunks and need fast approximate search, filtering and updates.

What does the original RAG paper say?

The 2020 paper by Lewis and colleagues combined a pre-trained sequence-to-sequence model (the model's built-in, parametric memory) with a dense vector index of Wikipedia accessed by a neural retriever (an external, non-parametric memory). The term RAG comes from that paper; today it is used more loosely for any system that retrieves text and adds it to a prompt.

Sources

• Lewis et al. (2020), Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv:2005.11401)
• Microsoft Learn: RAG and generative AI in Azure AI Search
• Anthropic: Introducing Contextual Retrieval
• OpenAI API docs: Vector embeddings