Ask a plain LLM “what’s our refund policy?” and it will confidently make something up — it was never trained on your company’s docs, and its training data has a cutoff date anyway. Retrieval-Augmented Generation (RAG) fixes this without retraining the model at all: instead, you hand it the relevant paragraphs right before it answers — like giving someone an open book during an exam instead of making them memorize the textbook.
Two phases that are easy to confuse
The single biggest source of confusion for beginners is treating RAG as one pipeline. It’s actually two separate pipelines that run at very different times:
Indexing — done once
- Runs when you add or update documents
- Reads, chunks, and embeds your content
- Writes vectors into a database
- The user is never waiting on this
Querying — done every time
- Runs on every single user question
- Embeds the question, not the documents
- Searches the already-built vector index
- Has to be fast — the user is waiting
Phase 1 — Turning documents into something searchable
Before RAG can answer anything, your knowledge base has to be prepared. This is the slow, offline part — click through the steps below.
Load Docs
You point the system at your source material — PDFs, help-center articles, internal wikis, a codebase. This happens once (and again whenever content changes), not on every question.
Phase 2 — What happens the instant someone asks a question
This is the part that runs in real time, in the few hundred milliseconds before the user sees a response start streaming in.
User Asks
A real question arrives from a user — "What’s our refund policy for annual plans?" — with no guarantee the model was ever trained on this information.
Wait — what is an “embedding,” really?
Think of an embedding as coordinates on a map, except the map has hundreds of dimensions instead of two. The embedding model reads a piece of text and places it somewhere on that map based on its meaning. “How do I cancel my subscription?” and “steps to end my membership” end up as nearby points, even though they don’t share a single word — because the model learned that they mean similar things.
Similarity search is then just asking: which stored points are closest to this new point? That’s the entire trick behind retrieval — no keyword matching required.
200–500 tokens
Typical chunk size
Top 3–8 matches
Chunks retrieved
~20–150ms
Search latency
Where beginners get tripped up
- Chunking too coarsely. A 5,000-token chunk might contain the right answer buried next to four unrelated topics — the model gets distracted.
- No overlap between chunks. If a sentence gets cut exactly in half across two chunks, neither chunk fully makes sense on its own.
- Retrieving but not grounding. Fetching the right chunks doesn’t help if the prompt doesn’t clearly instruct the model to use only that context — otherwise it still drifts back to guessing.
- Stale embeddings. If a document changes but you never re-run indexing, the vector database is confidently searching outdated content.
Where you’ve already seen this
Customer-support bots that answer from your help docs, internal tools that search company wikis, and coding assistants that pull in relevant files before suggesting a fix are all RAG under the hood. Claude Code’s use of the Model Context Protocol (MCP) to pull in project-specific context before responding is the same core idea — retrieve what’s relevant, then generate.