What is RAG?

The problem: an 800-page document

Imagine you have an 800-page financial document and one question about it: "What risk factors does this company have?" The answer is in there. The problem is getting the relevant part of the document to the model, because a model can only read a limited amount of text per answer - its context window.

The obvious move is to paste the whole document into the prompt above the question. Written out, that prompt is three parts - an instruction, the document, the question - with a tag around each so the model can tell them apart:

It is one line of code to build and it fails four ways at once:

• It may not fit. Prompts have a hard length limit, and 800 pages can be past it.
• The model reads less carefully as the prompt grows. The right paragraph is buried among 799 pages of noise.
• Every question pays for every page. Tokens cost money, and all of them are sent again each time.
• Every answer waits for every page. More text in means more time before the first word out.

Underneath is a deeper problem. A large language model answers from what it learned in training, which never includes your documents or anything written after its cutoff. Ask anyway and it does not say "I do not know" - it produces fluent, plausible text, which is what hallucination is. Retraining it on your documents is slow, expensive, and stale the moment a document changes.

The fix: look it up first

Retrieval-augmented generation (RAG) leaves the model alone and changes what it reads. Break the document into pieces ahead of time, find the piece that answers the question - here, the "Risk Factors" section - and put only that in the prompt.

The name is the recipe, three steps on every question:

• Retrieve - search your documents for the few passages most likely to hold the answer. A few pages, not 800.
• Augment - build the prompt: your instructions, those passages, and the question. Tell the model to answer only from the passages, and to say so when they do not contain the answer.
• Generate - the model writes the answer from what is in front of it, and can point to the passage each claim came from.

The augment step is the part you write, and it is the option 1 prompt with one slot swapped - the 800 pages replaced by the passages the search returned:

Same shape, one slot different. The second instruction is new, and it earns its place: only from the passages puts a boundary on where facts may come from, and say so gives the model somewhere to go when the search returns nothing useful, instead of inventing something. The chunk numbers are what make a citation possible later.

The pieces have a name: chunks - each a few hundred words, cut once, ahead of any question. How the search picks the matching ones out of thousands, and how the vector database it searches is kept current, is the machinery of the level below this one. Here it only has to come back with a few good passages.

Notice what did not change: the model. It is doing what it always does - continuing text - but the text now contains the facts. A risk-factors section sitting in the prompt beats a risk-factors section the model never read. And the four failures go with it: three passages fit where 800 pages did not, the model's attention has nowhere irrelevant to wander, and both the bill and the wait now scale with the passages you sent instead of the document you own.

What RAG costs you

Prompt stuffing has exactly one virtue: there is nothing to build. RAG gives that up, and what replaces it is machinery you own. Four parts of the bill:

• A preprocessing step. Nothing can be asked until the documents have been split and indexed, and they have to be re-indexed whenever they change. That is a job to run, schedule, and watch.
• A search mechanism. "Relevant" stops being a figure of speech and becomes code - something that ranks passages, that you tune, and that can be wrong while looking fine.
• Chunks that arrive without their context. A retrieved passage can be the right passage and still not answer: a table of figures cut away from the header row that names the year, a clause that says "the Company" where the definition sat 40 pages earlier. The search did its job; the chunk just does not carry enough to be read on its own.
• A decision about where to cut. Six tidy sections is the friendly version. Real documents do not divide themselves that neatly, and where the lines fall decides which answers are possible at all - which is why the level below this one gives chunking a section of its own.

None of that work exists if the document fits in the prompt. That is the trade: RAG buys scale and speed with complexity. It pays for 800 pages, for a thousand documents instead of one, and for a collection that keeps changing - and below that it is machinery in search of a problem.

What to remember

The model did not learn anything. It read something, right before it answered, because a search put it there. So when RAG is wrong, the search usually missed - the answer was in the documents but not among the passages that came back, or the index was a stale copy of them - and the model filled the gap from memory. Make the search good and the answers follow.

And when the collection is small enough to fit in the prompt, skip the search and paste it: a 20-page policy needs no index. RAG is for collections that are large, change often, or must show their sources.

Everything this article skipped - how the chunks are cut and how to keep a chunk from losing its context, how "nearest" is computed, why real systems add keyword search beside it, and how to tune it when it misses - is the next level down: Inside a RAG pipeline. Who triggers the search, your code before every call or the model through a tool, is RAG vs MCP: pipeline or tool?