Different kinds of thing

You cannot replace RAG with MCP any more than you can replace caching with HTTP. RAG describes what happens: relevant text is found and put in front of the model before it answers. MCP describes how a model reaches a system: a standard way for a host to discover a server's tools and resources and call them. One is a pattern, the other is a wire. Put them in the same sentence and the honest version is "RAG over MCP", not "RAG or MCP".

The question is still worth answering, because behind it sits a real fork. Retrieval can run in two shapes. In the first, your code searches before every model call and pastes the results into the prompt - the classic pipeline. In the second, search is a tool the model can call during its turn, and the model decides whether, when, and how often to use it. MCP is the most common way to hand a model that tool, which is how the protocol ended up in the argument. The rest of this article compares the two shapes and says where the wire matters.

Shape one: retrieval as a step

The pipeline is RAG as most teams first built it. A question arrives; your code embeds it, queries the index, takes the top k passages, builds a prompt, and makes exactly one model call. The model never knows a search happened. From its point of view the facts were simply in the prompt.

What you get for that rigidity is control. You know precisely what the model read, so you can cite it. Retrieval is a plain function with an input and an output, so you can test it on its own: for a set of known questions, did the right passage come back in the top k? Cost and latency are the same on every question - one embedding, one index query, one generation - which is what a budget wants. And the whole thing works with any model, including ones that cannot call tools at all.

What you give up is judgement. The pipeline searches when there is nothing to find and searches once when the answer needed two hops. A question that needs a policy passage and a customer record gets one blended query, and the passages that come back match neither well. The pipeline is only as good as the query your code writes, and your code has never read the question.

Shape two: retrieval as a tool

Give the model a function called search_docs(query, k) with a description of what it searches, and retrieval moves inside the model's turn. The model reads the question, decides a search is needed, writes the query itself, reads the results, and either answers or searches again with a sharper query. This is the agent loop with retrieval as one of the tools in it.

MCP enters here. The search function has to be exposed to whatever is running the model, and if that is a host you do not own - Claude Desktop, an IDE, a colleague's agent - you cannot edit its prompt, so a pipeline is not even available to you. An MCP server that publishes search_docs as a tool is the door every one of those hosts can open. The protocol's own primitives map onto the two shapes cleanly: a tool is model-controlled, so a search tool is shape two; a resource is application-controlled, so a host that pulls a resource into the prompt before the model speaks is running shape one over the same wire. MCP does not pick a side. It carries both.

The tool shape buys adaptivity and pays in predictability. The model can chain searches, reformulate, and stop early when the first hit is enough. It can also decide not to search and answer from memory, or search six times on a question worth one, and every search is another model turn on the bill and the clock. What it read is in the trace, but what it used is not - unless you instruct it to cite, the citation is now the model's claim rather than your code's record.

The same search, both ways

Both shapes can share one retrieval function - drop the retrieve() from Inside a RAG pipeline into search.py. The pipeline calls it before the model; the tool version hands it to the model. Install openai, mcp, and the voyageai package the retriever embeds with, set OPENAI_API_KEY and VOYAGE_API_KEY, and the two files below are complete.

PYTHON pipeline.py · shape one
from openai import OpenAI from search import retrieve # the retrieve() from Inside a RAG pipeline client = OpenAI() def answer(question): context = "\n".join(retrieve(question, k=4)) # your code decides: always search, always four passages response = client.responses.create( model="gpt-5.6-luna", instructions="Answer only from the context. If the context does not contain the answer, say so.", input=f"Context:\n{context}\n\nQuestion: {question}", ) return response.output_text # exactly one model call per question, every time
PYTHON search_server.py · shape two
from mcp.server.fastmcp import FastMCP from search import retrieve # the same function, now behind a tool mcp = FastMCP("docs") @mcp.tool() def search_docs(query: str, k: int = 4) -> list[str]: """Search the company documentation. Returns the k passages most relevant to the query.""" return retrieve(query, min(k, 8)) # the model picks the query and k; you set the ceiling if __name__ == "__main__": mcp.run() # stdio transport; any MCP host connects, lists search_docs, and calls it when it decides to

Read the two files as a diff and the comparison writes itself. The pipeline has a query string chosen by your code (the raw question) and a k chosen by your code; the server has neither - both are now parameters the model fills in, and your only remaining lever is the ceiling. The pipeline's instruction, "answer only from the context", lives in your prompt; in the tool version the docstring is your instruction, because the description is the only thing the model reads before deciding whether to call. Write it as carefully as you would write the prompt.

Side by side

The two shapes on the axes that actually decide between them:

Axis Pipeline (retrieval as a step) Tool (retrieval over MCP)
Who decides to search Your code, before every call The model, when it judges it needs to
Searches per question Exactly one, fixed k Zero to many; query and k chosen by the model
Cost and latency Constant: one retrieval, one generation Variable: every search adds a model turn
Freshness As fresh as the last re-index Live if the tool queries the source; index-fresh if it wraps one
Evaluating retrieval In isolation: did the right passage come back? Inside the run: read the trace to see what it searched
Citations You know exactly what the model read You know what it fetched, not what it used, unless it cites
Access control A filter on the query, in your code The server's permissions, per user and per tool
Hosts you do not own Unavailable: you have to own the prompt Any MCP host can list the tool and call it
Typical failure A silent retrieval miss; the model answers from memory The model never searches, or searches far too often

Where they combine

Most production systems do not choose. Three combinations come up again and again:

  • The index behind the tool. Everything from the RAG article - chunking, embeddings, the vector store, hybrid keyword search - still has to exist. The MCP server is a thin door in front of it. "RAG or MCP" dissolves here: the pipeline's hard part is the index, and the tool shape reuses it unchanged.
  • Pre-fetch, then let the model dig. Your code runs one pipeline retrieval on the raw question so the first model turn already has the obvious passages, and the same search is also available as a tool for follow-ups. Cheap questions finish in one turn; hard ones get the loop.
  • Resources for the known, tools for the unknown. Context you can name in advance - the customer's account, the open ticket, the current file - goes in as an MCP resource the host attaches before the model speaks. Context that depends on what the question turns out to be goes behind a tool.

The common thread is that the split is by who knows what to fetch. When your code knows, fetch it up front and spend no model turns on it. When only the model can tell after reading the question, hand it the tool.

Which to pick

If you remember one rule: pick by who is in a position to write the query.

  • Pipeline when every question needs the same kind of context, when cost and latency have to be predictable, when you need to evaluate retrieval separately from generation, or when the model in front of you does not call tools. A support bot over one policy corpus is a pipeline.
  • Tool over MCP when questions vary in what they need, span several sources, or take more than one hop; when the consumer is a host you do not control; or when several hosts should share one retrieval capability. An internal assistant that answers from the wiki, the ticket tracker, and the data warehouse is a set of tools.
  • Both when you have a good index already and the agent is the new consumer. Put the pipeline's search behind a server and keep the pipeline for the surfaces you own.

And the answer to the question this started with - "do we still need RAG now that we have MCP" - is that MCP changed who calls the search and how it is plugged in. It did not change what the search is made of. The index still has to be built, kept fresh, and tested for misses, whichever side of the wire triggers it.

References