RAG Is a Search Problem Wearing an AI Costume

A model only knows what it was trained on. Ask it about your company's refund policy, last week's incident, or a private contract, and it has two options: refuse, or invent something that sounds right. Neither is what you shipped it for.

You learned to talk to it cleanly in the last post, but no prompt teaches a model a fact it never saw. This rung fixes the knowledge, not the phrasing.

RAG, retrieval-augmented generation, is the fix. You find the relevant text yourself, paste it into the prompt, and let the model answer from it. That is the whole idea. The model stops guessing from memory and starts reading from your knowledge base, the store of documents you actually trust. This move is called context retrieval: fetch first, then generate.

Here is the claim this whole post defends. RAG is a search problem wearing an AI costume. It succeeds and fails as search does, not as intelligence does. Get that backwards and you will spend weeks tuning prompts to fix a problem that lives in your retriever.

Retrieval Beats Retraining

The obvious objection: why not just train the model on your data? Bake the knowledge in with fine-tuning and skip the retrieval plumbing entirely.

Because fine-tuning teaches behaviour, not facts. It is good at shaping tone, format, and style. It is bad at memorising specific documents you can recite back. Worse, your facts change. Prices move, policies update, tickets close. Retraining every time a wiki page edits is absurd. Retrieval reads the current version on every request.

Retrieval wins on three counts that matter in production. It is fresh, because it reads live data. It is cheap, because indexing a document costs far less than a training run. And it is honest, because you can show the exact passage an answer came from. Fine-tuning gives you none of that. Reach for it when you need new behaviour. Reach for RAG when you need new knowledge.

Everything Upstream of the Model Is Search

Now the machinery. A document does not go into a knowledge base whole. You cut it into pieces first, a step called chunking. Chunk too large and you bury the one relevant sentence inside noise the model has to wade through. Chunk too small and you slice a single idea across two pieces, so neither one answers the question alone. The trade-off is real and there is no universal right size. You tune it against your own documents.

Each chunk is then turned into a vector, a list of numbers that encodes its meaning, by an embedding model. This is the same trick that powers how a model reads text in the first place, covered in what an LLM actually is. Close meanings produce close vectors. Those vectors live in a vector store, also called a vector database, a store built for one job: given a query vector, find the nearest chunk vectors fast.

Finding the nearest ones is similarity search. The query becomes a vector too, and the store returns the chunks whose vectors sit closest to it. Closest by what measure? Usually cosine similarity, which scores two vectors by the angle between them rather than their length. Small angle, similar meaning. That score is the entire basis on which your model decides what to read.

Look at that diagram and count how many stages happen before the model does anything. Chunk, embed, store, search, rank, filter. Every one is a search decision. The model is the last box, not the pipeline.

The Part Everyone Skips

Most RAG tutorials stop at similarity search. Embed, store, retrieve top-k, done. Then it works in the demo and falls apart on real questions, and nobody knows why. The why is almost always here, in the stages people skip.

Hybrid search is the first one. Pure vector search is great at meaning and terrible at exact strings. Ask for error code SIG-4021 or a product SKU and the embedding blurs it into similar-looking codes. So you run vectors alongside BM25, the classic keyword-ranking algorithm that has powered search engines for decades. Vectors catch meaning, BM25 catches exact terms, and together they beat either one alone.

Reranking is the second. Your first-pass search is fast but rough, so it returns maybe fifty plausible chunks. A reranker is a slower, sharper model that reads the query against each candidate and rescores them properly, so the genuinely best passages rise to the top. First pass for recall, rerank for precision. Feeding the model raw top-k without this is the single most common reason RAG answers feel almost right but not quite.

Metadata filtering is the third, and it runs before similarity even starts. Every chunk carries tags: source, date, author, access level. Filter on them first and you never retrieve a document this user is not allowed to see, or a policy that expired last year. This is not an optimisation. It is how you keep RAG from leaking data across permission boundaries.

Last, fix the query itself. Users write lazy, ambiguous questions. Query rewriting reshapes a vague question into one that retrieves well, and query expansion fires several phrasings at once to widen the net. A good rewrite step recovers answers a literal search would have missed entirely.

Stack these and you no longer have a lookup. You have a retrieval pipeline: rewrite, filter, hybrid search, rerank, then hand the survivors to the model.

You Cannot Fix What You Cannot Measure

Here is where the search framing pays off. When a RAG system gives a bad answer, the instinct is to blame the model and rewrite the prompt. Usually the model was fine. It answered faithfully from context that never contained the answer, because retrieval missed it.

So measure retrieval on its own, before the model runs. Build a set of real questions with the chunks that should answer each one. Then ask two search questions of your pipeline. Did the right chunk come back at all? That is recall. Did it come back near the top, where the model will actually weight it? That is precision at k. Track those numbers and every improvement, better chunking, hybrid search, a reranker, becomes a measurable move instead of a guess.

If the right chunk never makes the shortlist, no prompt on earth saves the answer. Fix the search.

That is the whole point. A model can only answer from what it was trained on or what you put in front of it. RAG is how you control the second half. Treat it as an AI problem and you will tune the one box that was already working. Treat it as the search problem it is, and it starts answering from your knowledge instead of guessing at it.

Next, we give the model the power to act, not just read: giving models tools. Back up one level to talking to models, or see the whole map in the applied AI engineering ladder.