RAG Is a Search Problem Wearing an AI Costume

A model only knows what it was trained on. Ask it about your company's refund policy, last week's incident, or a private contract, and it has two options: refuse, or invent something that sounds right. Neither is what you shipped it for.

You learned to talk to it cleanly in the last post, but no prompt teaches a model a fact it never saw. This rung fixes the knowledge, not the phrasing.

RAGRAGRetrieval Augmented Generation: fetch relevant text at question time and put it in the context, instead of retraining., retrieval-augmented generation, is the fix. You find the relevant text yourself, paste it into the prompt, and let the model answer from it. That is the whole idea. The model stops guessing from memory and starts reading from your knowledge baseknowledge baseThe corpus your retrieval searches. Its quality sets the ceiling on every answer., the store of documents you actually trust. This move is called context retrievalcontext retrievalThe whole job of getting the right passages in front of the model before it answers.: fetch first, then generate.

Here is the claim this whole post defends. RAG is a search problem wearing an AI costume. It succeeds and fails as search does, not as intelligence does. Get that backwards and you will spend weeks tuning prompts to fix a problem that lives in your retriever.

Retrieval Beats Retraining

The obvious objection: why not just train the model on your data? Bake the knowledge in with fine-tuningfine-tuningFurther training on a narrower dataset to change behaviour. The alternative to retrieval when you need new behaviour, not new facts. and skip the retrieval plumbing entirely.

Because fine-tuning teaches behaviour, not facts. It is good at shaping tone, format, and style. It is bad at memorising specific documents you can recite back. Worse, your facts change. Prices move, policies update, tickets close. Retraining every time a wiki page edits is absurd. Retrieval reads the current version on every request.

Retrieval wins on three counts that matter in production. It is fresh, because it reads live data. It is cheap, because indexing a document costs far less than a training run. And it is honest, because you can show the exact passage an answer came from. Fine-tuning gives you none of that. Reach for it when you need new behaviour. Reach for RAG when you need new knowledge.

Everything Upstream of the Model Is Search

Now the machinery. A document does not go into a knowledge base whole. You cut it into pieces first, a step called chunkingchunkingSplitting documents into passages small enough to retrieve precisely and large enough to still make sense.. Chunk too large and you bury the one relevant sentence inside noise the model has to wade through. Chunk too small and you slice a single idea across two pieces, so neither one answers the question alone. The trade-off is real and there is no universal right size. You tune it against your own documents.

Each chunk is then turned into a vector, a list of numbers that encodes its meaning, by an embedding modelembedding modelThe model that turns text into those vectors. Its choice sets what your retrieval can find.. This is the same trick that powers how a model reads text in the first place, covered in what an LLM actually is. Close meanings produce close vectors. Those vectors live in a vector storevector storeA store built to find the nearest embeddings to a query embedding quickly., also called a vector databasevector databaseThe same idea as a product: storage, indexing and filtering for embeddings at scale., a store built for one job: given a query vector, find the nearest chunk vectors fast.

Finding the nearest ones is similarity searchsimilarity searchFinding the passages whose embeddings sit closest to the question embedding.. The query becomes a vector too, and the store returns the chunks whose vectors sit closest to it. Closest by what measure? Usually cosine similaritycosine similarityThe usual way to measure how close two embeddings are, by the angle between them rather than their length., which scores two vectors by the angle between them rather than their length. Small angle, similar meaning. That score is the entire basis on which your model decides what to read.

Look at that diagram and count how many stages happen before the model does anything. Chunk, embed, store, search, rank, filter. Every one is a search decision. The model is the last box, not the pipeline.

The Part Everyone Skips

Most RAG tutorials stop at similarity search. Embed, store, retrieve top-k, done. Then it works in the demo and falls apart on real questions, and nobody knows why. The why is almost always here, in the stages people skip.

Hybrid searchHybrid searchCombining keyword matching with semantic similarity, because each finds what the other misses. is the first one. Pure vector search is great at meaning and terrible at exact strings. Ask for error code SIG-4021 or a product SKU and the embedding blurs it into similar-looking codes. So you run vectors alongside BM25BM25The classic keyword ranking function. Strong on exact terms, names and codes, where embeddings are weak., the classic keyword-ranking algorithm that has powered search engines for decades. Vectors catch meaning, BM25 catches exact terms, and together they beat either one alone.

RerankingRerankingA second, more expensive pass that reorders the first set of results by real relevance. is the second. Your first-pass search is fast but rough, so it returns maybe fifty plausible chunks. A rerankerrerankerThe model that does that second pass, usually scoring each passage against the query directly. is a slower, sharper model that reads the query against each candidate and rescores them properly, so the genuinely best passages rise to the top. First pass for recall, rerank for precision. Feeding the model raw top-k without this is the single most common reason RAG answers feel almost right but not quite.

Metadata filteringMetadata filteringNarrowing by source, date or permission before similarity runs, so the search never sees what it must not. is the third, and it runs before similarity even starts. Every chunk carries tags: source, date, author, access level. Filter on them first and you never retrieve a document this user is not allowed to see, or a policy that expired last year. This is not an optimisation. It is how you keep RAG from leaking data across permission boundaries.

Last, fix the query itself. Users write lazy, ambiguous questions. Query rewritingQuery rewritingReshaping the user question into something a retriever can actually match. reshapes a vague question into one that retrieves well, and query expansionquery expansionSearching several phrasings of one question to catch documents that use different words. fires several phrasings at once to widen the net. A good rewrite step recovers answers a literal search would have missed entirely.

Stack these and you no longer have a lookup. You have a retrieval pipelineretrieval pipelineThe full path from question to passages: rewrite, filter, search, rerank, assemble.: rewrite, filter, hybrid search, rerank, then hand the survivors to the model.

You Cannot Fix What You Cannot Measure

Here is where the search framing pays off. When a RAG system gives a bad answer, the instinct is to blame the model and rewrite the prompt. Usually the model was fine. It answered faithfully from context that never contained the answer, because retrieval missed it.

So measure retrieval on its own, before the model runs. Build a set of real questions with the chunks that should answer each one. Then ask two search questions of your pipeline. Did the right chunk come back at all? That is recall. Did it come back near the top, where the model will actually weight it? That is precision at k. Track those numbers and every improvement, better chunking, hybrid search, a reranker, becomes a measurable move instead of a guess.

If the right chunk never makes the shortlist, no prompt on earth saves the answer. Fix the search.

That is the whole point. A model can only answer from what it was trained on or what you put in front of it. RAG is how you control the second half. Treat it as an AI problem and you will tune the one box that was already working. Treat it as the search problem it is, and it starts answering from your knowledge instead of guessing at it.

So far the model can read your world. Next it gets to act on it, which is where a chatbot turns into something that does the work.

Check what stuck

1Your model needs to answer from your company documents. Why is fine-tuning usually the wrong tool?

Fine-tuning is too expensive to considerFine-tuning shapes behaviour and style, it does not reliably install specific facts you can citeFine-tuned models cannot use toolsIt only works on small models

Check answer

2Semantic search keeps missing an exact product code your users search for. Why?

The embedding model is brokenEmbeddings capture meaning, and an exact rare string has little meaning to match on, which is what keyword search is good atThe code is too short to embedVector stores cannot index numbers

Check answer

3What goes wrong when chunks are too large?

Nothing, larger chunks always carry more contextRetrieval gets imprecise, because a chunk matches on one relevant line and drags in a page of noise that fills the windowThe vector store refuses themEmbeddings cannot be computed

Check answer