Five decisions in a trench coat

Every RAG system starts the same afternoon. Chunk the documents, embed them, stuff the top-k into the prompt, demo it. It works beautifully, because you asked it the questions you were thinking about while you built it.

Then a real user asks something your chunking strategy has no answer for, quality falls off a cliff, and you discover there is nothing to debug. It is one function. There is no seam to look inside.

The fix is not a better embedding model. It is noticing that "retrieval" was never one decision — it was five, standing on each other's shoulders under a long coat.

Unbutton the coat

• Route — does this question need retrieval at all? Plenty do not.
• Rewrite — turn a conversational question into something a retriever can actually match.
• Retrieve — possibly several ways at once: dense, sparse, plain structured lookup.
• Rerank — cheap and broad first, expensive and precise second.
• Synthesise — answer, with permission to say the context does not contain it.

Separated, each one is independently testable, independently swappable, and — the part that actually saves you — independently blameable.

The cheapest win is not retrieving

A surprising share of production queries need no retrieval whatsoever. "Summarise what we just discussed" does not want a vector search, and running one actively hurts: you inject four loosely-related chunks and the model, being agreeable, works them into the answer. A small classifier at the front deletes an entire category of confident wrong answers.

pythonasync def answer(q: Query) -> Answer:
    route = await router.classify(q)          # direct | retrieve | tool
    if route is Route.DIRECT:
        return await synthesise(q, context=[])

    rewritten = await rewriter.run(q)
    hits = await gather(
        dense.search(rewritten),
        sparse.search(rewritten),
    )
    top = await reranker.rank(rewritten, dedupe(hits))[:8]
    return await synthesise(q, context=top)

Embeddings are bad at names

Dense retrieval understands meaning and is unreliable with exact tokens — error codes, SKUs, surnames. Sparse retrieval is the exact opposite. Running both and merging costs you one extra query and removes a whole genre of embarrassing miss. Best value-per-line change available in most RAG systems.

Measure the parts, not the vibe

A monolith can only be evaluated end to end, which tells you something got worse and nothing about what. With the stages apart you can watch retrieval recall separately from answer quality — so when things degrade you know whether the retriever stopped finding the document or the model stopped using it. Those look identical from outside and have nothing in common as problems.

If you cannot name the stage that regressed, you do not have a pipeline. You have a mood.