Hybrid Retrieval: Not Every Question Needs an LLM Call
At Acumen Academy I built a knowledge assistant over a founders knowledge base: course material, guides, and resources that founders in our programmes ask questions about. The stack was Remix and TypeScript, with all the content living in Contentful. This post is a retrospective on the retrieval design, because the retrieval is where all the interesting decisions were.
The standard RAG recipe is well known by now. Chunk your documents, embed the chunks, store the vectors. At query time, embed the question, fetch the top-k nearest chunks, and hand them to an LLM to generate an answer. It works, and it is where we started.
The problem is that the standard recipe charges every question the same price. Embedding call, vector search, generation call, several seconds of latency. That price is fair for a genuinely fuzzy question. It is a bad deal for a lookup.
Most questions are lookups
Once we watched real usage, a pattern was hard to miss: a large share of queries were not really questions at all. They were retrieval requests wearing a question mark. "Where is the financial modelling template?" "What are the dates for the accelerator?" People were using the assistant as a faster search box, which is a compliment, but it meant we were paying LLM prices for search-box queries.
The keyword layer answers first
So we made the retrieval hybrid. Before anything is embedded, the query runs through a keyword layer that matches against document titles, headings, and the metadata we generate from Contentful. When a query resolves confidently at this layer, the assistant returns the document or section directly, with no LLM call at all.
The response to "where is the financial modelling template" is the template, linked, in well under a second. No model was ever going to improve on that answer; it could only make it slower and more expensive.
Only queries that fail to resolve at the keyword layer go on to the embedding search and generation step. The LLM became the escalation path, not the pipeline.
A tree, not a pile of chunks
The second decision was about structure. Naive chunking flattens your content into a pile of fragments, and a fragment retrieved without its context is how you get confidently wrong answers stitched together from three unrelated guides.
Our content already had a hierarchy in Contentful: programmes contain courses, courses contain guides, guides contain sections. We kept that shape. Retrieval works over a tree that mirrors the content model, and the metadata for each node (what it is, what it belongs to, who it is for) is generated from the Contentful entries themselves rather than maintained by hand.
That bought us two things:
- Chunks arrive with their ancestry. When a section is retrieved, the model also knows which guide and programme it came from, which makes answers more grounded and citations actually useful.
- The structure maintains itself. When editors reorganise content in Contentful, webhooks regenerate the affected parts of the tree. Nobody curates an index by hand, so the index never rots.
Pre-warming the head of the distribution
Usage of a knowledge base is never uniform. A small set of documents accounted for a large share of everything retrieved, and that set was stable enough to predict.
So we pre-warm a cache of the most-used documents and their metadata. The queries people ask most often are exactly the ones that respond fastest, which is the opposite of what you get by default, where popular content just means more repeated work.
The lesson I keep reusing
The mental shift that mattered was this: the LLM is not the system. The LLM is the most expensive tier of a retrieval system, and your job is to make sure it only sees the traffic that deserves it. Cheap, deterministic layers go in front. Expensive, clever layers go in the back. Results flow downhill so that tomorrow's identical question costs less than today's.
I ended up applying the same shape to a completely different problem, a WhatsApp fact-checking agent, which I have written about in AdieraBot: The Architecture Behind My WhatsApp Fact-Checking Agent. Different domain, same flywheel.