AI
Retrieval-Augmented Generation in Production: Architecture Beyond the Demo
A weekend RAG demo is trivial. A retrieval system that stays accurate, fast, and grounded under real traffic is not. We cover chunking, hybrid retrieval, reranking, evaluation, and the freshness loop most teams skip.
Retrieval-augmented generation demonstrates well and productionises badly. The demo works because the corpus is small, clean and homogeneous, and because the questions asked of it are the questions it was built to answer. Every property that changes at enterprise scale — corpus size, document heterogeneity, access control, freshness — attacks retrieval quality specifically, and generation quality is bounded by it.
Retrieval is the system. Generation is the easy half.
If the retrieved context does not contain the answer, no model produces it — the best case is a refusal and the usual case is a confident synthesis from adjacent material. Almost every RAG failure we diagnose is a retrieval failure being attributed to the model, and the diagnostic is simple enough that it should be the first step: check whether the correct passage was in the context window at all.
Recall@k = | Rk ∩ G | / | G | nDCG@k = DCG@k / IDCG@k
Instrument these separately from end-to-end answer quality, and build the labelled question-to-passage set that makes them measurable. It is tedious work — a few hundred questions with their supporting passages identified — and it is the difference between tuning a pipeline and guessing at one.
Chunking: respect document structure, not a token count
Fixed-size chunking with overlap is the default and it is wrong for most enterprise material. It severs tables from their headers, splits procedures mid-step, and separates clauses from the definitions that give them meaning. A chunk that begins "...and in that case the limit shall not apply" is retrievable and useless.
- Chunk on structural boundaries — sections, clauses, table rows, function definitions — using the document's own markup rather than a character count.
- Prepend inherited context to each chunk: document title, section path, effective date. A clause retrieved without knowing which contract it came from cannot be reasoned about.
- Keep tables intact, with their headers, and serialise them in a form the model reads reliably. A table split across three chunks is three chunks of unlabelled numbers.
- Index a summary alongside the chunk for long passages, so semantic search matches on topic while the model receives the full text.
Hybrid retrieval, because dense search fails in specific and predictable ways
Dense embeddings capture semantic similarity, which is precisely what you want for paraphrase and concept matching, and precisely what fails on exact identifiers. A part number, an error code, a statute reference or a customer ID has no useful semantics — it has an exact form, and vector search will happily return a near neighbour that is a different part.
Lexical search handles those perfectly. Running both and fusing the rankings recovers the strengths of each, and reciprocal rank fusion does it without requiring the two scoring systems to be calibrated against each other.
RRF(d) = ∑i ∈ retrievers 1 / ( k + ranki(d) )
Above fusion, a cross-encoder reranker gives the largest single quality gain available in most pipelines. Bi-encoders embed query and document independently, so they never see the pair together; a cross-encoder scores the pair jointly and is substantially more accurate. It is far too slow to run over the corpus, which is exactly why the architecture is retrieve broadly with cheap methods, then rerank the top 50–100 precisely.
Index parameters determine your latency and your recall
HNSW is the default index for good reason: it builds a navigable small-world graph with a hierarchy of layers, giving approximately logarithmic search complexity. Its parameters are a direct recall-versus-cost dial, and leaving them at defaults is leaving one of the two on the table.
| Parameter | Raising it | Cost |
|---|---|---|
| M (graph degree) | Better recall, denser graph | Memory, linear in M |
| efConstruction | Better graph quality | Build time only |
| efSearch | Better recall at query time | Query latency, directly |
Memory is usually the binding constraint at scale. Ten million chunks at 1024 dimensions in float32 is roughly 41 GB of raw vectors before graph overhead. Product quantisation compresses this by an order of magnitude at a recall cost, and the usual resolution is a two-stage search: quantised index for the broad pass, full-precision rescoring of the survivors.
Access control cannot be a post-filter
This is the failure mode with genuine consequences, and it is architecturally easy to get wrong. Enterprise corpora contain material that specific users must not see — HR records, unreleased financials, legal privilege. Retrieving without regard to permissions and filtering afterwards produces two problems: the correct top-k is silently degraded for restricted users, and any leak in the filtering path is a direct exposure of confidential material.
Permissions must be applied during search, as a filter on the index, evaluated against the requesting user's live entitlements. That has real implications: the index carries per-chunk ACLs, they are updated when source permissions change, and the search path is authenticated end to end. It is more work than post-filtering. It is also the difference between a system that can hold sensitive material and one that cannot.
Freshness, and the answer that was true last quarter
A rebuild-nightly index is a system that confidently cites a superseded policy all day. Incremental indexing on document change is the requirement, and it needs deletion to work correctly — a superseded document whose chunks remain in the index is worse than one that was never indexed, because it is retrievable and authoritative-looking.
Where documents have versions and effective dates, put them in the metadata and filter on them at query time. "What is the current travel policy" and "what was the travel policy in March" are different queries against the same corpus, and a system that cannot distinguish them will answer one of them wrongly.
The evaluation set is the asset
Everything above is a tuning decision, and tuning without measurement is superstition. The durable asset in a RAG programme is not the pipeline — components get replaced as models improve — it is the evaluation set: questions, their supporting passages, and acceptable answers, curated from real user queries and maintained as the corpus evolves. It is what lets you swap an embedding model in an afternoon and know whether you improved anything, and it is the reason some RAG systems get better over two years while others are rebuilt every six months.
Explore how Root Digit can support your team
From discovery workshops to production deployment, our engineers and consultants partner with you across the lifecycle of your AI, robotics, and IoT initiatives.