Retrieval-Augmented Generation in Production

A retrieval-augmented generation demo takes an afternoon. Load some documents, embed them, drop the top few chunks into a prompt, and the answers look remarkable. The version that survives a thousand real users asking real questions about a real corpus is a different system, and the distance between the two is where most enterprise AI budgets quietly disappear.

This guide is about that distance. It assumes you have already built the demo and are now being asked the harder questions: why does it confidently cite the wrong policy document, why did it answer from a file that employee should not be able to read, why does it take nine seconds, and why did last week's fix make three other answers worse. Everything here is meant to be copied and adapted rather than admired. It is the companion to our evaluation guide, which is what tells you whether any of these changes actually helped.

What this guide covers

1. When RAG is the right architecture, and the three cases where it is not
2. The pipeline, stage by stage, and where quality is actually won
3. Corpus decisions: what to ingest, and what to refuse
4. Chunking that respects document structure
5. Embeddings and the vector store, with the index parameters that matter
6. Hybrid retrieval and reciprocal rank fusion
7. Re-ranking, and when its cost is justified
8. The generation contract: grounding, citations, and refusal
9. Permissions, freshness, and deletion
10. Latency and cost budgets, with a worked example
11. A failure taxonomy mapped to the stage that owns each failure
12. A production readiness checklist

When RAG is right, and when it is not

RAG earns its place when answers must be grounded in a body of knowledge that changes faster than you can retrain, that is too large to fit in a context window, or that must be attributable to a source. Enterprise document question-answering, customer support over a knowledge base, and internal policy lookup are all natural fits.

Three situations where reaching for RAG is a mistake:

• The corpus is small and stable. If the entire knowledge base is a few thousand tokens and rarely changes, put it in the system prompt and cache it. You get better answers, lower latency, and no retrieval infrastructure to operate.
• The question is computational, not informational. "What was revenue last quarter by region" is a database query. Wrapping a SQL question in semantic search over documents produces a system that is worse at both. Route to a query engine.
• The task needs behaviour, not knowledge. If the model needs to adopt a format, tone, or specialised reasoning pattern, that is a prompting or fine-tuning problem. Retrieval adds facts; it does not change how the model thinks.

Most "our RAG is bad" conversations end with the discovery that a third of the traffic was never a retrieval question in the first place. Classify intent before you retrieve.

The pipeline, and where quality is won

A production RAG system has more stages than the demo suggests, and the quality contribution is not evenly distributed:

query
  -> intent classification        (route: retrieve | query engine | refuse | direct)
  -> query transformation         (rewrite, expand, decompose)
  -> retrieval                    (dense + lexical, in parallel)
  -> fusion                       (reciprocal rank fusion)
  -> filtering                    (permissions, recency, source policy)
  -> re-ranking                   (cross-encoder over the fused candidates)
  -> context assembly             (dedupe, order, budget)
  -> generation                   (grounded, with enforced citations)
  -> post-check                   (citation validity, refusal policy)

In our engagements, the largest quality gains almost always come from the least glamorous stages: chunking, hybrid retrieval, and re-ranking. Teams tend to spend their time on prompt wording, which is the stage with the least headroom once the model is actually being handed the right passages. If the correct passage is not in the context, no prompt saves you.

Corpus decisions

Before a single embedding is computed, decide what belongs in the index. The instinct to ingest everything is the most expensive instinct in RAG, because every irrelevant document is a permanent source of plausible distractors.

• Exclude superseded content. Three versions of a policy in the index means the retriever will sometimes surface the 2019 one. If old versions must be kept for audit, keep them out of the index or mark them with a metadata filter applied by default.
• Exclude low-signal formats. Meeting transcripts, email threads, and chat exports produce chunks that retrieve well on vocabulary overlap and answer badly. Ingest them only with a specific question in mind.
• Decide the PII position explicitly. Either PII is out of the corpus, or it is in and the system inherits every access-control and retention obligation that comes with it. There is no third option, and this is a conversation to have with the customer's security lead in week one rather than week six.
• Record provenance per document. Source system, owner, last modified, and sensitivity. Every downstream feature - filtering, freshness, permissions, citations - depends on metadata you can only capture at ingestion.

Chunking that respects structure

Fixed-size character splitting is the default in every tutorial and the wrong default for most real corpora. It cuts tables in half, separates a heading from the paragraph it governs, and strips the context that made a passage interpretable.

What works, in rough order of impact:

• Split on document structure first. Headings, sections, list boundaries. A chunk should be a semantically complete unit: one procedure, one policy clause, one table with its caption.
• Prepend inherited context. Store the document title and heading path at the top of each chunk. A chunk reading "Submit within 30 days" is useless; the same chunk prefixed with "Refund Policy v4 > Partial refunds > Deadlines" is retrievable and interpretable.
• Keep tables intact. Convert to markdown and keep the header row with every fragment if a table must be split at all.
• Overlap modestly. Ten to fifteen percent overlap protects against boundary loss. Large overlaps mostly inflate the index and produce near-duplicate results that crowd out genuine alternatives.

On size: 300 to 800 tokens suits most prose corpora. Smaller chunks retrieve precisely but fragment reasoning across many results; larger chunks carry context but dilute the embedding, since a single vector must represent everything in the chunk. When in doubt, start at roughly 500 tokens with structural splitting and let your eval set decide, one variable at a time.

Chunking is the highest-leverage and least-discussed decision in RAG. A team that fixes chunking usually finds that half their prompt engineering was compensation.

Embeddings and the vector store

Choose an embedding model on your own data, not on a leaderboard: build a small retrieval eval, run two or three candidates, and compare recall@10. Domain vocabulary and language mix matter more than a benchmark average, and for multilingual or Indian-language content the gap between models is often dramatic.

Practical constraints worth knowing before you commit:

• Dimensions cost memory forever. A 3072-dimension vector is four times the index footprint of a 768-dimension one. Verify the larger model actually wins on your eval before paying that indefinitely.
• Changing the model means a full re-index. Budget for it, and keep the ingestion pipeline reproducible so a re-index is a scheduled job rather than a project.
• Store the model version alongside every vector. Mixed-model indexes produce silently wrong similarity, and it is nearly impossible to diagnose after the fact.

For the store itself, pgvector is the right first answer for the majority of engagements. Keeping vectors in Postgres means one database, real transactions, and metadata filters expressed in SQL you already know. Reach for a dedicated vector database when scale, specialised index types, or multi-tenant isolation genuinely demand it - not by default. The relevant HNSW parameters:

m               graph connectivity   (16 typical; higher = better recall, more memory)
ef_construction build-time effort    (64 to 200; higher = better index, slower build)
ef_search       query-time effort    (40 to 200; the runtime recall/latency dial)

ef_search is the knob you tune in production, because it trades latency for recall at query time with no re-index. Measure both before settling on a value.

Hybrid retrieval and fusion

Dense vector search captures meaning and misses exact tokens. Lexical search (BM25) captures exact tokens and misses paraphrase. Real queries need both: product codes, error strings, policy numbers, and surnames are precisely where embeddings fail, and they are disproportionately common in enterprise questions.

Run both retrievers in parallel and merge with reciprocal rank fusion, which combines rankings without needing the two score scales to be comparable:

score(d) = sum over retrievers r of  1 / (k + rank_r(d))

k = 60 is the standard constant and a reasonable default.
A document at rank 1 in one retriever and absent from the other
still outranks a document sitting at rank 8 in both.

This single change is frequently worth more than any amount of prompt tuning, and it is a contained piece of work. Retrieve 20 to 50 candidates from each retriever before fusion; the re-ranker will cut them down.

Re-ranking

Bi-encoders embed the query and document separately, which is what makes search fast. A cross-encoder reads the query and document together and scores relevance directly. It is far more accurate and far too slow to run over a whole corpus - which is exactly why it belongs at the end, over 20 to 50 fused candidates.

Typical effect: precision@5 improves substantially while adding 50 to 200 milliseconds. For most enterprise question-answering that is the best latency-for-quality trade in the pipeline. Skip it when your latency budget is genuinely tight, when the corpus is small enough that recall@5 is already near ceiling, or when a measured eval says it is not helping your traffic. Decide with numbers, not with the architecture diagram.

The generation contract

Retrieval quality sets the ceiling; the generation stage decides how much of it you keep. Four rules do most of the work:

• Instruct grounding explicitly. The model answers from the provided context only. If the context is insufficient, it says so. This instruction is worth little without the eval to enforce it, and worth a great deal with one.
• Make citations structural, not stylistic. Give each context passage an id and require claims to reference those ids. Then validate after generation: every cited id must exist in the context you actually supplied. Hallucinated citations are common and trivially detectable, and detecting them is the cheapest trust win available.
• Order the context deliberately. Models attend unevenly across a long context, with the middle attended least. Put the highest-ranked passages at the edges rather than burying your best evidence.
• Make refusal a first-class outcome. A system that never says "I do not have that information" is not confident, it is unmonitored. Include unanswerable questions in the eval set and measure the refusal rate as a metric in its own right.

Permissions, freshness, and deletion

These three turn a prototype into something an enterprise can actually deploy, and all three are hard to retrofit.

Permissions. Retrieval must be filtered by the requesting user's access rights, enforced in the query rather than after it. Post-filtering a result set leaks information through result counts and latency, and worse, tends to return empty answers that look like system failure. Carry access control lists or group identifiers as chunk metadata at ingestion, and make the filter a mandatory predicate. The failure mode here is not a bad answer; it is an employee reading a document they should not see, which is an incident.

Freshness. Decide the staleness budget per source and design ingestion to meet it. Incremental sync on document change is worth building early; full re-index as a nightly job is a reasonable interim. Surface the source's last-modified date in the answer so users can judge for themselves.

Deletion. When a document is deleted at source, its chunks must leave the index. Under GDPR and India's DPDP Act this is not a nice-to-have. Test it: delete a document, then query for its content and confirm it is gone. Very few teams have run that test, and it is the first thing a serious security review asks about.

Latency and cost budgets

Write the budget down before optimising, because "it feels slow" is not an engineering target. A worked example for an interactive assistant with a two-second p95 target:

stage                        p50      p95     notes
-------------------------------------------------------------------
intent + query rewrite        90ms    250ms   small model, cacheable
dense retrieval               25ms     60ms   ef_search tuned
lexical retrieval             15ms     40ms   runs in parallel with dense
fusion + filtering             5ms     10ms   in-process
re-ranking                    80ms    200ms   cross-encoder, 30 candidates
generation (streamed)        600ms   1300ms   time to first token matters most
-------------------------------------------------------------------
total (parallel-aware)       ~810ms  ~1820ms  within a 2s p95 budget

Two observations that hold across engagements. First, generation dominates, so streaming the response changes perceived performance more than shaving milliseconds anywhere else. Second, retrieval is cheap in latency and cheap in money; the temptation to skip re-ranking to save 200 milliseconds usually costs more quality than it buys speed.

On money, per-query cost is dominated by generation tokens, and context length is the lever. Passing 12 chunks where 5 would do inflates every single query for the life of the system. Retrieve broadly, re-rank hard, and pass few. Prompt caching on a stable system prompt and cached embeddings for repeated queries are the two other reliable savings.

Failure taxonomy

When an answer is wrong, the useful question is which stage owns it. This mapping turns a vague complaint into a specific investigation:

Symptom	Likely stage	First check
Answer omits information that exists in the corpus	Retrieval	Is the passage in the top 50 at all? If not, chunking or lexical coverage
Right document retrieved, wrong passage used	Re-ranking or context order	Rank of the correct chunk after fusion versus after re-ranking
Answer contradicts the retrieved context	Generation	Faithfulness eval; check context ordering and length
Citations point to passages that do not exist	Generation, post-check missing	Validate cited ids against supplied context ids
Answer from an outdated document version	Corpus and ingestion	Superseded content in the index; recency filter
User sees content they should not	Filtering	Permission predicate applied pre-retrieval, not post
Exact identifiers and codes never match	Retrieval	Is lexical search actually in the mix, or dense only
Confident answer to an out-of-scope question	Intent routing and refusal policy	Unanswerable cases in the eval set; refusal rate

Production readiness checklist

• Intent classification routes non-retrieval questions elsewhere
• Chunking follows document structure, with heading context prepended
• Hybrid retrieval with reciprocal rank fusion, not dense-only
• Cross-encoder re-ranking, or a measured decision not to use it
• Permission filters applied inside the retrieval query
• Citations enforced and validated after generation
• Refusal behaviour measured on unanswerable cases
• Deletion propagates from source to index, and has been tested
• Latency budget written down, with streamed generation
• Per-query cost known, and context length deliberately bounded
• An eval set that gates changes in CI, with every incident added as a case
• Traces retained per query: retrieved ids, ranks, scores, final context

That last item deserves emphasis. Without per-query traces recording what was retrieved and what was passed to the model, every quality investigation becomes guesswork. Log them from the first day of the pilot; the cost is trivial and the diagnostic value compounds.

Honest boundaries

RAG does not make a model reason better. It does not fix a corpus that is contradictory, out of date, or missing the answer entirely, and a substantial share of "the AI got it wrong" incidents turn out to be documentation problems wearing an AI costume. It adds real operational surface: an ingestion pipeline, an index, a re-ranker, and a permission model, all of which need owners.

The upside is that every stage above is measurable and independently improvable, which is exactly what makes RAG a reasonable thing to promise a customer. Pair it with an evaluation harness that can tell you whether last week's change helped, and you have a system that gets better on a schedule rather than by luck.