RAG Pipelines#

Sub-chapter 3 of AI/ML · Retrieval done seriously

RAG - retrieval-augmented generation - is the pattern behind almost every "chat with your docs" feature you've seen. The cheap version is a weekend hack. The version that actually works under customer data, in production, with citations and tenant isolation, is real engineering.

This sub-chapter is the field guide. By the end you should know not just how to build a RAG, but which knob to turn when it returns plausible-but-wrong results.

Outline#

1. The RAG pipeline - ingest, retrieve, generate, evaluate
2. Chunking - the most under-rated decision
3. Embeddings, properly - which model, what dimensions, when to re-index
4. Retrieval - vector, lexical, hybrid - why hybrid wins
5. Re-ranking - the cheap quality lift
6. Citations - never ship a RAG without them
7. Metadata filtering - tenant isolation and the #1 production bug
8. Evals for RAG - recall, precision, faithfulness, end-to-end
9. Operational reality - re-index strategy, staleness, cost

101 Primer#

The pipeline#

┌───────────────┐    ┌──────────┐    ┌────────────┐
│   ingest      │ →  │   store  │ →  │  retrieve  │
│ chunk+embed   │    │   vec DB │    │ + re-rank  │
└───────────────┘    └──────────┘    └─────┬──────┘
                                           │
                                           ▼
                                    ┌────────────┐
                                    │  generate  │
                                    │ LLM + cites│
                                    └────────────┘

Failures show up at any stage. Cheap-version bugs cluster in chunking and retrieval. Serious-version bugs cluster in evals (you don't know you're failing) and metadata filtering (you return tenant A's data to tenant B).

Chunking - get this right or nothing else matters#

The chunk is the unit of retrieval. Bad chunks → no amount of LLM cleverness will fix.

What "bad" looks like:

• Too large - relevant content is buried in noise; the LLM's attention dilutes; cost explodes.
• Too small - context is lost; "the company increased prices by 30%" becomes "30%."
• Cut at the wrong boundary - a single sentence split across two chunks, both useless.
• Loses structure - a table flattened to prose; a markdown header detached from its content.

Strategies, ordered by my preference:

1. Semantic chunking - split on headings, paragraph breaks, list boundaries. Respect markdown / HTML structure. Add the parent heading as a prefix to each child chunk for context.
2. Recursive character splitting with a target size (e.g., 400–600 tokens) and overlap (10–15%). The standard fallback when you don't have structure.
3. Sentence-window - embed the centre sentence, retrieve, then expand to ±N sentences for the LLM. Decouples retrieval granularity from generation context.
4. Late-chunking (newer) - embed the whole document, then chunk for storage. Preserves long-range context in the embedding. Worth a try if quality is critical.

Anti-patterns:

• Fixed-size chunks (e.g., "every 1024 characters") on heterogeneous corpora.
• Stripping headings during normalisation. You just deleted the most important context.
• Chunking PDFs as flat text. Parse the structure (unstructured.io, docling, or Adobe PDF Extract) and chunk by section.

Embeddings#

Once chunks are right, embeddings are mostly a defaulted decision:

• Default: text-embedding-3-small (OpenAI, 1536d). Cheap, good, broadly compatible.
• Self-hosted: BGE-large-en-v1.5 or Jina embeddings v3.
• Multilingual: BGE-m3 or multilingual-e5-large.
• Code corpora: Voyage AI code embeddings or Jina code v2.

Rules:

• Never mix embedding models in a collection. Vectors aren't compatible.
• Store the embedding model name + version with each vector. You'll rotate models.
• Re-embedding is a backfill, not a config change. Plan the migration.

Retrieval - vector, lexical, hybrid#

Pure vector search loses to hybrid search (vector + BM25 lexical) on most real corpora. The reason: vector search captures meaning but misses exact tokens. A user searching for OrderId=ABC-123 wants exact match; vector search will return semantically related orders.

Hybrid pattern:

# Pseudocode
vec_hits = qdrant.search(query_vec, limit=20)
bm25_hits = lex_index.search(query_text, limit=20)
merged = reciprocal_rank_fusion(vec_hits, bm25_hits, k=60)
top = merged[:10]

Reciprocal rank fusion is a one-liner that combines ranked lists without needing to normalise scores. Use it.

Tools that give you hybrid out of the box: Weaviate, Qdrant (via recommendation_strategy="rrf"), Vespa, Elastic with vector + BM25. If you're on pgvector, you'll wire BM25 yourself via tsvector and fuse.

Re-ranking - the cheap quality lift#

Retrieve top-20 → re-rank to top-5 with a cross-encoder. This is one of the highest-ROI moves in RAG.

# After hybrid retrieval, before LLM
reranked = cohere_rerank(query, retrieved_chunks)  # or BGE rerank, Jina rerank
top_k = reranked[:5]

Why it works: the retrieval step uses cheap dual-encoder embeddings (each side encoded independently). The re-ranker uses a cross-encoder that scores (query, chunk) pairs jointly - slower but much more accurate. You only run it on the top-20, not the full corpus, so the latency is bounded.

Expected lift: NDCG@5 typically jumps 5–15 points. Easy money.

Citations - never ship a RAG without them#

If your RAG doesn't show the user which sources the answer came from, you have built a hallucination machine wearing a confidence costume.

The pattern:

1. Each chunk has a stable ID and a "source" (URL, doc ID, page number).
2. The LLM prompt instructs: "For each claim in your answer, append a citation [N] where N is the chunk's source index."
3. The UI renders the citations as hover-able / clickable references.

Even better: design the answer format so the LLM produces an array of {claim, source_id} pairs, and you render the assembled answer. Eliminates the "model forgot to cite" failure mode.

Metadata filtering - the production bug you cannot afford#

Vector stores let you filter by metadata at retrieval time:

qdrant.search(
    collection="docs",
    query_vector=q,
    query_filter=Filter(must=[
        FieldCondition(key="tenant_id", match=MatchValue(value=tenant)),
    ]),
    limit=10,
)

Three things will absolutely happen on your watch if you're not deliberate:

1. Missing the filter on one query path → you return tenant A's data to tenant B. This is a critical incident.
2. Indexing the metadata field too late → filter silently no-ops (queries return matches but the filter doesn't apply).
3. Tenant collisions in clustering / IDs → same chunk-id across tenants overwrites.

Engineering practices that prevent this:

• A wrapper function tenant_search(tenant_id, query) that's the only way to query. Direct SDK calls are banned in app code.
• Integration test that calls tenant_search for tenant A and asserts zero results contain tenant-B-only content.
• Indexed metadata fields are part of the collection setup, reviewed in PR.

Evals for RAG#

You need three levels of eval, top to bottom:

1. Retrieval eval - given a query, did the retriever return the right chunk in the top-K? Metric: Recall@K, MRR.
2. Generation eval - given retrieved chunks + query, is the answer correct and grounded? Metric: faithfulness (no claims unsupported by sources), answer relevance (does it answer the question).
3. End-to-end eval - given a query, is the final answer good? Metric: human label or LLM-as-judge against goldens.

Build all three. Cheaply. A spreadsheet of 50 (query → expected chunk → expected answer) rows is enough to start. Add an LLM-judge prompt that scores faithfulness and you have a real test suite.

Tools: Ragas, TruLens, Braintrust, or a 100-line script in your repo. The framework matters less than running the evals on every prompt change.

Operational reality#

• Re-indexing is a known event. Plan: new collection, dual-write during cutover, swap when ready, retire the old.
• Staleness - define "how fresh must the data be?" per source. Build the re-ingest cadence around that.
• Cost - embeddings + storage + query compute. Most cost lives in re-embedding during model rotation, not steady state.
• Caching the (query → top-K chunks) result is high-ROI for FAQ-shaped traffic.
• Tracing every RAG call - query, retrieved chunks, final answer, citations. Without traces you can't debug.

Hands-on Checkpoints#

• Set up Qdrant (Docker) or pgvector. Ingest 100 markdown docs (or 50 PDFs of your choosing).
• Implement semantic chunking respecting headings. Compare to naïve fixed-size chunks on 10 questions by eye.
• Add hybrid retrieval (vector + BM25 via tsvector or Elasticsearch) with reciprocal rank fusion.
• Add a re-ranker step (Cohere rerank, BGE, or Jina). Measure NDCG@5 lift.
• Render citations in your answer UI. Make them clickable.
• Enforce tenant filtering through a wrapper. Write an integration test that fails if direct SDK access is used.
• Build a 20-row eval set with (query → expected chunk_id → expected answer). Compute Recall@5 + an LLM-judge faithfulness score.
• Change one knob (chunk size, embed model, re-ranker on/off). Measure the delta. Decide.

Further reading#

• Anthropic - Contextual retrieval - the chunk-context trick that improves recall dramatically
• Pinecone - Retrieval-augmented generation - surprisingly even-handed
• Eugene Yan - Patterns for LLM-based search
• Jerry Liu (LlamaIndex) - Advanced RAG techniques
• Hamel Husain - Evaluating RAG
• [BGE / Jina / Cohere rerank docs] - pick one, read it cover-to-cover

Local resources#

PDFs live in the chapter folder and open in the in-app viewer. Reference one with a /r/<chapter-dir>/<filename>.pdf link:

• Sample paper (in-app viewer) - demo PDF to confirm the viewer works on your machine

Welzin opinion: Every RAG starts as a magic trick and ends as engineering. The magic is gone by week two. What's left is chunking discipline, re-ranking, citations, tenant filters, and evals. Build those from day one and skip the magic-trick phase.