RAG engineering: chunking, embeddings, and reranking

The quality of a RAG system is decided before the model generates a single token. It's decided by how the document was split, which embedding represents each piece, how that piece competes against others during retrieval, and whether the pipeline can objectively measure whether it's retrieving the right content. This article covers those four decisions.

Structure-aware chunking, not fixed size

Splitting documents into fixed-size blocks (say, 500 tokens with 50 of overlap) is the most common starting point and the most frequent cause of poor retrieval. The problem: fixed size ignores the document's semantic structure, cutting a table in half or separating a heading from its explanation.

The alternative is chunking driven by document structure:

• Documents with markup (Markdown, HTML, styled DOCX) — use headings as natural chunk boundaries, preserving hierarchy (a subsection chunk carries its parent section's title as context).
• Tables — treat as an atomic unit, never split a table across chunks; if the table is too large, serialize row by row with the header repeated in each chunk.
• Legal/contractual documents — use numbered clauses or articles as the unit, since that's the granularity that will be cited later.
• Source code — use function or class as the unit, not line or character count.

A practical test: if a human reading only that chunk, in isolation, can't understand what it's about without extra context, the chunk was cut wrong.

Embedding choice and versioning

The choice of embedding model isn't final — models evolve, and switching embeddings is inevitable over a system's lifetime. This has a direct engineering implication: the vector index must be versioned alongside the model that generated it.

embedding_model_id     string   -- e.g. "text-embedding-v3"
embedding_dim          int
index_version          string
chunk_id               uuid
vector                 float[]
generated_at           datetime

Never mix vectors from different embedding models in the same similarity search index — distances between embeddings from different models aren't comparable. Switching models requires a full reindex, not an incremental one.

Objective criteria for choosing an embedding model: performance on a domain-specific benchmark (not just generic benchmarks like MTEB, which may not reflect your domain's vocabulary), cost per token, dimensionality (affects storage cost and search latency), and support for your corpus's predominant language.

Reindexing

Reindexing happens in three scenarios, each with a distinct strategy:

For a model change, keep both indexes running in parallel during the transition and validate the new index's retrieval quality against the same evaluation set before cutover — never switch the production index without that gate.

Reranking

Vector search retrieves candidates by approximate semantic similarity, but embedding similarity isn't synonymous with relevance to the specific question. A reranking stage, applied to the top-k candidates (typically k=20 to 50) before selecting the few that go into the prompt (typically 3 to 8), corrects much of that mismatch.

Reranking with a cross-encoder (which evaluates the question-chunk pair jointly, unlike the bi-encoder used in vector search) has higher computational cost per pair evaluated, but is applied to a small candidate set, which makes it viable at latency. In practice, reranking tends to be the most visible and cheapest quality gain in the whole RAG pipeline — usually more impactful than switching embedding models.

Retrieval evaluation

Without a metric, there's no way to know whether a pipeline change (new chunking, new embedding, new reranker) improved or worsened retrieval. The standard metrics:

• Recall@k — of all chunks relevant to a question, how many appear among the top-k retrieved. Measures whether the right content is in the candidate set.
• nDCG (normalized Discounted Cumulative Gain) — weighs not just whether the relevant chunk appeared, but at what position, penalizing relevant chunks that appear far from the top.

Building this evaluation set requires a manual step: a set of real (or representative) questions with the correct chunks annotated by a human. Without this gold set, any pipeline change is judged by impression, not data.

query: "what is the warranty period for product X?"
relevant_chunk_ids: ["doc_42#chunk_7", "doc_42#chunk_8"]

Run this evaluation as part of your CI pipeline whenever chunking, embedding, or reranker change — treat a recall@k regression the way you'd treat an automated test regression.

Source citation and current version

Every chunk retrieved and cited in the final answer should carry enough metadata to point back to the source document and the version current at the time of the answer. This serves two purposes: it lets the user verify the information, and it enables later auditing if the source document was updated or corrected after the answer was generated.

What to do on Monday

1. 1.Review your current chunking strategy: if it's fixed size without regard to structure, prioritize migrating to structural chunking for the document types with the highest query volume.
2. 2.Add ${'embedding_model_id'} and ${'index_version'} as mandatory index metadata, if they don't exist yet.
3. 3.Build an evaluation set with 30 to 50 real annotated questions and measure current recall@k and nDCG before making any changes.
4. 4.If you don't have reranking yet, add a cross-encoder stage over the top-20 before any other RAG investment — it's the highest return-on-effort adjustment available.

Further reading

Executive track:

• RAG is not magic: the real cost of giving AI enterprise context — the business view of this same topic.