Chunking strategies for retrieval

Chunking strategies for retrieval#

Track 02 - RAG · Notebook 01 · Runtime: ≈3 min on CPU

Prerequisites: you know what a dictionary / hash map is. You’ve used str.split() before. That’s literally it.

What you’ll know by the end: why “retrieval-augmented generation” depends as much on how you cut up your documents as on which embedding model you use, and how to reason about that tradeoff with numbers instead of vibes.

What problem are we even solving?#

You have a pile of documents. A user asks a question. You want to paste the most relevant bits of those documents into the LLM’s prompt so it can answer grounded in real content, not in whatever the model hallucinates.

The standard recipe, top to bottom:

Chunk every document into pieces (this notebook).
Embed each chunk into a vector (dense retrieval) or a sparse term vector (BM25 / SPLADE).
At query time, embed the query and find the closest chunks by vector similarity.
Stuff those chunks into the LLM prompt as context.

Step 1 sounds trivial - “split the text.” It isn’t. If your chunks are too big, irrelevant sentences drown out relevant ones during retrieval. If they’re too small, a complete thought gets cut across boundaries and neither chunk on its own matches the query. Everything downstream is bottlenecked by how you chose to cut.

So: four chunking strategies, one small corpus, one set of queries, one Recall@10 number per strategy. That’s the whole notebook.

A one-minute primer on embeddings#

An embedding model is a function text → vector in R^d where similar-meaning texts get similar vectors (close in cosine distance). You can think of it as a soft, semantic version of hashing:

A regular hash (hash("cat") = 4127…) gives you exact-equality lookups. hash("feline") is a completely different number.
An embedding (embed("cat") ≈ [0.12, -0.43, …]) gives you semantic lookups. embed("feline") lives very close to embed("cat") in vector space, because the model has learned that they mean roughly the same thing.

Retrieval = “find the chunk whose embedding has the highest dot product with the query’s embedding.”

Fallback embedder (no model download required)#

Real RAG uses sentence-transformers/BAAI/bge-small-en-v1.5 or similar. In this notebook we try to load that model, and if sentence-transformers isn’t installed we fall back to a deterministic bag-of-ngrams hash embedder. It’s not as good as a trained model, but on a small corpus it still distinguishes topics well enough to make the chunking comparison meaningful - and it runs in CI with no downloads.

from llm_systems_cookbook.nb import bootstrap

import random
import re

import numpy as np

s = bootstrap("02_rag_01_chunking_strategies")

Step 1 - A tiny synthetic corpus with known ground truth#

We can’t use a real BEIR benchmark here (too much data, too many model downloads). Instead we construct a corpus where we know which documents are relevant to each query: each document is 2-3 real topical paragraphs concatenated, and each query targets one specific topic.

Ground-truth relevance is computed at construction time: a document is relevant to query Q if and only if one of the document’s topics matches Q’s topic. This is idealised but enforces a clear signal so the chunking comparison below isn’t muddied by noise in the corpus itself.

TOPICS = [
    ("photosynthesis", "Photosynthesis is the biochemical process by which plants convert light energy into chemical energy. "
                       "Chlorophyll in plant cells absorbs red and blue wavelengths of light. "
                       "The Calvin cycle fixes carbon dioxide into carbohydrates inside the chloroplast."),
    ("mitochondria", "Mitochondria are double-membrane organelles that generate adenosine triphosphate via oxidative phosphorylation. "
                     "They carry their own circular DNA, a remnant of an ancient endosymbiotic bacterium. "
                     "Crista folding in the inner membrane increases the surface area available for electron transport."),
    ("neuron_action_potential", "A neuron's action potential is a rapid, self-propagating change in membrane voltage. "
                                 "Voltage-gated sodium channels open when the membrane depolarises past threshold. "
                                 "Saltatory conduction along myelinated axons dramatically increases signal velocity."),
    ("enzyme_kinetics", "Enzyme kinetics is described by the Michaelis-Menten equation at steady state. "
                        "Km is the substrate concentration at which reaction velocity is half of Vmax. "
                        "Competitive inhibitors raise the apparent Km without affecting Vmax."),
    ("crispr", "CRISPR-Cas9 is a genome-editing tool derived from bacterial adaptive immunity. "
               "A guide RNA directs Cas9 to a complementary DNA sequence flanked by a PAM motif. "
               "The resulting double-strand break is repaired by non-homologous end joining or homology-directed repair."),
    ("plate_tectonics", "Plate tectonics explains the large-scale motion of Earth's lithosphere. "
                        "Mid-ocean ridges are divergent boundaries where new crust is created. "
                        "Subduction zones recycle oceanic crust back into the mantle."),
    ("black_holes", "Black holes are regions of spacetime with gravity so strong that not even light escapes. "
                    "The event horizon is the boundary beyond which escape becomes impossible. "
                    "Hawking radiation predicts that black holes slowly evaporate via quantum effects at the horizon."),
    ("turbulence", "Turbulence is characterised by chaotic, multi-scale motion in fluids. "
                   "The Reynolds number measures the ratio of inertial to viscous forces. "
                   "Kolmogorov's -5/3 law describes the energy spectrum in the inertial subrange."),
    ("ferroelectric", "Ferroelectric materials exhibit spontaneous electric polarisation reversible by an applied field. "
                      "Below the Curie temperature the polarisation is stable and hysteretic. "
                      "Barium titanate is a canonical example used in capacitor dielectrics."),
    ("superconductivity", "Superconductivity is the disappearance of electrical resistance below a critical temperature. "
                          "BCS theory explains conventional superconductivity via Cooper-paired electrons. "
                          "Type-II superconductors admit magnetic flux as quantised vortices rather than expelling it entirely."),
]

rng = random.Random(42)
DOCS: list[dict] = []
for i in range(40):
    chosen = rng.sample(TOPICS, rng.randrange(2, 4))
    DOCS.append({
        "id": f"doc-{i:03d}",
        "text": "\n\n".join(t[1] for t in chosen),
        "topics": [t[0] for t in chosen],
    })

QUERIES: list[dict] = [
    {"id": "q0",  "text": "how do plants convert light into energy?",                         "topic": "photosynthesis"},
    {"id": "q1",  "text": "which organelle produces ATP via oxidative phosphorylation?",       "topic": "mitochondria"},
    {"id": "q2",  "text": "what causes a neuron to fire an action potential?",                 "topic": "neuron_action_potential"},
    {"id": "q3",  "text": "what is Km in enzyme kinetics?",                                     "topic": "enzyme_kinetics"},
    {"id": "q4",  "text": "how does Cas9 find its target DNA?",                                 "topic": "crispr"},
    {"id": "q5",  "text": "where is new oceanic crust created?",                                "topic": "plate_tectonics"},
    {"id": "q6",  "text": "what is the event horizon?",                                         "topic": "black_holes"},
    {"id": "q7",  "text": "what does the Reynolds number measure?",                             "topic": "turbulence"},
    {"id": "q8",  "text": "what happens to ferroelectrics below the Curie temperature?",        "topic": "ferroelectric"},
    {"id": "q9",  "text": "what forms Cooper pairs in superconductors?",                        "topic": "superconductivity"},
    {"id": "q10", "text": "what does Hawking radiation predict?",                               "topic": "black_holes"},
    {"id": "q11", "text": "what is saltatory conduction?",                                      "topic": "neuron_action_potential"},
]
QRELS = {q["id"]: {d["id"] for d in DOCS if q["topic"] in d["topics"]} for q in QUERIES}
print(f"corpus: {len(DOCS)} docs, {len(QUERIES)} queries")
print(f"example: {DOCS[0]['topics']}")

Step 2 - Choose an embedder (or fall back)#

The try/except here is the entire “works on CI” trick. If sentence-transformers imports, we use the real model. If not, we fall back to the hash embedder - which still preserves enough structure that the relative ordering of the chunking strategies is informative.

_TOKEN_RE = re.compile(r"[A-Za-z]{2,}")


def tokenize(text: str) -> list[str]:
    return [w.lower() for w in _TOKEN_RE.findall(text)]


class HashEmbedder:
    '''Bag-of-ngrams hash embedding - deterministic, no downloads.'''

    def __init__(self, dim: int = 256) -> None:
        self.dim = dim

    def encode(self, texts: list[str]) -> np.ndarray:
        out = np.zeros((len(texts), self.dim), dtype=np.float32)
        for i, text in enumerate(texts):
            toks = tokenize(text)
            for tok in toks:
                out[i, hash(("u", tok)) % self.dim] += 1.0
            for j in range(len(toks) - 1):
                out[i, hash(("b", toks[j], toks[j + 1])) % self.dim] += 0.5
        return out / (np.linalg.norm(out, axis=1, keepdims=True) + 1e-9)


try:
    from sentence_transformers import SentenceTransformer  # noqa: F401

    class STEmbedder:
        def __init__(self) -> None:
            from sentence_transformers import SentenceTransformer
            self.model = SentenceTransformer("BAAI/bge-small-en-v1.5")

        def encode(self, texts: list[str]) -> np.ndarray:
            return self.model.encode(texts, convert_to_numpy=True, normalize_embeddings=True).astype(np.float32)

    embedder: object = STEmbedder()
    print("using sentence-transformers (bge-small-en-v1.5)")
except Exception as e:
    embedder = HashEmbedder(dim=256)
    print(f"using hash fallback  (sentence-transformers unavailable: {type(e).__name__})")

Step 3 - Four ways to cut up a document#

Each strategy is a function text -> list[str]. They encode different assumptions about where the “natural” boundaries of a document are:

Strategy	Core assumption	When it fails
Fixed-length	Cut every N characters, with an overlap. Content structure is irrelevant.	Splits sentences mid-thought; an overlap helps but not always enough.
Recursive	Paragraphs are the unit. If a paragraph is too big, fall back to sentences. If still too big, hard cut.	Still struggles with very long paragraphs that cover multiple subtopics.
Semantic	Neighbouring sentences with similar embeddings belong together. Cut where similarity drops below a threshold.	Depends on the embedder’s signal; noisy on short docs.
Late	Embed the whole document once for global context, then tag each paragraph with a summary prefix so retrieval can use that context.	Loses some paragraph independence; the summary prefix has to stay short.

Fixed is the baseline. Recursive is the default in LangChain & LlamaIndex. Semantic is Jina’s 2024 idea. Late is a 2024 paper (Günther et al.) and often gives the best recall when implemented with a real long-context embedder.

def chunk_fixed(text: str, size: int = 200, overlap: int = 40) -> list[str]:
    chunks: list[str] = []
    i = 0
    while i < len(text):
        chunks.append(text[i : i + size])
        i += max(1, size - overlap)
    return chunks


def chunk_recursive(text: str, target: int = 200) -> list[str]:
    def split_big(block: str) -> list[str]:
        if len(block) <= target * 1.5:
            return [block]
        sents = re.split(r"(?<=[.!?])\s+", block)
        out, cur = [], ""
        for sent in sents:
            if len(cur) + len(sent) > target and cur:
                out.append(cur.strip()); cur = sent
            else:
                cur = (cur + " " + sent).strip()
        if cur:
            out.append(cur.strip())
        return [p if len(p) <= target * 2 else next(iter(chunk_fixed(p, size=target)), p) for p in out]

    return [c for p in text.split("\n\n") if p.strip() for c in split_big(p.strip())]


def chunk_semantic(text: str, embedder, percentile: float = 95.0) -> list[str]:
    sents = [s for s in re.split(r"(?<=[.!?])\s+", text.strip()) if s]
    if len(sents) <= 2:
        return [text]
    embs = embedder.encode(sents)
    dists = [1 - float(embs[i] @ embs[i + 1]) for i in range(len(sents) - 1)]
    cut = float(np.percentile(dists, percentile))
    chunks, cur = [], []
    for i, sent in enumerate(sents):
        cur.append(sent)
        if i < len(dists) and dists[i] >= cut:
            chunks.append(" ".join(cur)); cur = []
    if cur:
        chunks.append(" ".join(cur))
    return chunks


def chunk_late(text: str) -> list[str]:
    '''Approximation: prefix each paragraph with the first 120 chars of the doc for global context.'''
    paras = [p.strip() for p in text.split("\n\n") if p.strip()]
    head = text[:120]
    return [f"[DOC] {head} ... {para}" for para in paras]

Step 4 - Score them on Recall@10#

We embed every chunk, embed every query, rank chunks by dot product, and check: was at least one chunk from a relevant document in the top 10?

The metric is Recall@10: (# queries with ≥1 relevant chunk in top-10) / (# queries). Higher is better. On this easy corpus all strategies will be above 0.85 because the topics are well separated; the comparison is really about which one fails most gracefully when corpora get harder.

def build_chunks(strategy: str, embedder) -> tuple[list[str], list[str]]:
    chunk_texts: list[str] = []
    chunk_doc_ids: list[str] = []
    for doc in DOCS:
        if strategy == "fixed":     pieces = chunk_fixed(doc["text"], size=200, overlap=40)
        elif strategy == "recursive": pieces = chunk_recursive(doc["text"], target=200)
        elif strategy == "semantic":  pieces = chunk_semantic(doc["text"], embedder)
        elif strategy == "late":      pieces = chunk_late(doc["text"])
        else: raise ValueError(strategy)
        for p in pieces:
            chunk_texts.append(p)
            chunk_doc_ids.append(doc["id"])
    return chunk_texts, chunk_doc_ids


def recall_at_k(strategy: str, embedder, k: int = 10) -> float:
    chunks, chunk_doc_ids = build_chunks(strategy, embedder)
    chunk_embs = embedder.encode(chunks)
    query_embs = embedder.encode([q["text"] for q in QUERIES])
    top_k = np.argsort(-(query_embs @ chunk_embs.T), axis=1)[:, :k]
    hits = sum(
        bool({chunk_doc_ids[i] for i in top_k[q_i]} & QRELS[q["id"]])
        for q_i, q in enumerate(QUERIES)
    )
    return hits / len(QUERIES)


recall = {strat: recall_at_k(strat, embedder, k=10) for strat in ["fixed", "recursive", "semantic", "late"]}
for k, v in recall.items():
    print(f"  {k:>10}  Recall@10 = {v:.3f}")

s.check("fixed_recall_floor",     lambda: recall["fixed"]     >= 0.60, msg=f"{recall['fixed']:.3f}")
s.check("recursive_recall_floor", lambda: recall["recursive"] >= 0.70, msg=f"{recall['recursive']:.3f}")
s.check("semantic_recall_floor",  lambda: recall["semantic"]  >= 0.70, msg=f"{recall['semantic']:.3f}")
s.check("late_recall_floor",      lambda: recall["late"]      >= 0.65, msg=f"{recall['late']:.3f}")
s.check(
    "structured_chunkers_do_not_underperform_fixed",
    lambda: min(recall["recursive"], recall["semantic"], recall["late"]) >= recall["fixed"] - 0.1,
    msg=str(recall),
)

Recall vs chunk size#

A single number per strategy hides the knob that actually matters: chunk size. Sweep chunk_fixed across sizes with and without overlap, and hold the structured chunkers (recursive, semantic) as horizontal references. The fixed curve has a broad plateau - going below ~100 chars fragments sentences, going above ~400 dilutes the embedding signal.

import matplotlib.pyplot as plt

def fixed_recall(size: int, overlap: int) -> float:
    texts, ids = [], []
    for doc in DOCS:
        for p in chunk_fixed(doc["text"], size=size, overlap=overlap):
            texts.append(p); ids.append(doc["id"])
    qe = embedder.encode([q["text"] for q in QUERIES])
    top_k = np.argsort(-(qe @ embedder.encode(texts).T), axis=1)[:, :10]
    return sum(bool({ids[i] for i in top_k[qi]} & QRELS[q["id"]])
               for qi, q in enumerate(QUERIES)) / len(QUERIES)

sizes = [80, 120, 160, 200, 280, 400, 600]
r_no = [fixed_recall(sz, 0) for sz in sizes]
r_ov = [fixed_recall(sz, sz // 5) for sz in sizes]

fig, ax = plt.subplots(figsize=(6.5, 3.6))
ax.plot(sizes, r_no, "o-", label="fixed (overlap=0)")
ax.plot(sizes, r_ov, "o-", label="fixed (overlap=20%)")
ax.axhline(recall["recursive"], color="tab:green", linestyle="--",
           label=f"recursive={recall['recursive']:.2f}")
ax.axhline(recall["semantic"], color="tab:red", linestyle="--",
           label=f"semantic={recall['semantic']:.2f}")
ax.set_xlabel("chunk size (chars)"); ax.set_ylabel("Recall@10")
ax.set_title("chunk size vs retrieval recall")
ax.set_ylim(0, 1.02); ax.grid(True, alpha=0.3)
ax.legend(fontsize=8, loc="lower center")
fig.tight_layout(); plt.show()

Exercises#

Ablate overlap. Set chunk_fixed overlap to 0 and rerun. The only penalty should be on query terms that happened to straddle a chunk boundary; measure how much recall drops.
Change the percentile. Semantic chunking cuts at the 95th percentile of neighbour-embedding distances. What happens at 75? At 99? Where’s the sweet spot for this corpus?
Larger chunks. Rerun every strategy with target=500 or size=500. Recall often drops with bigger chunks because dense retrieval scores concentrate in short, focused passages. See for yourself.
Plug in a real embedder. Install sentence-transformers and rerun. The hash embedder cares about surface forms; a real model cares about meaning. Does the ranking of strategies change?

What’s next#

Notebook 02 in this track (02_faiss_dense_retrieval) swaps our naive dot-product search for FAISS indices (flat, IVF-PQ, HNSW) and measures the latency/recall Pareto. Notebook 05 adds a cross-encoder reranker on top.

s.summary()
s.save()