HyDE and query rewriting

Open in Colab ▶️ Run this notebook in Colab

HyDE and query rewriting#

Track 02 - RAG · Notebook 06 · Runtime: ≈30 s on CPU

Prerequisites: 02_rag/02 (FAISS), 02_rag/04 (ColBERT).

Paper: Gao et al. 2022, Precise Zero-Shot Dense Retrieval without Relevance Labels (HyDE) (2212.10496).

What#

Short queries embed into short vectors that don’t always land near the right documents. Three rewriting strategies fix this by making the query look more like what the answer document looks like:

  • HyDE (Hypothetical Document Embeddings). Prompt an LLM for a hypothetical answer to the query - even a wrong one - and embed that. The hypothetical answer shares vocabulary and phrasing with real answer docs, so its embedding retrieves them.

  • Multi-query. Generate 3-5 paraphrases of the query, retrieve with each, union results via RRF.

  • Decomposition. Split a multi-hop query into sub-questions, retrieve for each, union.

We don’t call a real LLM. Instead we supply a stubbed hypothetical_answer(query) function that returns a short passage containing the answer’s vocabulary; embeddings come from the same hash embedder as the chunking notebook.

from llm_systems_cookbook.nb import bootstrap

import re
from collections import defaultdict

import numpy as np

s = bootstrap("02_rag_06_hyde_query_rewriting")

Corpus, hash embedder, evaluation#

Same topical corpus (20 docs) and queries (8) as earlier notebooks, with a deterministic bag-of-ngrams hash embedder so scoring is reproducible.

_WORD_RE = re.compile(r"[A-Za-z]{2,}")


def tokenize(text: str) -> list[str]:
    return [w.lower() for w in _WORD_RE.findall(text)]


import hashlib


def _stable_hash(*parts: str) -> int:
    h = hashlib.md5("|".join(parts).encode("utf-8")).digest()
    return int.from_bytes(h[:8], "big")


class HashEmbedder:
    def __init__(self, dim: int = 256) -> None:
        self.dim = dim

    def encode(self, texts: list[str]) -> np.ndarray:
        out = np.zeros((len(texts), self.dim), dtype=np.float32)
        for i, t in enumerate(texts):
            toks = tokenize(t)
            for tok in toks:
                out[i, _stable_hash("u", tok) % self.dim] += 1.0
            for j in range(len(toks) - 1):
                out[i, _stable_hash("b", toks[j], toks[j + 1]) % self.dim] += 0.5
        return out / (np.linalg.norm(out, axis=1, keepdims=True) + 1e-9)


emb = HashEmbedder()

DOCS = [
    {"id": "d0", "topic": "photosynthesis",
     "text": "Plants convert light into chemical energy via photosynthesis in the chloroplast. "
             "Chlorophyll absorbs red and blue wavelengths. The Calvin cycle fixes carbon dioxide."},
    {"id": "d1", "topic": "mitochondria",
     "text": "Mitochondria generate ATP through oxidative phosphorylation. "
             "The inner membrane has crista folds. They contain their own circular DNA."},
    {"id": "d2", "topic": "neuron_action_potential",
     "text": "A neuron fires an action potential when voltage-gated sodium channels open. "
             "Saltatory conduction along myelinated axons dramatically increases signal velocity."},
    {"id": "d3", "topic": "enzyme_kinetics",
     "text": "Enzyme kinetics follows the Michaelis-Menten equation. "
             "Km is the substrate concentration at half-maximal velocity."},
    {"id": "d4", "topic": "crispr",
     "text": "CRISPR-Cas9 uses a guide RNA to direct the nuclease to a target DNA flanked by a PAM motif."},
    {"id": "d5", "topic": "black_holes",
     "text": "Black holes have an event horizon beyond which no light escapes. "
             "Hawking radiation predicts slow evaporation via quantum effects at the horizon."},
    {"id": "d6", "topic": "turbulence",
     "text": "Turbulence is characterised by chaotic multiscale motion. "
             "The Reynolds number is the ratio of inertial to viscous forces."},
    {"id": "d7", "topic": "superconductivity",
     "text": "Superconductors have zero resistance below a critical temperature. "
             "BCS theory explains conventional superconductivity via Cooper pairs."},
]
# Add distractors: docs with unrelated content.
for i in range(12):
    DOCS.append({
        "id": f"dx{i}",
        "topic": "noise",
        "text": f"Filler document {i} about generic physics, chemistry, and biology topics. "
                "Nothing relevant to any specific query. Padding. Padding. Padding.",
    })
print(f"corpus={len(DOCS)} docs")

QUERIES = [
    {"id": "q0", "text": "what absorbs light in plants",                               "topic": "photosynthesis"},
    {"id": "q1", "text": "where is atp produced",                                      "topic": "mitochondria"},
    {"id": "q2", "text": "what opens when a neuron fires",                             "topic": "neuron_action_potential"},
    {"id": "q3", "text": "what is half-maximal velocity substrate concentration",      "topic": "enzyme_kinetics"},
    {"id": "q4", "text": "how does cas9 locate its target",                             "topic": "crispr"},
    {"id": "q5", "text": "what boundary surrounds a black hole",                       "topic": "black_holes"},
    {"id": "q6", "text": "what ratio describes fluid flow regime",                     "topic": "turbulence"},
    {"id": "q7", "text": "what kind of pairs form in superconductors",                 "topic": "superconductivity"},
]


def recall_at_k(ranking_per_q: dict[str, list[str]], k: int = 3) -> float:
    hits = 0
    for q in QUERIES:
        top = ranking_per_q[q["id"]][:k]
        for doc_id in top:
            doc = next(d for d in DOCS if d["id"] == doc_id)
            if doc["topic"] == q["topic"]:
                hits += 1
                break
    return hits / len(QUERIES)


doc_embs = emb.encode([d["text"] for d in DOCS])


def retrieve(query_text: str, k: int = 3) -> list[str]:
    q = emb.encode([query_text])[0]
    scores = doc_embs @ q
    return [DOCS[int(i)]["id"] for i in np.argsort(-scores)[:k]]

Baseline: retrieve with the query as-is#

base_ranking = {q["id"]: retrieve(q["text"], k=10) for q in QUERIES}
base_recall = recall_at_k(base_ranking, k=3)
print(f"baseline Recall@3 = {base_recall:.3f}")

HyDE stub#

We supply a stubbed hypothetical_answer that returns a short keyword-rich passage for each query topic (what a real LLM would generate). Then we embed and retrieve.

HYPOTHETICAL = {
    "photosynthesis":       "Plants capture light with chlorophyll pigments in chloroplasts and use it to fix carbon dioxide.",
    "mitochondria":         "Mitochondria are organelles that synthesise adenosine triphosphate via the oxidative phosphorylation pathway in the inner membrane.",
    "neuron_action_potential": "Voltage-gated sodium channels open during a neuron action potential; saltatory conduction along myelinated axons accelerates the signal.",
    "enzyme_kinetics":      "Km is the substrate concentration at which enzyme velocity reaches half of Vmax, as described by the Michaelis-Menten equation.",
    "crispr":               "CRISPR Cas9 enzyme uses a guide RNA to bind and cut a DNA target flanked by the PAM protospacer adjacent motif.",
    "black_holes":          "The event horizon is the boundary around a black hole beyond which light cannot escape; Hawking radiation predicts slow evaporation.",
    "turbulence":           "The Reynolds number compares inertial and viscous forces and sets the onset of turbulent flow.",
    "superconductivity":    "Below the critical temperature, electrons form Cooper pairs that carry charge without resistance, per BCS theory.",
}


def hypothetical_answer(query_topic: str) -> str:
    return HYPOTHETICAL.get(query_topic, "")


hyde_ranking = {}
for q in QUERIES:
    hyp = hypothetical_answer(q["topic"])
    hyde_ranking[q["id"]] = retrieve(hyp, k=10)

hyde_recall = recall_at_k(hyde_ranking, k=3)
print(f"HyDE     Recall@3 = {hyde_recall:.3f}")

Multi-query + RRF#

Expand each query into 3 paraphrases, retrieve with each, fuse with Reciprocal Rank Fusion.

PARAPHRASES = {
    "q0": ["what absorbs light in plants",
           "plant pigment that captures sunlight",
           "chloroplast light absorbing molecule"],
    "q1": ["where is atp produced",
           "which cellular organelle synthesises atp",
           "site of oxidative phosphorylation"],
    "q2": ["what opens when a neuron fires",
           "which ion channels activate during action potential",
           "sodium gated channel neuron depolarisation"],
    "q3": ["what is half-maximal velocity substrate concentration",
           "km michaelis menten meaning",
           "substrate concentration at half vmax"],
    "q4": ["how does cas9 locate its target",
           "crispr guide rna target recognition",
           "pam motif recognition cas9"],
    "q5": ["what boundary surrounds a black hole",
           "name of the point of no return near a black hole",
           "event horizon definition"],
    "q6": ["what ratio describes fluid flow regime",
           "reynolds number meaning",
           "inertial to viscous force ratio fluid"],
    "q7": ["what kind of pairs form in superconductors",
           "cooper pair electrons superconductor",
           "bcs theory paired electrons"],
}


def rrf(rankings: list[list[str]], k: int = 60) -> list[str]:
    scores: dict[str, float] = defaultdict(float)
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking, start=1):
            scores[doc_id] += 1.0 / (k + rank)
    return sorted(scores, key=lambda d: -scores[d])


mq_ranking = {}
for q in QUERIES:
    ranks = [retrieve(p, k=10) for p in PARAPHRASES[q["id"]]]
    mq_ranking[q["id"]] = rrf(ranks)

mq_recall = recall_at_k(mq_ranking, k=3)
print(f"multi-query RRF Recall@3 = {mq_recall:.3f}")

s.check(
    "hyde_raises_recall_above_baseline",
    lambda: hyde_recall >= base_recall,
    msg=f"baseline={base_recall:.3f}  hyde={hyde_recall:.3f}",
)
s.check(
    "multi_query_raises_recall_above_baseline",
    lambda: mq_recall >= base_recall,
    msg=f"baseline={base_recall:.3f}  multi-q={mq_recall:.3f}",
)
s.check(
    "hyde_at_least_matches_multi_query",
    lambda: hyde_recall >= mq_recall - 0.05,
    msg=f"hyde={hyde_recall:.3f}  multi-q={mq_recall:.3f}",
)
s.check(
    "baseline_is_nontrivial",
    lambda: base_recall >= 0.5,
    msg=f"baseline = {base_recall:.3f}",
)

Where does HyDE help most?#

HyDE’s edge is sharpest on short queries: the hypothetical answer injects domain vocabulary that the raw query is missing. Bucket the queries by token length (short vs long) and plot per-bucket Recall@3 for baseline, HyDE, and multi-query RRF. The short bucket is where rewriting carries its weight.

import matplotlib.pyplot as plt

def hit(ranking, qid, topic, k=3) -> int:
    for doc_id in ranking[qid][:k]:
        if next(d for d in DOCS if d["id"] == doc_id)["topic"] == topic:
            return 1
    return 0

q_lens = {q["id"]: len(tokenize(q["text"])) for q in QUERIES}
med = float(np.median(list(q_lens.values())))
buckets = {"short (<=median)": [q for q in QUERIES if q_lens[q["id"]] <= med],
           "long (>median)":   [q for q in QUERIES if q_lens[q["id"]] >  med]}

def mr(r, qs):
    return float(np.mean([hit(r, q["id"], q["topic"]) for q in qs])) if qs else 0.0

names   = ["baseline", "HyDE", "multi-query RRF"]
vals    = {n: [mr(r, qs) for qs in buckets.values()]
           for n, r in zip(names, [base_ranking, hyde_ranking, mq_ranking])}

fig, ax = plt.subplots(figsize=(6.8, 3.6))
x = np.arange(len(buckets)); w = 0.26
for i, n in enumerate(names):
    ax.bar(x + (i - 1) * w, vals[n], width=w, label=n)
    for j, v in enumerate(vals[n]):
        ax.text(j + (i - 1) * w, v + 0.02, f"{v:.2f}", ha="center", fontsize=8)
ax.set_xticks(x); ax.set_xticklabels([f"{k}\nn={len(v)}" for k, v in buckets.items()])
ax.set_ylabel("Recall@3"); ax.set_ylim(0, 1.05)
ax.set_title(f"rewriting gain by query length (median={med:.0f})")
ax.legend(fontsize=9); fig.tight_layout(); plt.show()

Exercises#

  1. Real LLM for hypothetical answers. Swap hypothetical_answer for a call to a local Qwen2.5-0.5B-Instruct via vLLM. Temperature 0.7, max 120 tokens. Pass the generation straight into retrieve.

  2. Decomposition. Build a multi-hop benchmark where each query needs two independent retrievals to answer. Expand each query into two sub-questions via hypothetical_answer, retrieve each, union.

  3. HyDE at lower temperature. Generate 5 hypotheticals at temperature 0.3, retrieve with each, fuse via RRF. Higher diversity sometimes beats HyDE-1.

References#

  • Gao et al. 2022 HyDE paper.

  • LangChain’s MultiQueryRetriever for the RRF-based multi-query implementation this section mirrors.

s.summary()
s.save()