RAGAS - evaluating a RAG pipeline

Open in Colab ▶️ Run this notebook in Colab

RAGAS - evaluating a RAG pipeline#

Track 02 - RAG · Notebook 09 · Runtime: ≈30 s on CPU

Prerequisites: any of the retrieval notebooks in this track.

Paper: Es et al. 2023, RAGAS: Automated Evaluation of Retrieval Augmented Generation (2309.15217).

What#

Four from-scratch metrics that capture how well a RAG pipeline does what it claims to do:

  • faithfulness: every claim in the answer is supported by the retrieved context. Measured by splitting the answer into atomic claims and checking each claim appears verbatim-or-paraphrased in the context.

  • answer_relevancy: the answer addresses the question. Measured by embedding the question and the answer and computing cosine similarity.

  • context_precision: fraction of retrieved chunks that are actually relevant.

  • context_recall: fraction of the reference answer’s key tokens that are present in the retrieved chunks.

We implement these with a deterministic LLM stub (fact-checking via token overlap, embeddings via a hash embedder) and compare a “good pipeline” (retrieves the right context) against a “bad pipeline” (retrieves random noise).

from llm_systems_cookbook.nb import bootstrap

import re
from collections import Counter

import numpy as np

s = bootstrap("02_rag_09_ragas_evaluation")

Questions, references, and two answer pipelines#

8 short factoid questions. For each we have:

  • reference_answer - the ground-truth answer.

  • good_context - passages that actually contain the answer.

  • bad_context - random passages from other questions.

  • good_answer - a synthesised answer drawn from good_context.

  • bad_answer - the correct answer with an added hallucinated claim.

RAGAS on the good pipeline should give high faithfulness and context precision; on the bad pipeline it should flag the hallucination and the low-quality context.

QA = [
    {"q": "where is atp produced in the cell",
     "ref": "mitochondria produce atp via oxidative phosphorylation",
     "good_ctx": ["mitochondria generate atp through oxidative phosphorylation",
                  "the inner mitochondrial membrane folds into crista"],
     "bad_ctx":  ["chlorophyll absorbs red and blue wavelengths"]},
    {"q": "what does the reynolds number measure",
     "ref": "reynolds number is the ratio of inertial to viscous forces",
     "good_ctx": ["the reynolds number characterises the ratio of inertial forces to viscous forces in fluid flow"],
     "bad_ctx":  ["cas9 uses a guide rna to find dna"]},
    {"q": "what is the pam motif",
     "ref": "pam is a short dna sequence flanking the cas9 target",
     "good_ctx": ["a pam motif must flank the target sequence for cas9 to bind"],
     "bad_ctx":  ["photosystem ii splits water"]},
    {"q": "what absorbs light in plants",
     "ref": "chlorophyll absorbs red and blue light wavelengths",
     "good_ctx": ["chlorophyll absorbs red and blue wavelengths of light"],
     "bad_ctx":  ["mitochondria have crista folds"]},
    {"q": "where is new oceanic crust formed",
     "ref": "at mid-ocean ridges, divergent boundaries",
     "good_ctx": ["mid-ocean ridges are divergent boundaries where new crust forms"],
     "bad_ctx":  ["saltatory conduction along myelin accelerates neurons"]},
    {"q": "what starts a neuron action potential",
     "ref": "voltage-gated sodium channels open when the membrane depolarises past threshold",
     "good_ctx": ["a neuron action potential depends on voltage-gated sodium channels",
                  "when the membrane depolarises past threshold sodium channels open"],
     "bad_ctx":  ["rubisco is the enzyme responsible for carbon fixation"]},
    {"q": "what is km in enzyme kinetics",
     "ref": "km is the substrate concentration at half-maximal velocity",
     "good_ctx": ["km is the substrate concentration at which reaction velocity is half of vmax"],
     "bad_ctx":  ["black holes have event horizons"]},
    {"q": "what is an event horizon",
     "ref": "the boundary around a black hole beyond which nothing escapes",
     "good_ctx": ["the event horizon is the boundary beyond which escape becomes impossible"],
     "bad_ctx":  ["the calvin cycle fixes carbon dioxide"]},
]


def tokenize(text: str) -> list[str]:
    return re.findall(r"[a-z0-9]+", text.lower())


def good_answer(q: dict) -> str:
    return q["ref"]  # straight from the reference


def bad_answer(q: dict) -> str:
    # Correct answer + hallucinated claim that isn't in context.
    hallucination = "this was confirmed in a 2019 nature paper by smith and jones"
    return q["ref"] + ". " + hallucination


print(f"{len(QA)} questions")

Faithfulness#

Split the answer into atomic claims (sentences). Each claim is supported if ≥ 60 % of its content tokens appear in the context. The fraction of supported claims is the faithfulness score.

_STOP = {"the", "a", "an", "is", "are", "was", "in", "of", "to", "for", "with", "by",
         "on", "at", "this", "that", "it", "be", "and", "or", "as", "which", "from"}


def claims(answer: str) -> list[str]:
    return [c.strip() for c in re.split(r"(?<=[.!?])\s+", answer) if c.strip()]


def content_tokens(text: str) -> set[str]:
    return {t for t in tokenize(text) if t not in _STOP and len(t) > 1}


def faithfulness(answer: str, contexts: list[str]) -> float:
    ctx_tokens = set().union(*(content_tokens(c) for c in contexts)) if contexts else set()
    cls = claims(answer)
    if not cls:
        return 0.0
    supported = 0
    for cl in cls:
        cl_tokens = content_tokens(cl)
        if not cl_tokens:
            continue
        if len(cl_tokens & ctx_tokens) / len(cl_tokens) >= 0.6:
            supported += 1
    return supported / max(len(cls), 1)


good_f = float(np.mean([faithfulness(good_answer(q), q["good_ctx"]) for q in QA]))
bad_f  = float(np.mean([faithfulness(bad_answer(q), q["good_ctx"]) for q in QA]))
print(f"faithfulness  good={good_f:.3f}   bad={bad_f:.3f}")

Answer relevancy#

Embed the question and the answer with the same hash embedder used elsewhere. Cosine similarity = relevancy. A high relevancy with low faithfulness is the classic “hallucinated answer that sounds on topic.”

import hashlib


def _stable_hash(*parts: str) -> int:
    h = hashlib.md5("|".join(parts).encode("utf-8")).digest()
    return int.from_bytes(h[:8], "big")


def hash_embed(text: str, dim: int = 256) -> np.ndarray:
    v = np.zeros(dim, dtype=np.float32)
    toks = tokenize(text)
    for w in toks:
        v[_stable_hash("u", w) % dim] += 1.0
    for i in range(len(toks) - 1):
        v[_stable_hash("b", toks[i], toks[i + 1]) % dim] += 0.5
    return v / (np.linalg.norm(v) + 1e-9)


def answer_relevancy(question: str, answer: str) -> float:
    return float(hash_embed(question) @ hash_embed(answer))


good_r = float(np.mean([answer_relevancy(q["q"], good_answer(q)) for q in QA]))
bad_r  = float(np.mean([answer_relevancy(q["q"], bad_answer(q)) for q in QA]))
print(f"answer_relevancy  good={good_r:.3f}   bad={bad_r:.3f}")

Context precision and recall#

  • Precision: fraction of retrieved chunks that share significant content with the reference answer.

  • Recall: fraction of the reference’s content tokens that appear in any retrieved chunk.

def context_precision(reference: str, contexts: list[str], thresh: float = 0.3) -> float:
    if not contexts:
        return 0.0
    ref_tokens = content_tokens(reference)
    if not ref_tokens:
        return 0.0
    hits = 0
    for c in contexts:
        overlap = len(content_tokens(c) & ref_tokens) / max(len(ref_tokens), 1)
        if overlap >= thresh:
            hits += 1
    return hits / len(contexts)


def context_recall(reference: str, contexts: list[str]) -> float:
    ref_tokens = content_tokens(reference)
    if not ref_tokens:
        return 0.0
    ctx_tokens = set().union(*(content_tokens(c) for c in contexts)) if contexts else set()
    return len(ref_tokens & ctx_tokens) / len(ref_tokens)


good_p = float(np.mean([context_precision(q["ref"], q["good_ctx"]) for q in QA]))
bad_p  = float(np.mean([context_precision(q["ref"], q["bad_ctx"])  for q in QA]))
good_rc = float(np.mean([context_recall(q["ref"], q["good_ctx"]) for q in QA]))
bad_rc  = float(np.mean([context_recall(q["ref"], q["bad_ctx"])  for q in QA]))

print(f"context_precision  good={good_p:.3f}   bad={bad_p:.3f}")
print(f"context_recall     good={good_rc:.3f}   bad={bad_rc:.3f}")

s.check("good_faithfulness_high", lambda: good_f >= 0.70, msg=f"good faithfulness = {good_f:.3f}")
s.check(
    "bad_faithfulness_lower_than_good",
    lambda: bad_f < good_f,
    msg=f"good={good_f:.3f}  bad={bad_f:.3f}",
)
s.check("good_context_precision_high", lambda: good_p >= 0.80, msg=f"{good_p:.3f}")
s.check(
    "bad_context_precision_lower",
    lambda: bad_p < good_p,
    msg=f"good={good_p:.3f}  bad={bad_p:.3f}",
)
s.check(
    "good_context_recall_above_half",
    lambda: good_rc > 0.5,
    msg=f"{good_rc:.3f}",
)
s.check(
    "answer_relevancy_positive",
    lambda: good_r > 0 and bad_r > 0,
    msg=f"good={good_r:.3f}  bad={bad_r:.3f}",
)

Four metrics, two pipelines#

All four RAGAS scores as grouped bars. The good pipeline scores high across the board; the bad pipeline’s faithfulness and context precision collapse, but its answer relevancy stays nearly as high - the hallucination signature that relevancy alone can’t detect.

import matplotlib.pyplot as plt

metrics = ["faithfulness", "answer_relevancy", "context_precision", "context_recall"]
good_scores = [good_f, good_r, good_p, good_rc]
bad_scores  = [bad_f,  bad_r,  bad_p,  bad_rc]

fig, ax = plt.subplots(figsize=(8, 3.8))
x = np.arange(len(metrics)); w = 0.38
ax.bar(x - w/2, good_scores, width=w, color="tab:green", label="good pipeline")
ax.bar(x + w/2, bad_scores,  width=w, color="tab:red",   label="bad pipeline")
for i, (g, b) in enumerate(zip(good_scores, bad_scores)):
    ax.text(i - w/2, g + 0.02, f"{g:.2f}", ha="center", fontsize=8)
    ax.text(i + w/2, b + 0.02, f"{b:.2f}", ha="center", fontsize=8)
ax.set_xticks(x); ax.set_xticklabels(metrics, rotation=15, fontsize=9)
ax.set_ylabel("score"); ax.set_ylim(0, 1.05)
ax.set_title("RAGAS metrics: good vs bad pipeline")
ax.axhline(0.5, color="gray", linestyle=":", alpha=0.5)
ax.legend(fontsize=9)
fig.tight_layout(); plt.show()

Exercises#

  1. Real RAGAS. pip install ragas==0.2.14, load a small QA dataset, compute the four metrics with the real library, compare against our stubs.

  2. LLM judge for faithfulness. Replace the token-overlap heuristic with a Qwen2.5-0.5B-Instruct prompt: “Given this context, is claim X supported? yes/no”. Classification accuracy vs our heuristic tells you the ceiling RAGAS would have with a real judge.

  3. Stdev reporting. Each metric’s mean should always be reported with standard deviation across runs. Bootstrap 500 subsamples and report 95% CI bars.

References#

  • Es et al. 2023 RAGAS paper.

  • The ragas package source for production metric implementations.

s.summary()
s.save()