Contamination detection

Open in Colab ▶️ Run this notebook in Colab

Contamination detection#

Track 06 - Evaluation · Notebook 07 · Runtime: ≈30 s on CPU

Prerequisites: 06_eval/01 (perplexity).

Papers:

  • Shi et al. 2023, Detecting Pretraining Data from Large Language Models (2310.16789) - Min-K Prob.

  • Oren et al. 2023, Proving Test Set Contamination in Black Box Language Models (2310.17623).

What#

If a benchmark question is in the model’s training data, the model’s performance on it is meaningless - it’s memorisation, not generalisation. Two standard detectors:

  1. N-gram overlap. Does a test example share long n-gram spans with the training corpus? Fast, deterministic, but only catches verbatim leaks.

  2. Min-K Prob (Shi et al.). Under the model, contaminated text has a lower bottom-K fraction of token log-probs than non-contaminated text (the model is confident on every token). We simulate the per-token log-prob distribution and reproduce the separation.

from llm_systems_cookbook.nb import bootstrap

import numpy as np

s = bootstrap("06_eval_07_contamination_detection")

Part 1 - N-gram overlap#

Build a set of training-corpus n-grams, then check what fraction of a test example’s n-grams is also in that set. Above some threshold ⇒ likely contamination.

n = 13 is a common choice - long enough that random coincidences are negligible, short enough to catch paraphrased leaks.

def ngrams(text: str, n: int) -> set[tuple[str, ...]]:
    tokens = text.lower().split()
    return {tuple(tokens[i : i + n]) for i in range(len(tokens) - n + 1)}


def overlap_score(test: str, train_ngrams: set[tuple[str, ...]], n: int) -> float:
    test_ngrams = ngrams(test, n)
    if not test_ngrams:
        return 0.0
    return sum(1 for g in test_ngrams if g in train_ngrams) / len(test_ngrams)


# Synthetic "training" corpus.
TRAIN_DOCS = [
    "The mitochondrion is the powerhouse of the cell, producing ATP via oxidative phosphorylation.",
    "Chlorophyll absorbs red and blue wavelengths of light and reflects green, which is why plants appear green.",
    "The Reynolds number characterises the ratio of inertial forces to viscous forces in fluid flow.",
    "CRISPR-Cas9 uses a guide RNA to direct the nuclease to a complementary DNA sequence flanked by a PAM motif.",
    "Subduction zones are destructive plate boundaries where oceanic crust is recycled into the mantle.",
] * 4  # repetition to make n-grams real.

N = 7  # use 7-grams for a small corpus; 13 is industry standard on big ones.
TRAIN_NGRAMS = set().union(*(ngrams(d, N) for d in TRAIN_DOCS))
print(f"train corpus: {len(TRAIN_DOCS)} docs,  {len(TRAIN_NGRAMS)} unique {N}-grams")

# Two test examples:
contaminated = "The mitochondrion is the powerhouse of the cell, producing ATP via oxidative phosphorylation."
clean        = "Quantum field theory unifies quantum mechanics with special relativity via the framework of gauge fields."

ov_contam = overlap_score(contaminated, TRAIN_NGRAMS, N)
ov_clean  = overlap_score(clean,        TRAIN_NGRAMS, N)
print(f"contaminated example overlap = {ov_contam:.2%}")
print(f"clean        example overlap = {ov_clean:.2%}")

s.check(
    "contaminated_has_high_ngram_overlap",
    lambda: ov_contam >= 0.95,
    msg=f"overlap = {ov_contam:.2%}",
)
s.check(
    "clean_has_zero_ngram_overlap",
    lambda: ov_clean == 0.0,
    msg=f"overlap = {ov_clean:.2%}",
)

Part 2 - Min-K Prob#

For a text of length T, the model outputs T token log-probs (log p_1, ..., log p_T). Min-K Prob takes the mean of the bottom K% of those log-probs. The intuition: contaminated text has no “surprising” tokens (the model has seen them all before), so even its worst log-prob is above average. Non-contaminated text has a heavy lower tail.

We simulate per-token log-probs:

  • Contaminated: high mean log-prob (-0.5), tight distribution.

  • Non-contaminated: lower mean (-2.0), wider distribution (more tokens in the tail).

def min_k_prob(log_probs: np.ndarray, k_frac: float = 0.20) -> float:
    '''Mean of the bottom ``k_frac`` fraction of log-probs.'''
    k = max(1, int(round(len(log_probs) * k_frac)))
    sorted_lp = np.sort(log_probs)
    return float(sorted_lp[:k].mean())


rng = np.random.default_rng(0)
T = 128
n_contam = 60
n_clean = 60

contam_logprobs = [rng.normal(-0.5, 0.3, T) for _ in range(n_contam)]
clean_logprobs  = [rng.normal(-2.0, 0.8, T) for _ in range(n_clean)]

contam_scores = np.array([min_k_prob(lp) for lp in contam_logprobs])
clean_scores  = np.array([min_k_prob(lp) for lp in clean_logprobs])

print(f"contaminated   Min-K mean = {contam_scores.mean():.3f}   std = {contam_scores.std():.3f}")
print(f"non-contaminated           = {clean_scores.mean():.3f}   std = {clean_scores.std():.3f}")

AUROC between the two distributions#

The cleanest summary is the area under the ROC curve: probability that a randomly chosen contaminated example has higher Min-K score than a random clean one. AUROC = 1.0 means perfect separation, AUROC = 0.5 means no signal.

def roc_auc(pos: np.ndarray, neg: np.ndarray) -> float:
    '''Probability that a random pos > random neg (Mann-Whitney U / (n_pos*n_neg)).'''
    n_pos = len(pos)
    n_neg = len(neg)
    total = 0
    for p in pos:
        total += (neg < p).sum() + 0.5 * (neg == p).sum()
    return float(total) / (n_pos * n_neg)


auroc = roc_auc(contam_scores, clean_scores)
print(f"AUROC (contaminated vs clean on Min-K) = {auroc:.3f}")

# A threshold at the midpoint of the two means gives a concrete
# detector; report its accuracy.
threshold = (contam_scores.mean() + clean_scores.mean()) / 2
preds = np.concatenate([contam_scores > threshold, clean_scores > threshold])
labels = np.concatenate([np.ones_like(contam_scores), np.zeros_like(clean_scores)])
acc = (preds == labels).mean()
print(f"midpoint-threshold detector accuracy = {acc:.3f}")

s.check("mink_auroc_high",  lambda: auroc > 0.95, msg=f"AUROC = {auroc:.3f}")
s.check("mink_detector_accuracy_high", lambda: acc > 0.90, msg=f"acc = {acc:.3f}")
s.check(
    "contaminated_scores_higher_than_clean",
    lambda: contam_scores.mean() > clean_scores.mean(),
    msg=f"contam mean={contam_scores.mean():.3f}  clean mean={clean_scores.mean():.3f}",
)

Score distribution#

A histogram of the Min-K scores with the two classes overlaid is the cleanest picture of what a membership-inference detector is doing. The separation between the contaminated and clean modes is exactly the quantity the AUROC number summarises; the midpoint threshold used as a concrete detector is drawn in as a vertical line.

import matplotlib.pyplot as plt

lo = min(contam_scores.min(), clean_scores.min())
hi = max(contam_scores.max(), clean_scores.max())
bins = np.linspace(lo, hi, 25)

fig, ax = plt.subplots(figsize=(7.0, 4.0))
ax.hist(clean_scores,  bins=bins, alpha=0.6, color="tab:blue",
        label=f"clean  (n={len(clean_scores)})")
ax.hist(contam_scores, bins=bins, alpha=0.6, color="tab:red",
        label=f"contaminated  (n={len(contam_scores)})")
ax.axvline(threshold, color="black", linestyle="--",
           label=f"midpoint threshold = {threshold:.2f}")
ax.set_xlabel("Min-K log-prob score  (higher = more likely contaminated)")
ax.set_ylabel("count")
ax.set_title(f"membership-inference score distribution  (AUROC = {auroc:.3f}, acc = {acc:.3f})")
ax.legend(loc="upper left", fontsize=9)
ax.grid(alpha=0.3)
fig.tight_layout()
plt.show()

Exercises#

  1. Combine the two signals: use n-gram overlap as a fast filter and fall back to Min-K Prob for examples that don’t exact-match. Report combined AUROC on a mixed test set.

  2. Shi et al. test multiple k_frac values (0.1, 0.2, 0.3). Try them all and report the best.

  3. Plot the full ROC curve (TPR vs FPR) for Min-K Prob. The steepness near the origin tells you how easy it is to hit a low false-positive operating point.

References#

  • Shi et al. 2023 for Min-K Prob, including a refinement called Min-K%++ in a follow-up.

  • Magar & Schwartz 2022, Data Contamination: From Memorization to Exploitation, for the taxonomy of contamination types.

s.summary()
s.save()