lm-eval-harness and Inspect AI

lm-eval-harness and Inspect AI#

Track 06 - Evaluation · Notebook 08 · Runtime: ≈30 s on CPU

Prerequisites: 06_eval/02 (MMLU harness + calibration).

References:

EleutherAI lm-eval-harness: EleutherAI/lm-evaluation-harness

Inspect AI (UK AISI): https://inspect.aisi.org.uk/

The two frameworks every serious eval pipeline uses today. We won’t install them in this notebook; instead we reproduce the shape of each framework’s output and show how to reconcile disagreements across them.

What#

An evaluation number is only as good as the framework that produced it. Two different harnesses reporting the same task (say MMLU) on the same model will often disagree by 1-3 percentage points because of small prompt-template, tokenisation, and scoring differences. This notebook:

Defines a tiny task (a 20-question multiple-choice benchmark).
Simulates two framework runs with subtly different scoring decisions (different prompt templates, length normalisation on/off).
Measures the gap and decomposes it: how much comes from template, how much from length normalisation, how much from random tie-breaking.

from llm_systems_cookbook.nb import bootstrap

import numpy as np

s = bootstrap("06_eval_08_lm_eval_inspect_ai")

A 20-question synthetic task#

Each question has four choices. Ground truth is recorded. The “model” under test is deterministic: its logits depend only on the question id, the framework’s prompt template (different templates produce different logit shifts), and whether length normalisation is applied.

K = 4
N = 20
rng = np.random.default_rng(0)

# Ground truth + true difficulty per question. The model is simulated
# as "moderately strong": the correct option gets a +1.5-nat boost on
# top of baseline noise, which gives ~70% bare accuracy before any
# framework-specific shifts.
truth = rng.integers(0, K, size=N)
question_strength = rng.normal(0, 0.5, size=(N, K))
for i in range(N):
    question_strength[i, truth[i]] += 1.5
# Option-text lengths (in tokens) differ across options; length
# normalisation divides log-prob by this.
option_lengths = rng.integers(3, 20, size=(N, K))


def framework_logits(template_shift: np.ndarray) -> np.ndarray:
    '''Return model logits shaped (N, K) given a per-option template shift
    added to the intrinsic question_strength. Different frameworks use
    different prompt templates -> different shifts.'''
    return question_strength + template_shift


def score_task(
    logits: np.ndarray, *, length_normalise: bool, lengths: np.ndarray
) -> float:
    '''Return accuracy given logits and a scoring decision.'''
    if length_normalise:
        logits = logits / lengths
    preds = logits.argmax(axis=-1)
    return float((preds == truth).mean())


# Two framework templates: lm-eval-harness uses a bracketed "A." prefix,
# Inspect AI uses "Answer: A". The difference in tokenisation slightly
# shifts per-option logits. Model this as a small per-option bias.
LM_EVAL_SHIFT  = rng.normal(0, 0.15, size=(N, K))
INSPECT_SHIFT  = rng.normal(0, 0.15, size=(N, K))

lm_logits = framework_logits(LM_EVAL_SHIFT)
in_logits = framework_logits(INSPECT_SHIFT)
print(f"lm-eval template shift (RMS)  = {LM_EVAL_SHIFT.std():.3f}")
print(f"inspect template shift (RMS)  = {INSPECT_SHIFT.std():.3f}")

Two frameworks, four scoring configurations#

Same logits × on/off length normalisation × either framework’s template. We’ll produce all four numbers.

acc = {
    "lm_eval_lenorm_off":    score_task(lm_logits, length_normalise=False, lengths=option_lengths),
    "lm_eval_lenorm_on":     score_task(lm_logits, length_normalise=True,  lengths=option_lengths),
    "inspect_lenorm_off":    score_task(in_logits, length_normalise=False, lengths=option_lengths),
    "inspect_lenorm_on":     score_task(in_logits, length_normalise=True,  lengths=option_lengths),
}
for k, v in acc.items():
    print(f"  {k:<22}  accuracy = {v:.3f}")

gap_template = abs(acc["lm_eval_lenorm_off"] - acc["inspect_lenorm_off"])
gap_lenorm   = abs(acc["lm_eval_lenorm_off"] - acc["lm_eval_lenorm_on"])
gap_full     = abs(acc["lm_eval_lenorm_off"] - acc["inspect_lenorm_on"])
print(f"\ntemplate-only gap     = {gap_template:.3f}")
print(f"lenorm-only gap       = {gap_lenorm:.3f}")
print(f"combined (worst-case) = {gap_full:.3f}")

s.check(
    "all_accuracies_above_random",
    lambda: all(v > 1.0 / K for v in acc.values()),
    msg=f"{acc}",
)
s.check(
    "framework_gap_nonzero",
    lambda: gap_template > 0 or gap_lenorm > 0,
    msg=f"template={gap_template:.3f}  lenorm={gap_lenorm:.3f}",
)
s.check(
    "combined_gap_bounded_by_sum",
    lambda: gap_full <= gap_template + gap_lenorm + 1e-9,
    msg=f"combined={gap_full:.3f}  sum={gap_template + gap_lenorm:.3f}",
)
# Reconciliation: if you fix template + lenorm to the same choice, both
# frameworks must produce the same number (because the only remaining
# difference is the template shift which both accept).
# Here our "fix both knobs" protocol is to force ``length_normalise=False``
# on the same template - both frameworks with LM_EVAL template.
forced = score_task(lm_logits, length_normalise=False, lengths=option_lengths)
s.check(
    "reconciled_frameworks_agree",
    lambda: forced == acc["lm_eval_lenorm_off"],
    msg=f"forced={forced:.3f}  baseline={acc['lm_eval_lenorm_off']:.3f}",
)

Cross-harness agreement#

The meaningful picture for reconciliation is a scatter of the two harnesses’ scores. We bootstrap sub-task accuracies (random five- question subsets, scored by each framework) and plot one point per sub-task. Points on the y = x diagonal are perfect agreement; vertical spread is the framework gap. Template + length-normalisation choices determine how tightly the cloud hugs the line.

import matplotlib.pyplot as plt

rng_plot = np.random.default_rng(1)
n_subtasks, subtask_size = 40, 5

def subtask_acc(logits, idx, *, length_normalise):
    L = logits[idx] / option_lengths[idx] if length_normalise else logits[idx]
    return float((L.argmax(axis=-1) == truth[idx]).mean())

pts = []
for _ in range(n_subtasks):
    idx = rng_plot.choice(N, size=subtask_size, replace=False)
    for lenorm in (False, True):
        pts.append((subtask_acc(lm_logits, idx, length_normalise=lenorm),
                    subtask_acc(in_logits, idx, length_normalise=lenorm),
                    lenorm))
pts = np.array(pts, dtype=float)
corr = float(np.corrcoef(pts[:, 0], pts[:, 1])[0, 1])

fig, ax = plt.subplots(figsize=(5.8, 5.4))
ax.plot([0, 1], [0, 1], "k--", alpha=0.5, label="perfect agreement")
off = pts[pts[:, 2] == 0]; on = pts[pts[:, 2] == 1]
ax.scatter(off[:, 0], off[:, 1], color="tab:blue", alpha=0.7, label="lenorm off")
ax.scatter(on[:, 0],  on[:, 1],  color="tab:red",  alpha=0.7, marker="s", label="lenorm on")
ax.set_xlim(0, 1); ax.set_ylim(0, 1)
ax.set_xlabel("lm-eval-harness sub-task accuracy")
ax.set_ylabel("Inspect AI sub-task accuracy")
ax.set_title(f"cross-harness agreement  (Pearson r = {corr:.3f}, gap = {gap_full:.3f})")
ax.legend(loc="lower right", fontsize=9); ax.grid(alpha=0.3)
fig.tight_layout(); plt.show()

Exercises#

Run the real thing. pip install lm-eval and execute the MMLU task on a small model. Compare the printed accuracy to a manual rerun of the same prompts via transformers. Reconcile.
Inspect AI tasks. Install inspect_ai and convert a few MMLU questions into an Inspect task definition. Observe where each framework reports numbers differently.
Agreement heatmap. Pick six open models and run both frameworks on MMLU; compute pairwise framework-level correlation per model and plot a heatmap. High correlation = rankings are robust to framework choice.

References#

lm-eval-harness README.md §”Reproducing Published Results”.
Inspect AI docs on task authoring and the evaluation protocol.
Biderman et al. 2024 for empirical evidence that MMLU numbers vary by framework by up to 5 points on the same model.

s.summary()
s.save()