Evaluation

Evaluation#

What you’ll learn in this part

  • Perplexity three ways (sliding window / non-overlapping / single pass) and bits-per-character as a Shannon interpretation.

  • Multiple-choice scoring with calibration and Expected Calibration Error.

  • Chen et al. 2021 unbiased pass@k estimator for code benchmarks.

  • Position, verbosity, and self-enhancement bias in LLM-as-judge setups; position-swap mitigation.

  • Arena-style ratings via online Elo and Bradley-Terry.

  • Long-context stress tests (NIAH, RULER composite).

  • Contamination detection (n-gram overlap, Min-K Prob).

  • Reconciling numbers across lm-eval and Inspect AI.

Key terms used in this part#

  • perplexity is the core language-modeling metric.

  • calibration asks whether confidence estimates match empirical correctness.

  • Expected Calibration Error (ECE) summarizes calibration gap across confidence bins.

  • pass@k is the standard code-generation success metric.

  • LLM-as-judge, Elo, and Bradley-Terry are core concepts for preference-based evaluation.

  • GPQA and HLE are the 2025 frontier benchmarks that replaced MMLU as the discriminating tests for top-tier models.

  • LiveCodeBench is a contamination-resistant alternative to HumanEval for coding evaluation.

  • ARC-AGI is a reasoning benchmark whose hard-problem score became a proxy for AGI progress in 2024-2025.

  • SWE-bench and its live-updatable successor SWE-bench Live measure real GitHub issue resolution; Terminal-Bench 2.0 (January 2026) extends agentic evaluation to multi-step CLI workflows.

Reading order#

No mandatory prerequisites. All notebooks are CPU-safe and written from scratch.

  1. 01_perplexity_from_scratch: the canonical metric derived and implemented three ways.

  2. 02_mmlu_harness_calibration: logit-based multiple-choice plus ECE.

  3. 03_humaneval_unbiased_pass_k: sandboxed candidate execution plus unbiased estimator.

  4. 04_llm_as_judge_bias: position and verbosity bias, quantified.

  5. 05_arena_elo_bradley_terry: pairwise preferences to rankings.

  6. 06_long_context_niah_ruler: decay model plus RULER composite.

  7. 07_contamination_detection: n-gram overlap plus Min-K Prob.

  8. 08_lm_eval_inspect_ai: cross-framework reconciliation on a synthetic task.

Benchmark landscape (mid-2026)#

The evaluation frontier has shifted since 2024. Several benchmarks that were difficult in 2024 are now saturated:

Benchmark

2024 SOTA

2026 SOTA

Status

MMLU

~86%

88-94%

Saturated

HumanEval

~95%

~99%

Saturated

GSM8K

~97%

~99%

Saturated

GPQA-Diamond

~70%

91-94%

Active frontier

ARC-AGI-2

~5%

~60-65%

Active frontier

AIME 2025

N/A

91-94%

Active frontier

SWE-bench Verified

~45%

~77-80%

Active frontier

SWE-bench Pro

N/A

80.3%

Active frontier

FrontierCode (Diamond)

N/A

29.3%

Active frontier

SWE-bench Pro and FrontierCode (Cognition’s hardest coding split) are the new discriminating benchmarks for frontier coding agents. The leading model on SWE-bench Pro leads the next-best by over 11 points. The Hebbia Finance Benchmark covers senior-level reasoning over financial documents and is gaining adoption as a proxy for document-understanding capability.

As of July 2, 2026, BenchLM.ai tracks 249 benchmarks across 272 models; 37% of the 154 percentage-scaled benchmarks with meaningful coverage are now saturated (top model scores ≥ 90%). GSM8K is effectively solved (99%), and HellaSwag exceeds 95%. This is the case for evergreen benchmark hygiene made concrete: a benchmark’s useful lifetime as a discriminator is shorter each generation, which is why 07_contamination_detection and the lm-eval/Inspect AI comparison in 08_lm_eval_inspect_ai both treat “is this benchmark still measuring anything” as a first-class question rather than an afterthought.

The notebooks in this part teach the mechanics of evaluation on un-saturated tasks. The numerical thresholds in the scoring checks target 2026-accessible open-weight models (Qwen2.5-0.5B, Phi-3.5-mini, SmolLM2-360M) that still show non-trivial variance on MMLU and HumanEval, which keeps them useful for teaching even as frontier models approach the ceiling.