DSPy 3 + MIPROv2 — optimise a real classifier

Open in Colab Run this notebook in Colab

DSPy 3 + MIPROv2 — optimise a real classifier#

Track 08 - Production · Notebook 07 · Runtime: ~3 min LIVE, <1s replay

A dspy.Predict classifier on 20-train / 20-test customer support tickets. Baseline is zero-shot. MIPROv2 picks an instruction and 4 bootstrapped demos by Bayesian search over a held-out trainset. Same DSPy program, two prompts, ~15-point accuracy lift.

from llm_systems_cookbook.nb import bootstrap
from llm_systems_cookbook._utils import repo_root

import json
import os
from pathlib import Path

s = bootstrap("08_production_07_dspy_miprov2_optimizer")

try:
    import dspy  # noqa: F401
    HAS_DSPY = True
except ImportError:
    HAS_DSPY = False

LIVE = HAS_DSPY and bool(os.environ.get("ANTHROPIC_API_KEY")) and bool(os.environ.get("DSPY_LIVE"))
FIXTURE = json.loads((Path(repo_root()) / "notebooks/08_production/_fixtures/07_dspy.json").read_text())
print(f"mode={'LIVE' if LIVE else 'REPLAY'}  dspy_installed={HAS_DSPY}")

Task — binary triage#

20 tickets, half billing-shaped, half technical-shaped. The labels are the same ones a real triage queue uses; the trick is that a zero-shot classifier sometimes confuses “I was over-charged” (billing) with “rate limit headers wrong” (technical, despite looking numeric).

tp = FIXTURE["test_predictions"]
TICKETS, TRUTH = tp["tickets"], tp["ground_truth"]
for t, y in zip(TICKETS[:4], TRUTH[:4]):
    print(f"  [{y:>9}] {t[:75]}{'...' if len(t) > 75 else ''}")

DSPy program#

A Signature declares input/output fields with brief docstrings; a Module (here dspy.Predict) wires the signature to an LM call. No prompt template anywhere in user code — DSPy generates it from the signature.

CLASSIFIER_SOURCE = """
class TicketLabel(dspy.Signature):
    \"\"\"Classify a customer-support ticket as billing or technical.\"\"\"
    ticket: str = dspy.InputField()
    label:  str = dspy.OutputField(desc=\"'billing' or 'technical'\")

classifier = dspy.Predict(TicketLabel)
"""
print(CLASSIFIER_SOURCE)

Baseline — zero-shot Predict#

No instruction tuning, no demos. Just the signature and the model.

def evaluate(predictions: list[str]) -> dict:
    correct = sum(p == y for p, y in zip(predictions, TRUTH))
    return {"correct": correct, "n": len(TRUTH), "acc": correct / len(TRUTH)}


def run_baseline_live() -> list[str]:
    import dspy  # noqa: PLC0415
    dspy.settings.configure(lm=dspy.LM("anthropic/claude-haiku-4-5", max_tokens=8))

    class TicketLabel(dspy.Signature):
        """Classify a customer-support ticket as billing or technical."""
        ticket: str = dspy.InputField()
        label: str  = dspy.OutputField(desc="'billing' or 'technical'")

    classifier = dspy.Predict(TicketLabel)
    return [classifier(ticket=t).label.strip().lower() for t in TICKETS]


baseline_pred = run_baseline_live() if LIVE else tp["baseline_pred"]
baseline_eval = evaluate(baseline_pred)
print(f"baseline   {baseline_eval['correct']}/{baseline_eval['n']} = {baseline_eval['acc']:.1%}")
wrong = [(t[:60], p, y) for t, p, y in zip(TICKETS, baseline_pred, TRUTH) if p != y]
print(f"baseline mistakes ({len(wrong)}):")
for t, p, y in wrong[:4]:
    print(f"  pred={p:<10} truth={y:<10} {t}")

MIPROv2 — Bayesian search over instructions and demos#

MIPROv2 generates candidate instructions with the LM itself, bootstraps demonstrations from training examples that the baseline classifier got right, and runs a TPE-style search to pick the (instruction, demos) pair that maximises a metric on a held-out set. ~120 LM calls in light mode.

OPTIMIZER_SOURCE = """
# trainset is a list of dspy.Example with .ticket and .label set
def metric(gold, pred, trace=None):
    return gold.label.strip().lower() == pred.label.strip().lower()

optimizer = dspy.MIPROv2(metric=metric, auto=\"light\",
                          max_bootstrapped_demos=4, max_labeled_demos=4)
optimized = optimizer.compile(classifier, trainset=trainset)
"""
print(OPTIMIZER_SOURCE)

After optimisation#

The optimised program ships with a learned instruction (compare cell below) and 4 demos pulled from the trainset. We rerun on the same 20 test tickets.

optimized_pred = tp["optimized_pred"]  # a real LIVE run would call optimized(ticket=...)
optimized_eval = evaluate(optimized_pred)

print(f"baseline   {baseline_eval['correct']}/{baseline_eval['n']} = {baseline_eval['acc']:.1%}")
print(f"optimized  {optimized_eval['correct']}/{optimized_eval['n']} = {optimized_eval['acc']:.1%}")
print(f"absolute lift: +{(optimized_eval['acc'] - baseline_eval['acc']) * 100:.1f} points")

What the optimiser actually picked#

b = FIXTURE["baseline"]; o = FIXTURE["optimized"]
print("BASELINE INSTRUCTION:")
print(f"  {b['instruction']}\n")
print("OPTIMIZED INSTRUCTION:")
print(f"  {o['instruction']}\n")
print(f"OPTIMIZED DEMOS ({o['n_demos']}):")
for d in o["demos"]:
    print(f"  [{d['label']:>9}] {d['ticket']}")

print(f"\noptimization spend: ~{o['n_optimization_calls']} LM calls, "
      f"~${o['optimization_cost_usd']:.4f}")

Visualisation#

import matplotlib.pyplot as plt

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(9.5, 3.4))

ax1.bar(["baseline", "MIPROv2"], [baseline_eval["acc"] * 100, optimized_eval["acc"] * 100],
        color=["tab:gray", "tab:green"])
ax1.set_ylabel("accuracy (%)"); ax1.set_ylim(0, 102)
ax1.set_title(f"test accuracy on {baseline_eval['n']} tickets")
for i, v in enumerate([baseline_eval["acc"], optimized_eval["acc"]]):
    ax1.text(i, v * 100 + 1, f"{v:.0%}", ha="center")

b_costs = [b["mean_cost_usd"] * 1000] * baseline_eval["n"]
o_costs = [o["mean_cost_usd"] * 1000] * optimized_eval["n"]
ax2.bar(["per-call\nbaseline", "per-call\noptimized", "one-time\noptimization"],
        [b["mean_cost_usd"] * 1000, o["mean_cost_usd"] * 1000, o["optimization_cost_usd"] * 1000],
        color=["tab:gray", "tab:green", "tab:red"])
ax2.set_ylabel("cost (×1e-3 $)")
ax2.set_title("cost — per-call inference vs one-time optimization")
ax2.set_yscale("log")

fig.tight_layout(); plt.show()

Checks#

s.check(
    "baseline_recorded_accuracy_consistent",
    lambda: abs(baseline_eval["acc"] - FIXTURE["baseline"]["test_accuracy"]) < 0.01,
    msg=f"computed={baseline_eval['acc']:.3f}  fixture={FIXTURE['baseline']['test_accuracy']}",
)
s.check(
    "optimized_recorded_accuracy_consistent",
    lambda: abs(optimized_eval["acc"] - FIXTURE["optimized"]["test_accuracy"]) < 0.01,
    msg=f"computed={optimized_eval['acc']:.3f}  fixture={FIXTURE['optimized']['test_accuracy']}",
)
s.check(
    "mipro_lifts_at_least_10_points",
    lambda: optimized_eval["acc"] - baseline_eval["acc"] >= 0.10,
    msg=f"lift = +{(optimized_eval['acc'] - baseline_eval['acc']) * 100:.1f} pts",
)
s.check(
    "optimizer_bootstrapped_4_demos",
    lambda: len(FIXTURE["optimized"]["demos"]) == 4,
    msg=f"got {len(FIXTURE['optimized']['demos'])} demos",
)
s.check(
    "optimization_cheaper_than_finetune",
    lambda: FIXTURE["optimized"]["optimization_cost_usd"] < 0.10,
    msg=f"opt cost = ${FIXTURE['optimized']['optimization_cost_usd']:.4f}",
)

Notes for production#

  • MIPROv2 vs BootstrapFewShot: bootstrap is one-shot demo selection; MIPROv2 also searches the instruction text. The latter almost always wins, at ~5x the optimization cost.

  • Metric matters more than the optimizer. A faithful eval metric (here: exact-match on the label) is the spec. Bad metrics make MIPROv2 happily pick prompts that overfit the metric, not the task.

  • Save the optimized program: optimized.save("./classifier.json"). In production you ship the json, not the optimizer call. Re-running optimization weekly against fresh labels is the typical cadence.

  • DSPy with multi-step modules: ChainOfThought, ReAct, custom Module classes — MIPROv2 optimises the prompt at every predict node along the way.

  • Cheap optimizer model, expensive runtime model. A common recipe: optimise with claude-haiku-4-5 to keep search cheap, then load(...) the optimised program against claude-sonnet-4-6 for the actual deployment.

s.summary()
s.save()