DSPy and MIPROv2 — prompts as parameters

Track 04 - Agents · Notebook 04 · Runtime: ~5s on CPU

Prerequisites: 04_agents/02 (structured outputs).

Papers: Khattab et al. 2023 (2310.03714), Opsahl-Ong et al. 2024 (MIPROv2 — 2406.11695).

DSPy treats the prompt text as a parameter. You declare a Signature, and an optimiser (MIPROv2 is the current default) searches over instruction strings and demo subsets to maximise a task metric.

To make the search behaviour visible without running a real DSPy compile loop (~120 LM calls), we replace the LM with a deterministic policy stand-in: a function predict_with_policy(text, instruction, demos) whose output depends mechanically on the policy implied by the instruction and the demo distribution. We then exhaustively score the 9-cell grid of (instruction × demo-pool) pairs and verify a small TPE-style sampler recovers the optimum.

For the same task against a real DSPy 3 program with claude-haiku-4-5, see 08_production/07_dspy_miprov2_optimizer.

from llm_systems_cookbook.nb import bootstrap

from dataclasses import dataclass

import numpy as np

s = bootstrap("04_agents_04_dspy_3_miprov2")

Task: classify sentences as positive / neutral / negative

A deterministic “LLM” scores each sentence with a rule: count positive / negative lexicon words; the effective accuracy depends on the instruction + demo pool given to it.

rng = np.random.default_rng(0)

POS = {"love", "great", "excellent", "amazing", "wonderful", "fantastic", "good", "happy"}
NEG = {"terrible", "awful", "bad", "horrible", "worst", "hate", "disappointing"}


def true_label(text: str) -> str:
    toks = text.lower().split()
    p = sum(1 for t in toks if t in POS)
    n = sum(1 for t in toks if t in NEG)
    if p > n:
        return "positive"
    if n > p:
        return "negative"
    return "neutral"


# Mix of clearly-labeled sentences and tied sentences. Tied sentences
# are where the instruction + demo pool matter — the "specific" instruction
# handles them by falling back to neutral, while "misleading" biases them
# toward positive.
CLEAR_POS = ["I love this product, it is amazing.",
             "A wonderful fantastic experience, great value.",
             "Good food, happy staff, excellent night."]
CLEAR_NEG = ["The food was terrible and service was awful.",
             "Horrible and disappointing. The worst.",
             "Bad weather ruined the awful trip."]
TIED_NEUTRAL = [
    "Great but also horrible.",
    "Love it, hate it.",
    "Good and bad in equal measure.",
    "Amazing but awful.",
    "Excellent and terrible in one package.",
    "Happy and disappointing ending.",
]
PURE_NEUTRAL = ["It was ok, neither strong nor weak.",
                "Nothing to write home about, just average.",
                "Just there. Did the job. So-so."]

EXAMPLES = (CLEAR_POS + CLEAR_NEG + TIED_NEUTRAL + PURE_NEUTRAL) * 2
rng.shuffle(EXAMPLES)
LABELS = [true_label(t) for t in EXAMPLES]

The policy stand-in

Given an instruction + demos + query, return a label. The instruction selects a tie-break policy (“on ties, output neutral” vs “on ties, output positive”); the demo pool’s class distribution can override it. The mapping is mechanical and fully deterministic — that’s what makes the search trajectory below interpretable.

INSTRUCTIONS = {
    "vague":    "Decide the sentiment of the sentence.",
    "specific": "Label each sentence with exactly one of: positive, neutral, negative. Count positive/negative words before deciding; if they tie, output neutral.",
    "misleading": "Always prefer the most intense emotion label. When unsure, output positive.",
}

DEMO_POOLS = {
    "balanced": [
        ("I love this.", "positive"),
        ("This is terrible.", "negative"),
        ("It is ok.", "neutral"),
    ],
    "skewed": [
        ("I love this.", "positive"),
        ("Great food.", "positive"),
        ("Wonderful.", "positive"),
    ],
    "empty": [],
}


_lm_rng = np.random.default_rng(0)


def lm_predict(text: str, instruction: str, demos: list[tuple[str, str]]) -> str:
    '''Simulated LM: the instruction governs the tie-break policy.

    - The *specific* instruction teaches the LM to output neutral on
      ties; accurate.
    - The *vague* instruction leaves ties to a random tie-break -
      models without clear guidance just guess.
    - The *misleading* instruction biases ties toward positive.
    Demo pools further shift the tie-break via implicit anchoring.
    '''
    toks = text.lower().split()
    p = sum(1 for t in toks if t in POS)
    n = sum(1 for t in toks if t in NEG)

    policy = "random"
    if "count positive/negative words" in instruction.lower():
        policy = "neutral"
    elif "prefer the most intense" in instruction.lower():
        policy = "positive"

    if demos:
        pos_frac = sum(1 for _, lbl in demos if lbl == "positive") / len(demos)
        neg_frac = sum(1 for _, lbl in demos if lbl == "negative") / len(demos)
        # Balanced demos + vague instruction = the balanced tie-break lands on neutral.
        if policy == "random" and pos_frac < 0.6 and neg_frac < 0.6 and pos_frac > 0 and neg_frac > 0:
            policy = "neutral"
        # Skewed demos override even the specific instruction slightly.
        if pos_frac > 0.6:
            policy = "positive"

    if p > n:
        return "positive"
    if n > p:
        return "negative"
    if policy == "neutral":
        return "neutral"
    if policy == "positive":
        return "positive"
    if policy == "negative":
        return "negative"
    # Random tie-break (degraded behaviour without guidance or demos).
    return str(_lm_rng.choice(["positive", "negative", "neutral"]))


def evaluate(instruction_name: str, demos_name: str) -> float:
    instr = INSTRUCTIONS[instruction_name]
    demos = DEMO_POOLS[demos_name]
    # Reset the LM's random tie-break stream for reproducibility across
    # (instruction, demos) evaluations.
    global _lm_rng
    _lm_rng = np.random.default_rng(0)
    correct = sum(lm_predict(t, instr, demos) == y for t, y in zip(EXAMPLES, LABELS, strict=True))
    return correct / len(EXAMPLES)

MIPROv2-style search

MIPROv2 uses TPE Bayesian optimisation over a candidate pool generated by a “teacher” LM. With only 9 grid cells, exhaustive search and TPE produce identical answers; we still simulate “evaluate 5 sampled candidates” to show the best-so-far trajectory.

accuracy_grid: dict[tuple[str, str], float] = {}
for instr in INSTRUCTIONS:
    for demo in DEMO_POOLS:
        accuracy_grid[(instr, demo)] = evaluate(instr, demo)

print("full grid (instruction x demos):")
for instr in INSTRUCTIONS:
    row = "   ".join(f"{demo}={accuracy_grid[(instr, demo)]:.3f}" for demo in DEMO_POOLS)
    print(f"  {instr:<12}  {row}")

best_pair = max(accuracy_grid, key=accuracy_grid.get)
best_acc = accuracy_grid[best_pair]
baseline_acc = accuracy_grid[("vague", "empty")]
print(f"\nbaseline (vague + empty demos) accuracy = {baseline_acc:.3f}")
print(f"optimised ({best_pair[0]} + {best_pair[1]}) accuracy = {best_acc:.3f}")


def mipro_search(n_candidates: int = 5, seed: int = 0) -> tuple[tuple[str, str], float]:
    rng = np.random.default_rng(seed)
    pairs = list(accuracy_grid.keys())
    sample_idx = rng.choice(len(pairs), size=min(n_candidates, len(pairs)), replace=False)
    candidates = [pairs[int(i)] for i in sample_idx]
    scored = [(p, accuracy_grid[p]) for p in candidates]
    return max(scored, key=lambda x: x[1])


mipro_pair, mipro_acc = mipro_search(n_candidates=5)
print(f"MIPRO-search (5 candidates) found {mipro_pair} at {mipro_acc:.3f}")

s.check(
    "optimiser_beats_baseline",
    lambda: best_acc > baseline_acc,
    msg=f"baseline={baseline_acc:.3f}  best={best_acc:.3f}",
)
s.check(
    "best_prompt_uses_informative_instruction_or_balanced_demos",
    lambda: best_pair[0] == "specific" or best_pair[1] == "balanced",
    msg=f"best = {best_pair}",
)
s.check(
    "mipro_within_2pp_of_optimal",
    lambda: mipro_acc >= best_acc - 0.02,
    msg=f"mipro={mipro_acc:.3f}  optimal={best_acc:.3f}",
)
s.check(
    "misleading_instruction_hurts",
    lambda: accuracy_grid[("misleading", "skewed")] < accuracy_grid[("specific", "balanced")],
    msg=f"misleading+skewed={accuracy_grid[('misleading', 'skewed')]:.3f}  "
        f"specific+balanced={accuracy_grid[('specific', 'balanced')]:.3f}",
)

Uncompiled vs compiled vs MIPROv2

Three numbers from the grid made visible. The left bar compares the uncompiled baseline (the first prompt anyone would write) against the hand-compiled best pair and against whatever MIPROv2 landed on in five noisy draws. The right panel is the optimiser’s best-so-far curve as it walks through candidate (instruction, demo) pairs - the shape you actually watch during a real MIPROv2 run.

import matplotlib.pyplot as plt

labels = ["uncompiled\n(vague+empty)", "compiled\n(specific+balanced)", f"MIPROv2 search\n{mipro_pair}"]
scores = [baseline_acc, accuracy_grid[("specific", "balanced")], mipro_acc]
colors = ["tab:red", "tab:green", "tab:blue"]

# Best-so-far trajectory as MIPROv2 evaluates candidates one at a time.
rng_t = np.random.default_rng(0)
pairs = list(accuracy_grid.keys())
order = rng_t.choice(len(pairs), size=len(pairs), replace=False)
best_so_far: list[float] = []
running = 0.0
for i in order:
    running = max(running, accuracy_grid[pairs[int(i)]])
    best_so_far.append(running)

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(9, 3.2))
ax1.bar(labels, scores, color=colors)
ax1.set_ylim(0, 1.05); ax1.set_ylabel("accuracy")
ax1.set_title("prompt optimisation lifts task quality")
for i, v in enumerate(scores):
    ax1.text(i, v + 0.02, f"{v:.2f}", ha="center")

ax2.plot(range(1, len(best_so_far) + 1), best_so_far, marker="o", color="tab:blue",
         label="best-so-far")
ax2.axhline(best_acc, color="tab:gray", linestyle=":", label=f"grid optimum {best_acc:.2f}")
ax2.axhline(baseline_acc, color="tab:red", linestyle=":", label=f"baseline {baseline_acc:.2f}")
ax2.set_xlabel("candidates evaluated"); ax2.set_ylabel("accuracy")
ax2.set_title("MIPROv2 best-so-far trajectory")
ax2.legend(); ax2.grid(True, alpha=0.3)
fig.tight_layout(); plt.show()

Exercises

1. Real DSPy. pip install dspy-ai, define a dspy.Signature, point it at a real LM via dspy.LM("anthropic/claude-haiku-4-5"), and run MIPROv2.compile. Compare the optimised instruction against the one you’d write by hand.

2. Add an axis. Extend the grid with temperature ∈ {0, 0.3, 0.7}. Grid search gets expensive; TPE earns its keep.

3. Bootstrapped demos. Replace the hand-written demo pool with demos generated by running a teacher LM on training data and keeping the ones it gets right.

References

• DSPy 3 docs and dspy.teleprompt.MIPROv2 source — the proposer and k-shot sampler are each ~200 lines.

• 08_production/07_dspy_miprov2_optimizer — the same machinery against a real model with a real ~$0.014 optimization budget.

s.summary()
s.save()