Agent evaluation - τ-bench and SWE-bench shapes

Open in Colab ▶️ Run this notebook in Colab

Agent evaluation - τ-bench and SWE-bench shapes#

Track 04 - Agents · Notebook 07 · Runtime: ≈30 s on CPU

Prerequisites: 04_agents/01 (ReAct), 04_agents/03 (state machines).

References:

  • Yao et al. 2024, τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (2406.12045).

  • Jimenez et al. 2024, SWE-bench (2310.06770).

What#

Single-turn success rate is the wrong metric for agents. Real agents loop: they fetch data, call tools, reason, retry. Three metrics tell the real story:

  1. Task success rate. Did the agent complete the task?

  2. Trajectory efficiency. How many tool calls did it take? Fewer is better (faster + cheaper).

  3. Error recovery. When a tool returned an error, did the agent adapt or loop?

We build a tiny τ-bench-shaped benchmark (5 multi-turn customer- support scenarios) and a SWE-bench-shaped benchmark (3 tiny code-patch tasks) with deterministic agents. Reports all three metrics and verifies ratio bounds.

from llm_systems_cookbook.nb import bootstrap

from dataclasses import dataclass

s = bootstrap("04_agents_07_agent_evaluation_suite")

τ-bench-shaped: customer-support scenarios#

Each scenario has a user goal, a sequence of required tool calls, and a success predicate. The agent succeeds iff it executes the required calls (in any order) and returns a message containing the expected answer.

@dataclass
class TauTask:
    name: str
    user_request: str
    required_tools: set[str]
    expected_answer_substring: str


TAU_TASKS = [
    TauTask("lookup_order", "find order 42",
            {"get_order"}, "order 42"),
    TauTask("refund_flow", "I want to refund order 17",
            {"get_order", "issue_refund"}, "refunded"),
    TauTask("address_change", "update shipping address to new one",
            {"get_user", "update_address"}, "address updated"),
    TauTask("cancel_and_refund", "cancel order 7 and refund the money",
            {"get_order", "cancel_order", "issue_refund"}, "refunded"),
    TauTask("double_refund_guard", "refund order 99 (already refunded)",
            {"get_order"}, "already refunded"),
]


# Deterministic agent: uses a scripted policy per task.
TOOL_SCRIPTS: dict[str, list[tuple[str, str]]] = {
    "lookup_order":         [("get_order", "found order 42 total $50")],
    "refund_flow":          [("get_order", "found order 17"),
                              ("issue_refund", "refunded $25 to card")],
    "address_change":       [("get_user", "user u9 found"),
                              ("update_address", "address updated")],
    "cancel_and_refund":    [("get_order", "found order 7"),
                              ("cancel_order", "order 7 cancelled"),
                              ("issue_refund", "refunded $30")],
    "double_refund_guard":  [("get_order", "order 99 is already refunded")],
}


def run_tau_agent(task: TauTask) -> tuple[bool, int, str]:
    '''Returns (success, n_tool_calls, final_answer).'''
    script = TOOL_SCRIPTS[task.name]
    called_tools: set[str] = set()
    observations: list[str] = []
    for tool, obs in script:
        called_tools.add(tool)
        observations.append(obs)
    final = " ".join(observations)
    # Success = all required tools called AND expected substring in final.
    ok = task.required_tools.issubset(called_tools) and task.expected_answer_substring in final
    return ok, len(script), final


tau_results = [run_tau_agent(t) for t in TAU_TASKS]
tau_success = sum(1 for ok, _, _ in tau_results) / len(tau_results)
tau_avg_calls = sum(n for _, n, _ in tau_results) / len(tau_results)
print(f"tau-bench: success={tau_success:.1%}  avg tool calls={tau_avg_calls:.1f}")
for t, (ok, n, final) in zip(TAU_TASKS, tau_results, strict=True):
    mark = "PASS" if ok else "FAIL"
    print(f"  {mark}  {t.name:<22}  {n} calls  final='{final[:60]}'")

SWE-bench-shaped: code-patch tasks#

Three tiny Python bugs; the agent must produce a unified diff or a replacement function. We verify that:

  • The patch parses.

  • The patched code passes reference tests.

SWE-bench’s real eval harness compiles the repo and runs pytest; ours is analogous with exec and tiny assertions.

@dataclass
class SWEProblem:
    name: str
    buggy_code: str
    tests: str
    agent_patch: str  # a replacement snippet (simpler than a diff)


SWE_PROBLEMS = [
    SWEProblem(
        name="off_by_one",
        buggy_code="def add_one(x):\n    return x + 2",
        tests="assert add_one(3) == 4\nassert add_one(0) == 1",
        agent_patch="def add_one(x):\n    return x + 1",
    ),
    SWEProblem(
        name="wrong_sign",
        buggy_code="def abs_val(x):\n    return -x",
        tests="assert abs_val(-3) == 3\nassert abs_val(5) == 5",
        agent_patch="def abs_val(x):\n    return x if x >= 0 else -x",
    ),
    SWEProblem(
        name="edge_case",
        buggy_code="def is_even(x):\n    return x == 2 or x == 4",
        tests="assert is_even(6) is True\nassert is_even(1) is False\nassert is_even(0) is True",
        agent_patch="def is_even(x):\n    return x % 2 == 0",
    ),
]


def apply_and_test(problem: SWEProblem) -> bool:
    ns: dict = {}
    try:
        exec(problem.agent_patch, ns)
        exec(problem.tests, ns)
    except Exception:  # noqa: BLE001 - any failure = failed patch
        return False
    return True


swe_results = [apply_and_test(p) for p in SWE_PROBLEMS]
swe_success = sum(swe_results) / len(swe_results)
for p, ok in zip(SWE_PROBLEMS, swe_results, strict=True):
    mark = "PASS" if ok else "FAIL"
    print(f"  {mark}  {p.name}")
print(f"swe-bench success rate = {swe_success:.1%}")

s.check(
    "tau_success_rate_at_least_80pct",
    lambda: tau_success >= 0.80,
    msg=f"tau success = {tau_success:.1%}",
)
s.check(
    "tau_avg_tool_calls_reasonable",
    lambda: 1.0 <= tau_avg_calls <= 5.0,
    msg=f"avg calls = {tau_avg_calls:.1f}",
)
s.check(
    "every_required_tool_gets_called",
    lambda: all(
        TAU_TASKS[i].required_tools <= {tool for tool, _ in TOOL_SCRIPTS[TAU_TASKS[i].name]}
        for i in range(len(TAU_TASKS))
    ),
    msg="each scripted policy must cover its task's required tools",
)
s.check(
    "swe_success_rate_100pct",
    lambda: swe_success == 1.0,
    msg=f"swe success = {swe_success:.1%}",
)

# Overall composite: a single agent score = 0.5 * tau_success + 0.5 * swe_success.
composite = 0.5 * tau_success + 0.5 * swe_success
print(f"composite score = {composite:.3f}")
s.check(
    "composite_score_above_threshold",
    lambda: composite >= 0.80,
    msg=f"composite = {composite:.3f}",
)

Benchmark report card#

The left bar gives the three headline numbers you would quote on a leaderboard: tau-bench success, SWE-bench success, and the 50/50 composite. The right panel breaks those rates into per-task pass/fail so regressions show up as individual red bars when you tune the agent. Both draw from the tau_results and swe_results lists already computed - no additional agent runs.

import matplotlib.pyplot as plt

benches = ["tau-bench", "SWE-bench", "composite"]
success = [tau_success, swe_success, composite]
colors = ["tab:blue", "tab:orange", "tab:green"]

tau_per_task = [1.0 if ok else 0.0 for ok, _, _ in tau_results]
swe_per_task = [1.0 if ok else 0.0 for ok in swe_results]
tau_calls = [n for _, n, _ in tau_results]

fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(9, 3.2))
ax1.bar(benches, success, color=colors)
ax1.set_ylim(0, 1.05); ax1.set_ylabel("success rate")
ax1.set_title("headline scores per benchmark")
for i, v in enumerate(success):
    ax1.text(i, v + 0.02, f"{v:.0%}", ha="center")

x_tau = list(range(len(TAU_TASKS)))
x_swe = list(range(len(TAU_TASKS), len(TAU_TASKS) + len(SWE_PROBLEMS)))
ax2.bar(x_tau, tau_per_task, color="tab:blue", label="tau tasks")
ax2.bar(x_swe, swe_per_task, color="tab:orange", label="swe tasks")
ax2.set_xticks(x_tau + x_swe)
ax2.set_xticklabels([t.name for t in TAU_TASKS] + [p.name for p in SWE_PROBLEMS],
                    rotation=35, ha="right", fontsize=8)
ax2.set_ylim(0, 1.2); ax2.set_ylabel("pass (1) / fail (0)")
ax2.set_title(f"per-task outcome (avg tau calls {tau_avg_calls:.1f})")
ax2.legend(loc="upper right", fontsize=8)
fig.tight_layout(); plt.show()

Exercises#

  1. Error recovery scoring. Inject a 20% random tool-call failure and measure how many tasks still succeed via retries. That’s the agent’s “robustness” dimension.

  2. Real SWE-bench-lite. Download the SWE-bench-lite dataset (300 issues) and run your agent against it. Even 5% success is a respectable baseline.

  3. User simulator. τ-bench’s real innovation is a user simulator that varies phrasing between turns. Add a simple paraphraser to the user requests and measure how your agent’s success rate drops.

References#

  • τ-bench paper for the simulator architecture and success metric.

  • SWE-bench paper and its swebench evaluation harness.

  • OpenAI’s evals framework for a lighter-weight alternative to both.

s.summary()
s.save()