Production patterns

Production patterns#

Real LLM code that runs against real APIs (Anthropic, OpenAI, Ollama, local vLLM). No regex stubs, no rule-based “policies” pretending to be models. Each notebook works in two modes:

  • LIVE: when an API key is set, hits the real provider and shows fresh numbers.

  • Replay: without keys, loads recorded responses from _fixtures/ so the notebook still runs end-to-end on a fresh Colab.

Recorded fixtures are regenerated with python scripts/refresh_fixtures.py on a real run; the responses you see in replay mode are real responses, just from a previous run.

Reading order#

  1. 01_claude_sdk_prompt_caching: cache_control on a long system prompt; measured cache hit rate, $ saved, latency drop.

  2. 02_litellm_router_fallbacks: multi-provider routing with cost and latency tracking; fallback when the primary provider 429s.

  3. 03_tool_use_agent: native Anthropic tool use, parallel tool calls, no parser. Compared against the regex-parser approach from 04_agents/01.

  4. 04_structured_outputs_real: head-to-head compliance/latency for Anthropic tool-use, Outlines + Qwen2.5, Instructor, BAML.

  5. 05_hybrid_rag_production: BGE-M3 dense + BM25 + RRF + reranker → Claude with citations on a 1k-doc corpus.

  6. 06_mcp_real_server: an MCP server (stdio transport, mcp SDK) that Claude Code / Cursor can connect to.

  7. 07_dspy_miprov2_optimizer: DSPy 3 program for classification; MIPROv2 optimises against held-out accuracy.

  8. 08_inspect_ai_eval_harness: Inspect AI task + scorer + solver on a real benchmark.

  9. 09_gpu_providers_pricing_and_model_fit: practical reference, GPU types in production, on-demand and spot pricing across ten cloud providers, vRAM math, and a calculator that maps a model size to the smallest cluster that holds it.

Models#

Defaults: claude-sonnet-4-6 (Anthropic), gpt-5.5 (OpenAI; GPT-5.5 Instant is the current ChatGPT default; GPT-5.5 Thinking is the unified successor to the o-series for reasoning tasks), Qwen/Qwen2.5-1.5B-Instruct (local via Ollama or vLLM). Override with the MODEL_* env vars listed at the top of each notebook.

The current frontier tier (mid-2026): Claude Fable 5 (claude-fable-5, GA July 1, 2026, \(10/\)50 per M tokens) leads on SWE-bench Pro (80.3%) and long-context agentic tasks, at roughly half the per-token price of its predecessor; Claude Sonnet 5 (claude-sonnet-5, June 30, 2026, 63.2% SWE-bench Pro) is the balanced-tier option one step down; GPT-5.6 Sol (OpenAI, July 9, 2026, \(5/\)30 per M tokens) edges out Fable 5 on the Artificial Analysis Coding Agent Index at under half the output tokens; Grok 4.5 (xAI, July 8, 2026) undercuts Opus-class pricing by over 60% while landing fourth on the Artificial Analysis Intelligence Index; Gemini 3.5 Pro (limited Vertex AI preview, 2M-token context window with Deep Think reasoning) targets enterprise document workloads. For cost-sensitive production use, claude-haiku-4-5-20251001 and gpt-5.5-instant remain the default latency-optimised choices.