A 70-notebook curriculum for LLM systems engineering#
Current status (May 2026): The repository ships 64 of 70 notebooks. The six remaining gaps (
03_training/03..08: tensor parallel, pipeline parallel, LoRA, QLoRA, DPO, GRPO) are fully specified below and targeted for v0.2. A v0.3 roadmap extending the curriculum to ~80 notebooks is outlined in the Implementation notes section and covers: (1) test-time compute / reasoning models, (2) BitNet and sub-2-bit quantization, (3) a multimodal/VLM track, and (4) safety and red-teaming. Every track outside the training stub is currently complete.
This report delivers 70 fully-specified Jupyter notebooks organized across eight engineering areas of modern LLM systems: inference engines, retrieval-augmented generation, training and fine-tuning, agent frameworks, serving and scaling, evaluation, GPU programming, and production patterns. Every notebook includes exact 2026 package pins, a demo model drawn from a Colab-T4-compatible shortlist (SmolLM2-135M/360M, Qwen2.5-0.5B, Llama-3.2-1B, Phi-3.5-mini), a 12-20 cell outline with fully type-hinted function signatures, 4-6 numerical scoring checks with concrete thresholds (e.g., “Triton matmul ≥ 70% cuBLAS TFLOPs at 4096² FP16”), expected runtime outputs, and 2-3 stretch goals. The design is lab-style throughout - no interview framing - and every notebook instantiates a common Scorer that emits scores/NN.json for CI aggregation. The Production track adds a LIVE/Replay shim so each notebook works against real APIs when keys are set and against recorded fixtures otherwise — runnable end-to-end on Colab without secrets. Difficulty progresses within each track from single-GPU toy kernels to multi-process disaggregated serving; shared utilities (_utils.py, scoring/harness.py) keep the surface area small. The specs below are written so an implementer can produce each notebook directly without re-researching APIs.
Inference engines - 10 notebooks (notebooks/01_inference)#
01 Autoregressive decoding and KV-cache anatomy (CPU/T4, 20 min). Implements kv_cache_bytes(num_layers, num_kv_heads, head_dim, seq_len, batch, dtype_bytes=2) -> int = 2·L·H·D·T·B·bytes; runs generate_no_cache vs generate_with_cache using HF DynamicCache; sweeps context {128..2048}; verifies measured max_memory_allocated within 5% of formula and cache gives ≥5× tokens/sec. Stretch: int8 KV (~2× memory reduction), torch.compile fullgraph. Papers: 2309.06180.
02 Attention from scratch and roofline (1×GPU, 15 min). naive_attention matches F.scaled_dot_product_attention to 1e-3; derives attention_flops, attention_bytes, arithmetic_intensity; microbenchmarks peak FLOPs and BW; plots roofline with ridge in (5, 200) FLOPs/byte; classifies prefill@N=1024 as compute-bound and decode@N=1 as memory-bound; demonstrates empirical O(N²) memory growth (ratio >12 for 4× N). Papers: 2205.14135, Williams CACM 2009.
03 PagedAttention block allocator (CPU-only, 15 min). Pure-Python BlockAllocator(num_blocks, block_size=16), PagedKVManager with translate(seq_id, logical_pos) -> (physical_block, offset), fork_sequence and copy_on_write with ref counting. Tests verify 100 random-length sequences waste <5% aggregate and <block_size each; paged saves ≥10× over max_seq_len=4096 contiguous. Stretch: swapping-to-CPU with LRU, preemption-vs-swap recovery. Papers: 2309.06180.
04 Continuous batching scheduler (Orca) (CPU, 15 min). simpy-based simulator comparing StaticBatcher vs ContinuousBatcher on 500 Poisson arrivals (λ=30 req/s, log-normal lengths). Calibrated step_latency(batch_size, total_tokens) = 0.005 + 0.0008·B + 0.00002·T. Checks continuous throughput ≥2.5× static; TTFT p99 ≥40% lower; selective-attention beats padded by ≥1.5×; throughput sweet-spot at batch ∈ {16,32,64}. Papers: Orca OSDI’22.
05 FlashAttention-2 in Triton (1×GPU Ampere+, 25 min). Kernel _fa2_fwd with BLOCK_M, BLOCK_N constexprs parallelizing over Q-row tiles (FA2’s key change); causal and non-causal variants match SDPA to 1e-3 max-abs; memory grows ≤20× when N grows 16× (linear); ≥3× naive at N=2048; within 1.5× of cuDNN SDPA. Stretch: backward with recomputation, varlen with cu_seqlens. Papers: 2205.14135, 2307.08691, 2407.08608.
06 RadixAttention prefix cache (T4, 20 min). RadixCache.match_prefix(tokens) -> (matched_blocks, node) with edge splitting; insert, evict (LRU over ref_count=0 leaves); lpm_schedule (longest-prefix-match scheduler). Benchmark: 200 multi-turn conversations with shared 128-token system prompt → hit-rate ≥0.8, LPM beats FIFO ≥1.3× under bounded cache, TTFT ≥3× lower than no-cache baseline. Papers: 2312.07104.
07 Speculative decoding (1×GPU, 25 min). speculative_generate with γ=4 draft tokens, rejection rule min(1, p(x)/q(x)) with residual sampling from normalize(relu(p-q)). Checks: tokenizers match, total-variation distance of unigram distribution <0.05 vs target-only (distributional equivalence), acceptance α≥0.6 on 20 prompts, wall-clock ≥1.5× at α>0.6, closed-form E[tok/step] = (1-α^(γ+1))/(1-α) within 15% of empirical. Papers: 2211.17192, 2302.01318.
08 Medusa + EAGLE tree speculation (T4/L4, 30 min). Train 4 Medusa heads (500 steps on ultrachat_200k slice, ≥30% loss reduction); static tree [4,3,2,2] with 88 nodes (4 + 12 + 24 + 48) truncated to 64; verify_tree in a single forward with ancestor mask; dynamic pruning (EAGLE-2 style) with threshold 0.03. Tree accepts ≥1.3× more tokens/step than linear; wall-clock ≥1.7× speedup. Stretch: EAGLE-3 multi-layer fusion. Papers: 2401.10774, 2401.15077, 2503.01840.
09 SARATHI-Serve chunked prefill (1×GPU sim, 20 min). Fits latency a + b·P + c·D + d·max(P,D) on real forwards (R²≥0.9). ChunkedPrefillScheduler(chunk_size=1024, budget=1536) with decode-maximal fill. On 300 mixed requests (20% long prefills): TPOT p99 cut ≥30%, TTFT p99 within 1.2×, throughput within 5%; sweet-spot at chunk ∈ {1024, 1536}. Papers: 2308.16369, 2403.02310.
10 Disaggregated prefill/decode serving (T4, 25 min). Two processes via multiprocessing; KV serialized into SharedMemory blocks; serialize_kv_to_shm with KVHandoff(req_id, shm_name, layers, num_kv_heads, head_dim, seq_len, dtype); decode worker attaches and deserializes. Greedy outputs bitwise-match collocated; KV bytes match formula ±1 KB; shm bandwidth ≥1 GB/s; on skewed workload TTFT p99 and TPOT p99 each improve ≥20%; transfer overhead <15% of TTFT. Papers: 2401.09670, 2407.00079, 2311.18677.
Retrieval-augmented generation - 9 notebooks (notebooks/02_rag)#
01 Chunking strategies (T4, 18 min) compares fixed, recursive, semantic (embedding-distance breakpoint at 95th percentile), and late chunking (full-doc encode with Jina v2, mean-pool token spans) on BEIR/scifact 300-dev. Recall@10 thresholds: fixed ≥0.72, recursive ≥0.78, semantic ≥0.82, late ≥0.80. Model: BAAI/bge-small-en-v1.5.
02 FAISS dense retrieval (T4, 22 min) embeds 10K Wikipedia mini-corpus with bge-small, sweeps IndexFlatIP, IndexIVFPQ(nlist=256, M=48, nbits=8), IndexHNSWFlat(M=32, efConstruction=200). Target: HNSW@efSearch=64 recall@10 ≥0.95 within 2× flat latency; IVF-PQ memory ≤25% of flat; Pareto plot of recall-vs-latency. Stretch: matryoshka dims on nomic-embed.
03 BM25, SPLADE, and RRF hybrid (T4, 20 min). Custom BM25 with k1=1.2, b=0.75 verified against rank_bm25 to 1e-4; SPLADE-v3 encoding w = log(1+ReLU(logits)) max-pooled, nnz ∈ [80, 300]/doc; RRF Σ 1/(60+rank_i(d)). Hybrid RRF NDCG@10 ≥ max(BM25, dense) + 0.02 on BEIR/scifact.
04 ColBERTv2 late interaction (T4, 24 min). Via pylate 1.1.7. maxsim(q_emb, d_emb) = (q @ d.T).max(dim=1).values.sum(). On scifact dev: Recall@10 ≥0.82, MRR@10 ≥ bge-small + 0.03. Storage blow-up bounded ≤50×. Stretch: PLAID centroid prefiltering.
05 Two-stage reranking (T4, 18 min). Bi-encoder top-100 (bge-small) → cross-encoder (cross-encoder/ms-marco-MiniLM-L-6-v2, BAAI/bge-reranker-v2-m3, jina v2). On BEIR/fiqa dev: MiniLM uplift ≥0.05 NDCG@10, bge-v2-m3 uplift ≥0.07; top_n=100 saturates within 0.01 of 200; MiniLM p50 ≤80 ms on T4.
06 HyDE and query rewriting (T4, 22 min). HyDE generates hypothetical answer with Qwen2.5-0.5B, embeds, retrieves. Multi-query (4 paraphrases + RRF union) and decomposition (max 3 sub-queries). On HotpotQA dev 500: HyDE Recall@10 ≥ baseline+0.03, decomposition ≥ baseline+0.04 on 2-hop subset. Papers: 2212.10496.
07 RAPTOR hierarchical (T4, 28 min). UMAP(10d) → BIC-selected GMM → summarize with Qwen2.5-0.5B → recurse up to 3 levels. On NarrativeQA 20-doc / 150-query subset, thematic-query Recall@5 ≥ flat + 0.05; collapsed-RAPTOR F1 ≥ flat on full set. Tree depth ∈ [2, 4]. Papers: 2401.18059.
08 GraphRAG with Leiden (T4, 30 min). LLM entity+relation JSON extraction on 500-doc AG-News slice → NetworkX graph (|V|≥500, |E|≥1000) → leidenalg.find_partition (modularity >0.3, 5-25 communities) → community summaries → map-reduce QA. GraphRAG ≥60% on global-sensing queries vs <30% naive RAG; GraphRAG not worse than naive on factoid by more than 10 pp. Papers: 2404.16130.
09 RAGAS evaluation harness (T4, 20 min). 50-query SQuAD subset; good pipeline (retrieve+ground) vs bad pipeline (ignore context). Three-run mean±std on faithfulness, answer_relevancy, context_precision, context_recall. Good faithfulness ≥0.80, bad <0.50, good > bad on all four metrics; judge variance std ≤0.08; 8-gram contamination histogram reported. Papers: 2309.15217.
Training and fine-tuning - 8 notebooks (notebooks/03_training)#
01 Mixed precision + gradient accumulation + checkpointing (T4, 8-12 min). Ablation: fp32 → bf16 (≥30% peak memory reduction) → bf16+accum=4 (loss within 2% of non-accum) → bf16+accum+activation checkpointing (≥30% further reduction, ≤25% step-time overhead). Uses torch.utils.checkpoint(block, use_reentrant=False). Mini-GPT2 from SmolLM2 config.
02 DDP vs FSDP2 (2×L4 or CPU gloo fallback, 10-15 min). Uses torch.distributed.fsdp.fully_shard (not legacy FSDP1): fully_shard(layer, mesh=mesh, mp_policy=MixedPrecisionPolicy(param_dtype=torch.bfloat16)). Verifies param.placements == (Shard(0),) on ≥90% of wrapped params; FSDP2 memory ≤ (1/N + 0.1)·DDP; final loss within 1%. Papers: 1910.02054, 2304.11277.
03 Tensor parallel from scratch (CPU/T4, 5-8 min). Hand-rolled ColumnParallelLinear (split out-dim, optional all-gather) and RowParallelLinear (split in-dim, simulate all-reduce via torch.stack(shards,0).sum(0)). Applied to QKV (column) + output (row) and up/down SwiGLU. Output matches single-GPU to 1e-5 for TP ∈ {2, 4, 8}. Papers: 1909.08053.
04 Pipeline parallelism - GPipe + 1F1B (CPU/T4, 6-10 min). 4 stages as sequential nn.Modules. gpipe_schedule and one_f_one_b_schedule with warmup (P−1 forwards) / steady-state / cooldown. Empirical bubble matches theoretical (P−1)/(M+P−1) + 0.02 for 1F1B and (P−1)/M ± 0.05 for GPipe; gradients identical to 1e-4. Papers: 1811.06965, 1806.03377.
05 LoRA from scratch vs PEFT (T4, 15-20 min). ManualLoRALinear with B∈R^{out×r}, A∈R^{r×in}, scaling α/r; inject into q_proj, v_proj of Qwen2.5-0.5B; fine-tune on Alpaca-mini (1K). Manual matches peft.LoraConfig loss curve within 1% after 100 steps; trainable ≤1% of params; adapter ≤10 MB. Papers: 2106.09685.
06 QLoRA NF4 fine-tune (T4, 18-25 min). BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.bfloat16) on Llama-3.2-1B; TRL SFTTrainer with optim="paged_adamw_8bit". Fits in ≤10 GB VRAM; eval PPL within 0.5 of BF16-LoRA. Papers: 2305.14314.
07 DPO preference tuning (T4, 20-30 min). DPOTrainer on trl-lib/ultrafeedback_binarized 1K with β∈{0.1,0.3,0.5}. rewards/margin > 0 monotone non-decreasing by step 50; rewards/accuracies ≥ 0.65; win-rate vs SFT ≥55% by LLM-judge on 50 prompts. Reference-free ablation via loss_type="simpo". Papers: 2305.18290.
08 GRPO DeepSeek-R1-style (T4, 25-40 min). GRPOTrainer(reward_funcs=[correctness_reward, format_reward], args=GRPOConfig(num_generations=4, beta=0.04, ...)) on GSM8K. System prompt forces <reasoning>...</reasoning><answer>...</answer>. Mean reward ≥2× baseline over 100 steps; format fraction ≥0.8 by step 50; eval accuracy up ≥5 pp; within-group std >0 on ≥80% of steps. Stretch: apply DAPO (Decoupled Clip and Dynamic Sampling Policy Optimization; ByteDance, 2025) as a GRPO variant that removes the KL penalty and clips token-level losses independently, typically yielding faster convergence on math tasks; or GRPO-λ (arXiv 2510.00194) which introduces per-token credit assignment to improve GRPO’s reward attribution. Papers: 2501.12948, 2402.03300, 2503.14476, 2510.00194.
Agent frameworks - 7 notebooks (notebooks/04_agents)#
01 ReAct from scratch (CPU/T4, 6 min). Regex parser r"Action:\s*(\w+)\[(.*?)\]"; three typed tools (calculator via safe AST eval, wiki_search stub over 25 facts, get_datetime); 20-task bench (8 arithmetic, 8 lookup, 4 temporal). Success rate ≥0.70, parse errors ≤0.10, avg steps ≤5.0. Served via Ollama phi3.5 or vLLM Qwen2.5-0.5B. Papers: 2210.03629.
02 Structured outputs three ways (T4, 8 min). Person Pydantic v2 model; compares (a) prompt-only JSON, (b) Pydantic validator+retry, (c) Outlines generate.json FSM-constrained. Outlines validity=1.00, compliance≥0.98; prompt-only <0.95; Pydantic-retry lift ≥0.05 over prompt-only; Outlines latency ≤3× baseline. Stretch: swap Outlines for XGrammar-2 (xgrammar==0.2.*, May 2026) which ships Structural Tag — a composable JSON protocol unifying tool calling, reasoning channels, and custom output formats — and delivers ~7× lower end-to-end latency and ~80× faster grammar compilation vs the original XGrammar. Papers: 2307.09702, 2411.15100.
03 LangGraph state machines (CPU, 7 min). Typed StateGraph with Annotated[list[dict], operator.add] reducer; 4 nodes (supervisor/researcher/writer/critic); add_conditional_edges routes on state["route"]; InMemorySaver checkpointer; termination via iteration >= 4. Routes ≥9/10 test inputs; cycles terminate within 6 iters.
04 DSPy 3.0 + MIPROv2 (T4, 12 min). FactualQA signature; Predict baseline → ChainOfThought (+0.03 lift) → BootstrapFewShot → MIPROv2(auto="light") +≥0.10 absolute over baseline on 10-item trivia_qa test. Inspects optimized instructions and demos; saves/reloads .json with identical deterministic score. Papers: 2310.03714, 2406.11695.
05 MCP server/client (CPU, 5 min). %%writefile toy_server.py implementing @server.list_tools/@server.call_tool with echo(text), add(a,b), and notes://today resource via stdio transport. 100-call benchmark: median latency <100 ms, p95 <250 ms; unicode round-trip byte-exact; LLM succeeds on ≥2/3 end-to-end prompts with tool-call schema translation.
06 AutoGen 0.4 / AG2 vs CrewAI (CPU/T4, 10 min). Identical supervisor/worker/critic trio on 10 arithmetic word problems with programmatic checker. AutoGen/AG2 0.4: SelectorGroupChat with TextMentionTermination("APPROVED") | MaxMessageTermination(12); the community maintains the 0.2-compatible lineage as ag2ai/ag2 with streaming, event-driven architecture, and dependency injection. CrewAI: Process.hierarchical with manager_llm (v1.12 adds hierarchical memory isolation). Pass@1 ≥0.60 in at least one framework; both ≥0.40; critic flags ≥0.80 of 5 planted errors. Stretch: Google ADK SequentialAgent with the A2A protocol for cross-framework agent communication. Note: Microsoft Agent Framework 1.0 (April 2026) merges AutoGen 0.4 and Semantic Kernel into a production-grade SDK; the AutoGen patterns in this notebook migrate with minimal API changes (see migration guide at devblogs.microsoft.com/agent-framework). Papers: 2308.08155.
07 Agent evaluation suite (L4, 18 min). Synthetic τ-bench-retail (10 episodes, Pydantic RetailDB with 5 tools and LLM-driven UserSim) plus 3 real SWE-bench-Lite instances (clone repo at base_commit, agent emits diff, git apply + pytest subset). Pass@1 within 0.05 of reference on 3 SWE instances; τ-retail success ≥0.40; unauthorized-action rate ≤0.10. Rich scorecard printed and saved to artifacts/eval_report.json. Papers: 2310.06770, 2406.12045, 2311.12983.
Serving and scaling - 11 notebooks (notebooks/05_serving)#
01 Roofline analysis (T4, 12 min). Measures peak TFLOP/s (GEMM sweep) and HBM BW (large copy) within 30% and 25% of vendor spec respectively. Computes AI for MLP, prefill-attn, decode-attn using formulas. Ridge point in (5, 200) FLOPs/byte; decode correctly classified memory-bound at all tested shapes.
02 KV cache variants MHA/MQA/GQA/MLA (T4, 14 min). Four nn.Module variants with identical APIs. Analytic kv_bytes_per_token: MHA=4096, GQA(8)=512, MQA=64, MLA≈70 exact match. GQA reduces to MHA numerically with n_kv_heads=n_heads; MLA decode ≥3× faster than MHA at 8K context. Implements MLA’s absorbed-matmul trick. Papers: 2305.13245, 1911.02150, 2405.04434.
03 KV compression - StreamingLLM, H2O, SnapKV (T4, 18 min). Monkey-patches attention via attn_implementation="eager" to hook probabilities. StreamingLLM keeps first 4 sinks + last 1020 tokens; H2O top-256 heavy hitters + 256 recent; SnapKV uses 32-token obs window to rank prompt tokens. PPL within 1.0 at 1024-budget; heavy-hitter Jaccard ≥0.7 across adjacent windows; ≥1.8× tokens/s at 8K prompt. Papers: 2309.17453, 2306.14048, 2404.14469.
04 2-bit KV quantization (KIVI) (T4, 16 min). Per-channel K (group_size=128), per-token V, residual window=32 FP16. KIVI-2bit PPL within 0.5 of FP16 on WikiText-2; KV memory ≥6.5× reduction; per-channel K beats per-token K by ≥0.3 PPL at 2 bits; decode ≥1.5× at 8K context. Papers: 2402.02750.
05 GPTQ + AWQ weight quant (L4, 25 min). Hand-rolled GPTQ: build_hessian(x) = 2·XXᵀ/n + 0.01·mean(diag)·I; Cholesky-based column update loop; verify reconstruction MSE < 0.6× RTN. Production path via gptqmodel.quantize(bits=4, group_size=128, desc_act=True) and autoawq. PPL within 0.5 of FP16; AWQ int4 ≥2× FP16 throughput via vLLM V1 quantization="awq_marlin" at batch=8. Papers: 2210.17323, 2306.00978.
06 SmoothQuant + FP8 + NF4 (L4, 22 min). smooth_scale(act_max, weight_max, α) = max(|X|)^α / max(|W|)^(1-α), α-sweep {0.3, 0.5, 0.7, 0.85}. Absorbs 1/s into preceding LN/RMSNorm. SmoothQuant W8A8 ΔPPL <0.3 on SmolLM2-135M (naive W8A8 > 2.0 as control). NF4 double-quant reduces overhead ≥0.3 bits/param. FP8 block-wise simulation ΔPPL <0.2; real FP8 gated on capability >= (8,9). Papers: 2211.10438, 2305.14314, 2412.19437.
07 QuaRot + SpinQuant rotations (T4, 18 min). Walsh-Hadamard R via Sylvester construction (scaled 1/√n); fuses into embed/unembed/norms using computational invariance. Max activation magnitude reduced ≥5×; rotation-only (no quant) ΔPPL <0.05 (invariance verified); W4A4 with Hadamard ≤ W4A4 without rotation − 8.0 PPL; 200-step Cayley-SGD on R1 adds ≥0.3 further. Papers: 2404.00456, 2405.16406.
08 Batching strategies (T4, 15 min). simpy simulator for Static, Dynamic (token budget), Continuous, and Chunked-Prefill schedulers. Calibrated execution model prefill_ms ≈ a+b·tokens, decode_ms ≈ c+d·batch+e·kv_tokens. Continuous ≥3× static throughput; chunked p99 TPOT ≤0.7× unchunked; sim vs real vLLM V1 TTFT RMSE ≤30%.
09 MoE with expert parallelism (CPU/T4, 14 min). 8 experts, d=256, top-2, 1 shared expert. AuxLossMoE vs AuxFreeMoE (DeepSeek-V3 bias update b_i -= γ·(load_i - mean_load), γ=1e-3). Load CoV <0.15 with aux-free after 500 steps; proxy loss within 2% of aux-loss; 4-rank gloo all-to-all runs cleanly; EPLB replicas reduce max-expert load ≥25%. Papers: 2401.06066, 2412.19437.
10 Disaggregated serving (DistServe) (T4, 18 min). Prefill and decode as separate spawned processes with ZMQ control + multiprocessing.Queue for torch.save-serialized KV blobs. Load sweep RPS ∈ {1..16}: disagg TTFT p99 ≤ colocated at RPS≥4; TPOT p99 ≤0.7× at RPS≥8; transfer overhead <20% of TTFT; 2P:1D improves goodput ≥20% over 1P:1D on prompt-heavy. Maps to NVIDIA Dynamo’s KV-aware Router + NIXL + SLO Planner (GTC 2025); Dynamo demonstrated 30× higher request throughput on DeepSeek-R1 with GB200 NVL72. Papers: 2401.09670, 2407.00079, 2311.18677.
11 Serving observability + SLO + autoscaler (T4, 20 min). Launches vLLM V1 with Prometheus; minimal Prom-text parser extracts vllm:time_to_first_token_seconds, vllm:time_per_output_token_seconds, vllm:gpu_cache_usage_perc. SLO(ttft_p95_s=0.6, tpot_p95_s=0.05); Autoscaler with hysteresis (scale up if p95 TTFT > 1.1·SLO for 30s; down if <0.7·SLO for 60s). 5-minute sinusoidal burst workload: attainment ≥95%, scale-down within 120s, per-replica goodput up ≥40% during burst. Stretch: survey AIBrix (vllm-project/aibrix, ByteDance, v0.6.0) as a Kubernetes-native control plane for vLLM that adds high-density LoRA management, prefix-aware and load-aware routing, and a distributed KV cache reporting 50% throughput gain and 70% latency reduction in production benchmarks.
Evaluation - 8 notebooks (notebooks/06_eval)#
01 Perplexity from scratch (T4, 18 min). Naive non-overlapping ppl_naive then sliding-window ppl_sliding(stride) with -100 masking of overlap. Synthetic uniform test: PPL ≈ vocab size within 1%. SmolLM2-360M on WikiText-2 within ±10% of published (~10-12); monotone ppl(stride=128) < ppl(stride=2048) across all models.
02 MMLU harness + calibration (T4, 25 min). Both score_loglikelihood (length-normalized log-probs over {A,B,C,D}) and score_generation (regex \b([A-J])\b) with exact harness 5-shot prompt. 300 questions stratified × 15 subjects. Qwen2.5-0.5B-Instruct within ±3 pp of leaderboard (~47%); MMLU-Pro drop ≥10 pp; ECE <0.20 with reliability diagram. Papers: 2009.03300, 2406.01574, 1706.04599.
03 HumanEval with unbiased pass@k (T4, 35 min). pass_at_k(n, c, k) = 1 − C(n-c, k)/C(n, k) with numerical stability; unsafe_execute sandbox via multiprocess.Process with a hard wall-clock timeout. resource.setrlimit(RLIMIT_AS) is an optional hardening step documented in the exercises - the shipped notebook relies on process isolation + timeout rather than seccomp/rlimit, and warns readers that running LLM-generated code is still a trust-the-source operation. n=20 samples × T ∈ {0.2, 0.8}. Unit test: n=20,c=5,k=1 → 0.25 ± 0.01. Qwen2.5-Coder-0.5B pass@1 at T=0.2 ≥ 0.20; T=0.8 pass@10 ≥ T=0.2 pass@10; infinite-loop killed within timeout+1s. Papers: 2107.03374.
04 LLM-as-judge bias (T4, 25 min). 20 MT-Bench prompts, two candidates, both orders (AB/BA). position_flip_rate on non-tie→different-non-tie; length-bias Spearman ρ with bootstrap 95% CI; self-preference via second judge. Raw flip rate >0.15; after randomization+CoT <0.08; binomial test for self-preference. Papers: 2306.05685, 2305.17926.
05 Arena Elo + Bradley-Terry (CPU, 6 min). Synthetic 1000 battles with known θ. BT-MLE via LogisticRegression(fit_intercept=False) with ±1 design matrix; recovery RMSE <0.10; bootstrap CI width <0.20; online Elo Spearman rank >0.9 vs MLE; √n CI shrinkage verified (width@1000/width@100 ∈ [0.25, 0.45]). Papers: 2403.04132.
06 Long context NIAH + RULER (L4, 30 min). Token-accurate needle insertion at depth_pct ∈ {0.1,...,0.9}; grid over context ∈ {1K..16K} × 3 seeds = 75 runs. RULER MK-2 (two needles) and MV (4 values/key) procedurally generated. Phi-3.5-mini NIAH ≥0.8 at 4K/8K; RULER avg ≥0.7 at 4K; hit-rate monotone non-increasing past window. Papers: 2404.06654, 2307.03172.
07 Contamination detection (T4 or CPU-135M fallback, 20 min). Brief LoRA fine-tune on 100 “seen” WikiText docs (3 epochs, r=8) vs 100 held-out. ngram_set + containment; min_k_prob(k_pct=0.20) averaging lowest-20% token log-probs. Min-K% AUC ≥0.70 finetuned; ≈0.50 base (null control); n-gram AUC ≥0.90 (trivial verbatim); Oren exchangeability p<0.05 on one memorized doc. Note: SWE-bench Live (arXiv 2505.23419) is a live-updatable benchmark companion that mitigates contamination by continuously adding new GitHub-issue tasks created after training cutoffs; its methodology (1,319 initial tasks, 93 repos) is the practical successor to static SWE-bench subsets. Papers: 2310.17623, 2310.16789, 2505.23419.
08 lm-eval + Inspect AI (T4, 35 min). Both frameworks on ARC-Easy 25-shot and HellaSwag 10-shot (limit=500). inspect_ai.eval(tasks=[...], model="hf/Qwen/Qwen2.5-0.5B-Instruct"). Reproduces leaderboard within ±2 pp; cross-framework gap ≤0.02 absolute. Lighteval command strings as third cross-reference. Documents divergence sources (whitespace, length norm, letter extraction). Stretch: implement a 10-task Terminal-Bench 2.0 (Jan 2026) mini-harness — bash-based evaluation of file manipulation and system-administration tasks through subprocess execution — as a complementary agentic benchmark that tests CLI-driven tool use rather than code patch generation.
GPU programming - 8 notebooks (notebooks/07_gpu)#
01 GPU architecture tour (T4, 12 min). STREAM-style copy_kernel sweeping 1MB-1GB; achievable BW ≥80% of vendor peak for ≥64MB buffers. Coalesced vs strided-gather: stride=1 ≥5× stride-32. Theoretical occupancy via active_warps = min(max, regs_per_SM/regs/32, smem_per_SM/smem·blocks) agrees with CUDA runtime API within 1 block/SM. Papers: PMPP Ch. 4-5.
02 Triton 101 softmax (T4, 10 min). Vector-add (within 5% of torch.add); naive softmax (one row per program); online softmax with running (m, ℓ) via Milakov-Gimelshein recurrence; @triton.autotune sweeping BLOCK_N ∈ {128..2048}, num_warps ∈ {2,4,8}, num_stages ∈ {2,3,4}. Online ≥2× naive at N≥16384, matches torch.softmax at small N. Papers: 1805.02867.
03 Triton tiled matmul (A10, 15 min). Blocked GEMM with tl.dot; group-M swizzling:
group_id = pid // (GROUP_M * num_pid_n)
first_pid_m = group_id * GROUP_M
pid_m = first_pid_m + (pid % min(num_pid_m - first_pid_m, GROUP_M))
pid_n = (pid % (GROUP_M * num_pid_n)) // min(...)
Autotune BLOCK_M/N/K ∈ {64..256}, GROUP_M ∈ {4,8}. ≥70% of cuBLAS TFLOPs at 4096² FP16; Group-M ≥1.2× non-grouped. Papers: Simon Boehm matmul blog.
04 Triton FlashAttention-2 (A100, 15 min). Outer loop over Q tiles, inner over K/V tiles; online softmax O_new = O_old·exp(m_old-m_new) + P_new·V_j; causal mask via tl.where(m_idx ≥ n_idx, S, -inf) with early-exit if n_start*BLOCK_N > (pid_m+1)*BLOCK_M: break. Matches SDPA to 1e-3; memory linear in N (slope ≤2×); ≥0.7× SDPA TFLOPs at N=4096; causal ≥1.7× non-causal. Papers: 2205.14135, 2307.08691.
05 Fused RoPE + RMSNorm (T4, 8 min). Standalone kernels, then fused rmsnorm_rope_kernel that reads x once. On prefill (B=4, S=2048): ≥1.5× unfused eager; on decode (B=1, S=1): ≥2× due to launch-overhead amortization; ≥70% HBM BW. Matches HF Llama reference to 1e-3 FP16. Papers: 1910.07467, 2104.09864.
06 torch.compile deep dive (A10, 10 min). MiniBlock (RMSNorm → QKV → SDPA → residual → SwiGLU). Three modes (default, reduce-overhead, max-autotune); max-autotune ≥1.3× eager. torch._dynamo.explain: inject .item() break, observe reason, fix with torch.where; final ≤1 graph break. Dumps Inductor output_code.py and annotates a fused @triton.jit kernel. Dynamic shapes keep single graph across S ∈ {512, 1024, 2048}.
07 Nsight profiling (local GPU or Colab Pro, 15 min). nsys profile --trace=cuda,nvtx,osrt --stats=true around 64-token decode on SmolLM2-135M with NVTX ranges. ncu --kernel-name "regex:(gemm|flash|rms)" --set full. Expected sm__throughput: 60-90% for GEMM, 40-70% for attention, 20-40% for elementwise/norm; decode aggregate <50% (memory-bound), prefill >65% (compute-bound). Falls back to torch.profiler on free Colab.
08 JAX sharding and pipeline (CPU simulated 8 devices or 2+ GPU, 10 min). Mesh(devices.reshape(4,2), ("data","model")). Megatron-style TP: W1 column-shard (P(None,"model")), W2 row-shard (P("model",None)), shard_map with jax.lax.psum(y_local, "model"). Output matches single-device to 1e-5. Pipeline via jax.lax.scan over microbatches with jax.lax.ppermute; P=4, M=8 → theoretical bubble 3/11 ≈ 0.273; measured within 5% absolute.
Production patterns - 9 notebooks (notebooks/08_production)#
This track exercises real production LLM libraries (Anthropic SDK, LiteLLM, mcp, dspy-ai, inspect-ai, sentence-transformers) on real tasks, with a LIVE/Replay shim per notebook so the whole track runs end-to-end on Colab without API keys: when the relevant key is set the notebook hits the real provider; otherwise it loads recorded responses from notebooks/08_production/_fixtures/. Every notebook ships scoring checks tuned against the recorded fixture so a clean Colab run passes deterministically (46/46 in replay across the track).
01 Anthropic SDK prompt caching (CPU/T4, 1 min LIVE / <1s replay). cache_control={"type": "ephemeral"} on a 25k-char system prompt across 5 questions; measures cache_creation_input_tokens on the first call and cache_read_input_tokens on subsequent calls. Recorded fixture shows ~66% cost saving and ~45% wall-time reduction over the uncached baseline; pricing math pinned to claude-sonnet-4-6 rates (\(3 in, \)15 out, \(3.75 cache write, \)0.30 cache read per 1M). Notes 1-hour cache beta, multi-breakpoint caching, exact-prefix invariant, 1024-token minimum.
02 LiteLLM router and fallbacks (CPU/T4, 1 min LIVE / <1s replay). Same call site through three providers (Sonnet 4.6, Haiku 4.5, GPT-4o-mini); litellm.completion_cost per-call USD; litellm.Router with a fallbacks map recovers from injected primary 529s on 3 of 20 calls. Cost-vs-accuracy scatter across four strategies; routing-strategy and observability hooks documented.
03 Anthropic native tool-use agent (CPU/T4, 1 min LIVE / <1s replay). Real tool_use API with three tools (calculator AST eval, wiki_lookup, get_date) on the same 20 tasks as 04_agents/01. Native tool-use achieves 20/20 success with 0 parse errors, ~1.9× cheaper and ~1.8× faster than prompted-ReAct on the same model. Notes parallel tool calls, cached tool definitions, tool_choice forcing, and computer-use as the same protocol with a bigger surface.
04 Structured outputs head-to-head (CPU/T4, 5 min LIVE / <1s replay). Same Pydantic Person schema, four strategies (prompt-only, validate+retry, Anthropic tool-use, Instructor), 200 prompts, claude-haiku-4-5. Measures schema compliance, mean cost per call, mean LLM calls per successful output. Tool-use and Instructor at 100% compliance; validate+retry at 99.5%; prompt-only at 86%. Notes Outlines for local-model FSM-constrained decoding, BAML/TypeChat alternatives, structured-output-≠-structured-reasoning failure mode.
05 Production hybrid RAG with citations (CPU/T4, 1 min LIVE / <1s replay). 12-passage corpus, BM25 (pure Python, k1=1.2, b=0.75) + dense embeddings (sentence-transformers/bge-small-en-v1.5 in LIVE, baked similarity matrix in replay) + RRF fusion (Σ 1/(60+rank_i)); Anthropic call returns answer with inline [id] citations. Measures retrieval recall@3 and citation grounding (every cited id present in retrieved set; truth-match rate). RRF recall@3 = 100%, citation grounding = 100% on the recorded run. Notes cross-encoder rerankers, BGE-M3 multi-vector, model-graded faithfulness as a follow-up judge.
06 MCP real server and client (CPU, 10 s). Server defined with the official mcp SDK (Server, @server.list_tools(), @server.call_tool()) and stdio transport, spawned as a subprocess by mcp.client.stdio.stdio_client and driven through ClientSession. Tools: calculator (AST eval), wiki_lookup, get_date. Prints the JSON config snippet any MCP-aware editor (Cursor / Claude Code / Cody / Zed) needs to attach to the server. Notes streamable-HTTP transport (replaced HTTP+SSE in 2025-03 spec), OAuth 2.1 + PKCE for HTTP, resources/prompts primitives, real-server templates.
07 DSPy 3 + MIPROv2 optimizer (CPU/T4, 3 min LIVE / <1s replay). dspy.Predict classifier on 20-train / 20-test customer-support tickets, claude-haiku-4-5. MIPROv2 light bootstraps 4 demos and learns an instruction; baseline 80% → optimized 95% on held-out (~120 LM calls / ~$0.014 of optimization spend). Notes MIPROv2 vs BootstrapFewShot tradeoff, faithful-metric requirement, save/load patterns, and the cheap-optimizer / expensive-runtime model split.
08 Inspect AI evaluation harness (CPU/T4, 2 min LIVE / <1s replay). Custom @scorer numeric_match, generate() solver, Sample dataset of 10 math word problems, two models head-to-head (claude-haiku-4-5 vs claude-sonnet-4-6). Per-sample diff highlights where Sonnet wins. Notes the inspect eval CLI, inspect view browser UI, model-graded scorers (model_graded_qa, model_graded_fact), solver composition (chain(prompt_template, generate, self_critique)), and eval_set for budget-bounded model × task sweeps.
09 GPU providers, pricing, and model fit (CPU only, 5 s). Pure-data reference notebook: 13 GPU SKUs (NVIDIA Hopper/Blackwell, AMD MI300X, Intel Gaudi 3, Google TPU v5p, AWS Trainium2) × 10 cloud providers (AWS/GCP/Azure/Oracle/Lambda/CoreWeave/Together AI/Modal/RunPod/Vast.ai); spot/preemptible discount factors per provider; vRAM calculator weights + kv_cache + activations + overhead for arbitrary (params, dtype, layers, kv_heads, head_dim, batch, seq_len); “smallest cluster that fits” recommender for 10 popular open models (SmolLM2-1.7B → DeepSeek-V3) at FP16 and FP8; $ per million output tokens per GPU at realistic 7-8B-class decode throughput. Notes reservations vs spot, egress charges, MIG partitioning, AMD/Intel viability, and when hosted APIs (Together / Fireworks / Anyscale / Replicate / OpenRouter / DeepInfra) beat self-hosting.
Implementation notes and coverage gaps#
v0.2 gap: training track notebooks 03-08#
The six remaining training-track notebooks (tensor parallel, pipeline parallel, LoRA, QLoRA, DPO, GRPO) are fully specified above and targeted for v0.2. Implementers should note:
FSDP2’s
fully_shardhas noauto_wrap_policy/use_orig_paramskwargs — porting from FSDP1 tutorials that pass these will raise.DoRA (weight-decomposed LoRA, 2402.09353) should replace standard LoRA in notebook 03_training/05 as the default first-principles implementation; the PEFT library ships
use_dora=Trueas a drop-in flag since PEFT 0.10.ORPO (odds-ratio preference optimization, 2403.07691) should be added as a
loss_typeoption alongside DPO in notebook 03_training/07 — it removes the reference model, halving training memory.GRPO (notebook 03_training/08, 2501.12948) is now the canonical method for reasoning model training; the verifier framework from τ-bench evaluation (04_agents/07) can be reused as the GRPO reward function.
Open model landscape (May 2026)#
The dominant architecture shift since v0.1 is that MoE is now the default at scale: DeepSeek-V4-Pro (1.6T total / 49B active, 1M-token context), Llama 4 Maverick (400B / 17B active, MoE-first in the Llama line), Qwen3.5-397B-A17B (flagship reasoning/math MoE), and Gemma 4 (multimodal, April 2026) all use sparse MoE. DeepSeek-V4-Flash (284B / 13B active) is a cost-efficient open MoE option for demo notebooks requiring larger active param counts than Qwen3-0.6B. Notable frontier-class open models released in April–May 2026 that further close the open/closed gap: Kimi K2.6 (Moonshot AI; 1T/32B active MoE, 80.2% SWE-bench Verified, Agent Swarm system scaling to 300 domain-specialized sub-agents across 4,000 coordinated steps), Qwen 3.6 Plus (Alibaba; 1M-token context, 78.8% SWE-bench Verified, leads Terminal-Bench 2.0), GLM-5.1 (Z.AI; 754B/MoE, 58.4% SWE-bench Pro), and MiMo-V2.5 (Xiaomi; 310B/15B active MoE, 48T tokens, 1M-token context, fully open-sourced on HuggingFace, frontier-level multimodal agentic capability). Apache 2.0 has become the dominant permissive license for 2026 open models (Gemma 4, Qwen 3.5, Mistral Large 3); DeepSeek V4 ships MIT.
Google I/O 2026 (May 19–22) introduced new proprietary API models relevant to the production patterns track: Gemini 3.5 Flash (GA May 19) combines frontier-level agentic and coding accuracy (Terminal-Bench 2.1: 76.2%, MCP Atlas: 83.6%) with Flash-class speed (~4× faster output than peer frontier models) and a 1M-token context window; pricing \(1.50/\)9.00 per 1M tokens in/out, $0.15 cached-read. Gemini Spark is a new persistent 24/7 agent product built on Gemini 3.5 and the Antigravity framework, able to execute long-horizon tasks autonomously on Google Cloud VMs. Google ADK v1.0 reached stable release across Python, Go, Java, and TypeScript, and the A2A v1.0 protocol is now in production at 150+ organisations following its Linux Foundation adoption.
Additional v0.2 developments to incorporate#
FP4 quantization (NVIDIA Blackwell B200 native):
torch.float4_e2m1fnblock-wise quantization, 9000 FP4 TFLOPS on B200. A new serving notebook could benchmark FP4 vs FP8 on Llama-4-Scout.QuantSpec (arXiv 2502.10424, Apple ML Research): Self-speculative decoding with hierarchical quantized KV cache — combines speculative decoding and KV quantization in one system, reducing both latency and memory. Natural addition to inference track.
IndexCache (2025): Token-level attention index reuse across layers, 15-25% compute reduction on conversational workloads. Could extend the radix cache notebook (01_inference/06).
Agentic RAG / multi-step retrieval: The RAG track covers pipelines; v0.2 should add a notebook on self-correcting retrieval loops where the model critiques its retrieved context and re-queries.
Google ADK v1.0 and A2A v1.0 protocol: Google ADK reached stable v1.0 across Python, Go, Java, and TypeScript in 2026; the A2A (Agent-to-Agent) protocol v1.0 is now in production at 150+ organisations. A new agents notebook could implement a minimal A2A task server, and the existing notebook 04_agents/06 stretch goal (ADK
SequentialAgent) should be updated to reference the v1.0 stable API.Llama 4 MoE / Qwen3 MoE: Both families use fine-grained expert routing; stretch goals in notebook 05_serving/09 could demonstrate Llama-4-Scout routing vs dense-model routing patterns.
PegaFlow external KV cache (Novita AI, May 2026): A Rust-core KV cache storage engine that offloads GPU KV state to host memory or SSD and shares it across nodes via RDMA, integrating with vLLM and SGLang as a drop-in connector. Relevant extension to notebook 01_inference/06 (radix cache) and 05_serving/10 (disaggregated serving) — demonstrates production-grade KV offloading beyond the toy
SharedMemoryapproach.NVIDIA Vera Rubin platform (announced GTC 2026): Rubin GPU (336B transistors, 288 GB HBM4, 50 PFLOPS FP4), Vera CPU (72 ARM Grace cores), NVLink 6 interconnect, targeting 5× Blackwell inference throughput at 10× lower token cost. Partner availability (AWS, GCP, Azure, CoreWeave, Lambda) targeted H2 2026. The GPU pricing notebook (08_production/09) should be updated when Rubin SKUs and pricing are published. The Vera Rubin DSX AI Factory reference design and Omniverse DSX Blueprint reached general availability in July 2026.
MCP 2026-07-28 spec (release candidate): the largest MCP revision since launch — stateless protocol core (no
Mcp-Session-Id),initialize/initializedhandshake removed, authorization rewritten around OAuth/OIDC RFCs, and a new reverse-DNS extensions framework carrying the Tasks extension and MCP Apps. Still in release-candidate status as of mid-July 2026; the final text ships July 28, 2026. The Enterprise-Managed Authorization extension has already reached stable status (adopted by Anthropic, Microsoft, Okta) ahead of the core spec. Notebook 04_agents/05 (MCP server + client) and 08_production/06 (real MCP server) should migrate their handshake and session-handling code off the pre-2026-07-28 assumptions once the spec finalizes.Benchmark saturation (BenchLM.ai, July 2, 2026): 37% of 154 percentage-scaled benchmarks are now saturated (top model ≥ 90%); GSM8K is effectively solved at 99%. Reinforces that the eval track’s numerical thresholds should keep targeting 2026-accessible small open-weight models rather than frontier models, which are increasingly ceiling-bound on the older benchmarks.
Anthropic mid-2026 refresh: Claude Sonnet 5 (June 30) and Claude Fable 5 (GA July 1, SWE-bench Pro 80.3%) are the new production-track model defaults referenced in 08_production/index.md, at roughly half the per-token price of their predecessors.
New frontier entrants (July 2026): xAI’s Grok 4.5 (July 8) is its first coding/agentic-focused model, priced over 60% below Opus-class models. OpenAI’s GPT-5.6 family (Sol, Terra, Luna; July 9) sets a new state of the art on the Artificial Analysis Coding Agent Index via Sol, edging out Claude Fable 5 at roughly a third of the cost. Both are now referenced as production-track model options in 08_production/index.md.
Spec-level risks#
2026 pin stack compatibility: sentence-transformers 5.4.1 + pylate 1.1.7 have not been validated against
transformers5.x; RAG notebooks pintransformers==4.46.3. Some training notebooks want 4.52.x for newer TRL features — use one environment per track rather than a monolithic requirements file.vLLM V2 engine (
vllm>=0.8) uses an async-first scheduler and breaks theengine_use_rayandworker_use_rayarguments from V1. Notebook 05_serving/11 (observability) was written against V1 and may need minor API updates.
v0.3 roadmap (planned extensions)#
Four topic areas represent the most important coverage gaps given the 2025-2026 state of the field:
Test-time compute and reasoning models. The dominant 2025 scaling axis
shifted from training-time parameter growth to inference-time chain-of-thought.
A planned 01_inference/11_inference_time_scaling.ipynb would implement: (a)
best-of-N with a process reward model, (b) MCTS-style tree search over
reasoning steps, and (c) the budget forcing / “wait” token trick from S1
(2501.10921). Scoring threshold: best-of-N@16 ≥ 10 pp over greedy on GSM8K;
tree search ≥ best-of-N at same token budget. Dependencies: a fast CPU-friendly
reward model (Qwen/Qwen2.5-Math-RM-72B is too large; a fine-tuned
SmolLM2-360M reward head should be the fallback).
BitNet and sub-2-bit weight quantization. Microsoft’s BitNet b1.58-2B-4T
(April 2025, trained from scratch with ternary weights {-1,0,+1}) achieves
45 tok/s on Apple M2 CPU at 0.15 kWh/M tokens — ~15× more efficient than FP16
Llama 3 8B on the same hardware. A planned 05_serving/12_bitnet_ternary.ipynb
would: implement a from-scratch BitLinear with absmean activation
quantization and ternary weight rounding; benchmark PPL vs INT4/INT8 on
SmolLM2-135M; and demonstrate CPU-native inference via bitnet.cpp. Scoring:
BitLinear PPL within 0.5 of FP16 on WikiText-2 after ternary-aware training
for 500 steps; memory ≤ 2× INT4 (ternary packing benefit); CPU tok/s ≥ 2×
FP16 at batch=1.
Multimodal / Vision-Language Models. A new 09_multimodal/ track (5
notebooks) would cover: (01) SigLIP 2 dual-encoder fine-tune for image–text
retrieval on COCO captions; (02) LLaVA-style cross-modal projection from
scratch (vision encoder → MLP connector → LLM token space); (03) VLM
evaluation with MMBench / POPE / HallusionBench; (04) Phi-4-Multimodal
document understanding on DocVQA; (05) Vision-Language-Action (VLA) model
overview with LeRobot as a hands-on integration target. All five are
Colab T4-feasible with SmolVLM-256M or InternVL2-1B as the base.
Safety and red-teaming. A planned 10_safety/ track (4 notebooks) would
cover: jailbreak evaluation with HarmBench, constitutional AI self-critique,
watermarking (Kirchenbauer et al. 2302.03162 hard red-list), and output
toxicity scoring with Perspective API / Detoxify.
Environment compatibility notes#
The 2026 version pin stack is not fully mutually compatible: sentence- transformers==5.4.1 + pylate==1.1.7 have not been validated against
transformers==5.x, so RAG notebooks pin transformers==4.46.3, while
training notebooks may want 4.51.x for newer TRL features — use one
conda/venv per track rather than a monolithic install.
RAGAS 0.4.3 (Jan 2026) broke the 0.2.x metric API used in notebook
02_rag/09; the spec pins ragas==0.2.14 deliberately and flags the upgrade
as a v0.2 stretch goal.
vLLM V1 (vllm>=0.8) brings a redesigned execution engine. NVIDIA Dynamo
(GTC March 2025) is now an optional backend for disaggregated serving that
replaces the multiprocessing.SharedMemory + torch.save handoff used in
05_serving/10; enabling it requires nvidia-dynamo and nixl extras and
two A100-class GPUs, so the notebook retains the pure-Python shm path as the
default and documents the Dynamo upgrade path in the exercises.
The 70 notebooks are runnable end-to-end on a free Colab T4 with two
exceptions: FlashAttention notebooks (inference-05, gpu-04) require Ampere+,
and Nsight profiling (gpu-07) needs a local GPU or Colab Pro (the spec includes
a torch.profiler fallback). Notebook 05_serving/06 gates FP8 on
torch.cuda.get_device_capability() >= (8,9) and simulates with block-wise
torch.float8_e4m3fn casts otherwise. The Production track’s 9 notebooks all
run on CPU via recorded fixtures when API keys are not set. Total estimated
cumulative runtime across the curriculum on a single L4 is approximately
6-8 hours of wall-clock compute plus model-download time.
Conclusion#
The 70 specifications form a production-ready blueprint for an LLM systems curriculum that is simultaneously deep (each notebook either reproduces a named paper’s mechanism from scratch and validates against a production tool, or exercises the production library directly with measured cost / latency / compliance numbers) and broad (the eight tracks cover the full stack from CUDA warps to GPU pricing reference). Three design decisions differentiate it from typical “LLM from scratch” curricula: every scoring check is numerical and machine-checkable (no human-graded essays), every notebook has two paths - pedagogical from-scratch and production toolchain - enabling direct comparison, and the hardware bar is deliberately low (T4-compatible demo models throughout) so the curriculum is accessible without a datacenter.
The LLM systems landscape has shifted rapidly since the spec was first written. MMLU is now saturated at 88-94% by frontier models; the live eval frontier has moved to GPQA-Diamond, ARC-AGI-2, SWE-bench Verified, and AIME 2025/2026. MoE has become the dominant frontier architecture (DeepSeek-V3, Llama 4 Scout/Maverick, Qwen3-235B-A22B). vLLM V2 and SGLang 0.4 compete for the serving workbench role that TGI previously occupied. Reasoning model training (GRPO, DPO) and inference-time compute scaling are now mainstream. The curriculum’s architecture — numerical checks, two-path pedagogy, T4-accessible models — remains sound; stretching the model shortlist to include Qwen3-1.7B (hybrid thinking) and Llama-4-Scout (MoE, 10M context) is the most impactful near-term update.
The primary risk is environment fragility given 2026-era version pins across 40+ packages; the mitigation is one virtualenv per track rather than a monolithic requirements file, and explicit stretch goals for users who want bleeding-edge variants (RAGAS 0.4, Transformers 5.x, FSDP HYBRID_SHARD, EAGLE-3, QuantSpec, Google ADK/A2A, FP4 quantization).