Inference engines

What you’ll learn in this part

• Why decoding is memory-bound and prefill is compute-bound, and what that means for every optimisation that follows.

• The KV-cache byte formula and how paging reduces waste from 80 % to under 5 %.

• Scheduler designs: continuous batching, radix prefix cache, SARATHI chunked prefill, disaggregated prefill/decode.

• Speculative decoding, Medusa heads, EAGLE tree speculation.

• FlashAttention-2 drop-in for any transformer layer.

For H100/Hopper context: FlashAttention-3 (arXiv 2407.08608) extends these ideas with async TMA pipelining and FP8 support, reaching 740 TFLOPs/s, 2× faster than FA2 on the same chip. The 2024-2026 reasoning model wave (o1, o3, DeepSeek-R1, GPT-5.5 Thinking) introduces a new axis (test-time compute / inference-time scaling) where spending more generation tokens trades compute for accuracy; see the glossary for orientation. GPT-5.5 Thinking (May 2026) is the first unified model to auto-route between fast and extended chain-of-thought at inference, retiring the standalone o-series numbering.

On the production side, vLLM’s Model Runner V2 (MRV2), enabled via VLLM_USE_V2_MODEL_RUNNER=1 in vLLM ≥ 0.20, delivers 56% higher throughput on GB200 via GPU-native Triton kernels and zero-CPU-sync speculative decoding. SGLang is a competitive or leading alternative for MoE models, structured outputs, and speculative decoding workloads; SGLang v0.5 on GB300 NVL72 demonstrated 25× the inference performance of the H100 baseline (Feb 2026).

Key terms used in this part

• prefill and decode are the two inference phases; most optimizations here target one phase more than the other.

• KV cache and PagedAttention are the memory primitives behind modern LLM serving.

• continuous batching, chunked prefill, and disaggregated prefill/decode are scheduler/system design patterns used to improve throughput and tail latency.

• speculative decoding uses a draft/verify pattern to reduce target-model decode work.

Reading order

Prerequisites: Part I (GPU architecture tour + roofline).

1. 01_autoregressive_decoding_kv_cache: the KV cache is memoised Fibonacci, with numbers.

2. 02_attention_roofline: attention from scratch + its arithmetic intensity.

3. 03_pagedattention_block_allocator: vLLM’s block allocator in pure Python.

4. 04_continuous_batching_orca: iteration-level scheduling.

5. 05_flashattention2_triton: FA2 kernel wrapped in an nn.Module. Ampere+ for kernel execution; falls back on CPU.

6. 06_radix_prefix_cache: SGLang’s radix tree.

7. 07_speculative_decoding: the rejection rule and its closed-form speedup.

8. 08_medusa_eagle_tree_speculation: tree verification of draft candidates.

9. 09_sarathi_chunked_prefill: co-schedule decodes with prefill chunks.

10. 10_disaggregated_prefill_decode: KV handoff between separate GPU pools.

See also

Part III turns the compute/memory tradeoffs here into serving-level economics: goodput, SLO attainment, autoscaling.

NVIDIA Dynamo is the 2025 production successor to the hand-rolled disaggregated shm approach in chapter 10; the notebook documents the Dynamo upgrade path in its exercises.

Coming in v0.3

11: Inference-time compute scaling will implement best-of-N with a process reward model, MCTS-style tree search over reasoning steps, and the “wait” budget-forcing trick from S1 (2501.10921). This is the dominant 2025-2026 quality-scaling axis for reasoning models like DeepSeek-R1, GPT-5.5 Thinking, and QwQ.

Recent developments (2025-2026)

• EAGLE-3 (arXiv 2503.01840): multi-layer hidden-state fusion for draft candidates; surpasses EAGLE-2 on long-context tasks. Referenced as a stretch goal in 08_medusa_eagle_tree_speculation.

• QuantSpec (arXiv 2502.10424, Apple ML Research): combines self-speculative decoding with a hierarchical quantized KV cache, targeting both latency and memory simultaneously. It is a natural v0.2 addition to the speculative decoding notebooks.

• IndexCache (2025): reuses token-level attention indices across transformer layers and requests, cutting compute 15-25% on conversational workloads with no measurable quality loss.

• Llama 4 Scout 10M-token context: demonstrates practical very-long-context inference; stretch goal for 06_radix_prefix_cache (cache hit-rate analysis at 1M+ token prefixes).

• Mercury 2 (Inception): a production diffusion-based LLM that generates tokens in parallel rather than auto-regressively, reaching speeds above 1,000 tokens/sec. It contradicts the assumption that all LLM inference is sequential, and is relevant to the roofline analysis in chapters 1-2 as an alternative compute model.

• Inference now ~2/3 of AI compute: production inference has grown from roughly 1/3 of AI compute in 2023 to ~2/3 in 2026. Inference cost has fallen roughly 1,000× since late 2022, making large-scale agentic loops economically viable. Baseten’s \(1.5B funding round (a \)13B valuation) confirms open-source model serving is a major capital destination.

• Cloudflare Infire: Cloudflare’s custom inference engine that runs LLMs across multiple GPUs more efficiently, reduces memory usage, and starts models more quickly. It ships alongside Unweight, a weight compression system that reduces LLM model sizes 15-22% without measurable accuracy loss.

• Mixed-precision KV-cache quantization: 2026 research (e.g. PM-KVQ, ICLR 2026) moves past uniform INT8/FP8 KV quantization toward progressive, per-layer/per-token bit allocation, spending more bits where retrieval accuracy is most sensitive. vLLM’s --kv-cache-dtype fp8 now runs the QK/ScoreV attention matmuls themselves in FP8, not just cache storage — directly relevant to the KV cache mechanics in 02_kv_cache_from_scratch and the roofline analysis in 03_attention_roofline.