Roofline analysis for LLM serving

Roofline analysis for LLM serving#

Track 05 - Serving · Notebook 01 · Runtime: ≈30 s on CPU

Prerequisites: the GPU-architecture tour (07_gpu/01). You know what arithmetic intensity is and why it matters.

What you’ll know by the end: how to glance at an LLM workload (prefill? decode? batch size? context length?) and immediately say whether it’s compute-bound or memory-bound, and which GPU would run it fastest for the money.

The whole serving-economics argument in one picture#

Every LLM company you’ve heard of lives or dies by this question: for a given GPU, how many concurrent users can we serve before latency spikes?

The answer is encoded in one picture - the roofline. Once you know where your workload sits on the roofline, you know:

Whether buying a fatter GPU will help (compute-bound → yes, memory-bound → no).
Whether batching will help (always raises arithmetic intensity).
Whether caching / quantisation will help (reduces bytes read, helps memory-bound workloads).
Roughly how many tokens/second you can sustain per GPU.

No roofline magic - just min(peak_compute, bandwidth × intensity). The power is in knowing where your workload’s intensity lies.

Quick recap from the GPU track#

A GPU’s peak performance for any kernel is bounded by:

\[ \text{achievable\_TFLOPs} = \min(\text{peak\_compute},\; \text{peak\_bandwidth} \times \text{arithmetic\_intensity}) \]

Arithmetic intensity (AI) = FLOPs your kernel does per byte it reads from HBM. A higher number means more math per byte, which is good - it means you can saturate the compute units.
The crossover point - where compute peak equals bandwidth times intensity - is the ridge. Workloads below the ridge are bandwidth-bound; workloads above are compute-bound.

For a T4 the ridge is about 200 FLOPs/byte. For an A100 it’s ~150. For an H100 SXM5 it’s ~295 (FP16 dense / HBM3). All of them are well above the intensity of single-stream decode on any real LLM, which is why decode is universally memory-bound regardless of which GPU you run on.

from llm_systems_cookbook.nb import bootstrap

import numpy as np

s = bootstrap("05_serving_01_roofline_analysis")

Step 1 - The devices we’ll reason about#

Hardcoded peaks for five common inference GPUs, straight from the NVIDIA product briefs. For each we derive the ridge intensity.

Notice the compute / bandwidth ratio. A100 and H100 have enormous compute ceilings but their bandwidth doesn’t scale linearly - so the ridge is higher, which means you need more arithmetic intensity (longer context, bigger batch) to saturate them. This is why the “bigger GPU = more throughput” equation isn’t always true.

DEVICES: dict[str, dict[str, float]] = {
    "T4":   {"bw": 320.0,  "compute": 65.0},
    "L4":   {"bw": 300.0,  "compute": 120.0},
    "A10G": {"bw": 600.0,  "compute": 125.0},
    "A100": {"bw": 2039.0, "compute": 312.0},
    "H100": {"bw": 3350.0, "compute": 989.0},  # SXM5, FP16 tensor-core dense
}


def ridge_intensity(bw_gbps: float, compute_tflops: float) -> float:
    return compute_tflops * 1e12 / (bw_gbps * 1e9)


for name, d in DEVICES.items():
    d["ridge"] = ridge_intensity(d["bw"], d["compute"])
    print(f"{name:>5}  peak={d['compute']:>4.0f} TFLOPs  bw={d['bw']:>4.0f} GB/s  ridge={d['ridge']:5.1f} FLOPs/byte")

Step 2 - Three canonical LLM workloads#

Each one is a different point on the arithmetic-intensity spectrum.

Prefill (processing the prompt)#

You receive a 1024-token prompt. You want to do one giant forward pass that computes all attention scores and fills the KV cache. Every weight matrix is used in one big matmul - massive FLOPs, modest HBM traffic.

\[\text{AI}_\text{prefill} \approx \text{context\_length} / 3 \text{ FLOPs/byte (for square matmuls)}\]

At context 1024 that’s ~340 FLOPs/byte. Well above every GPU’s ridge. This workload lives in the compute-bound regime.

Decode (generating one token at a time)#

After prefill, you generate tokens one at a time. Each decode step reads the entire model weights plus the entire KV cache from HBM, does one token’s worth of math, and spits out one output.

\[\text{AI}_\text{decode, B=1} \approx 1 \text{ FLOP/byte}\]

Yes, one. The math is dwarfed by the bytes. This is the memory-bound wall you hit when trying to make LLM decoding faster - and it’s why decode latency is basically model_size / HBM_bandwidth, which on an H100 with a 70 B fp16 model is 140 GB / 3.35 TB/s ≈ 42 ms per token.

Batched decode#

Batching saves us. Run decode for 32 users simultaneously: you read the weights once, do 32× more compute. Intensity goes up roughly linearly with batch size. On T4 at batch 32 you land at ~30 FLOPs/byte

still below the ridge, but now you’re pushing the bandwidth ceiling instead of stalling on it.

def matmul_intensity(m: int, k: int, n: int, dtype_bytes: int = 2) -> float:
    flops = 2 * m * k * n
    bytes_ = dtype_bytes * (m * k + k * n + m * n)
    return flops / bytes_


def attention_intensity(batch: int, n_heads: int, head_dim: int, seq_q: int, seq_k: int,
                         dtype_bytes: int = 2) -> float:
    flops = 4 * batch * n_heads * seq_q * seq_k * head_dim
    bytes_ = dtype_bytes * batch * n_heads * head_dim * (seq_q + 2 * seq_k + seq_q)
    return flops / bytes_


def decode_step_intensity(batch: int, params: int, n_kv_heads: int, head_dim: int,
                           seq_k: int, dtype_bytes: int = 2) -> float:
    '''End-to-end decode AI: weights read once + per-request KV read.'''
    weight_flops = 2 * params * batch
    weight_bytes = dtype_bytes * params
    attn_flops = 4 * batch * n_kv_heads * seq_k * head_dim
    attn_bytes = dtype_bytes * batch * n_kv_heads * head_dim * (1 + 2 * seq_k + 1)
    return (weight_flops + attn_flops) / (weight_bytes + attn_bytes)


PARAMS_7B = 7_000_000_000
WORKLOADS: dict[str, float] = {
    "prefill_B1_T1024_matmul_7B":  matmul_intensity(1024, 4096, 11008),
    "prefill_B8_T1024_matmul_7B":  matmul_intensity(8 * 1024, 4096, 11008),
    "decode_B1_T1_kv2048":          decode_step_intensity(1,  PARAMS_7B, 8, 128, 2048),
    "decode_B32_T1_kv2048":         decode_step_intensity(32, PARAMS_7B, 8, 128, 2048),
    "flash_attn_B1_H32_T2048":      attention_intensity(1, 32, 128, 2048, 2048),
}
for name, ai in WORKLOADS.items():
    print(f"{name:<35}  AI = {ai:>8.1f} FLOPs/byte")

Step 3 - Draw the roofline and drop the workloads onto it#

Five GPUs, five workloads, one plot. Find your workload on the x-axis, read up to the lowest roof below any given GPU’s ceiling - that’s your achievable throughput.

The dashed vertical lines mark each workload’s intensity. Anywhere a line crosses below a GPU’s roof corner, that GPU is bandwidth-bound for that workload; anywhere it’s above, the GPU is compute-bound.

import matplotlib.pyplot as plt

fig, ax = plt.subplots(figsize=(8, 5))
x = np.logspace(-1, 3.5, 400)
for name, d in DEVICES.items():
    y = np.minimum(d["bw"] * 1e9 * x / 1e12, np.full_like(x, d["compute"]))
    ax.loglog(x, y, label=f"{name} (ridge {d['ridge']:.0f})")

for w_name, ai in WORKLOADS.items():
    ax.axvline(ai, linestyle=":", alpha=0.3, color="black")

ax.set_xlabel("arithmetic intensity (FLOPs / byte)")
ax.set_ylabel("throughput (TFLOPs)")
ax.set_title("Roofline: LLM serving workloads vs common inference GPUs")
ax.legend()
ax.grid(True, which="both", alpha=0.3)
plt.tight_layout()
plt.show()

Step 4 - The five assertions that matter#

If you really understand the roofline, each of these should feel inevitable:

Prefill at batch 1 is compute-bound on T4. Even the smallest prefill has enough math to saturate a modest GPU.
Prefill at batch 8 is compute-bound on A100. Bigger GPU, but we also threw in more prefill → still above the ridge.
Single-stream decode is memory-bound everywhere. AI ≈ 1 is far below every GPU’s ridge.
Batching raises decode intensity. Going from B=1 to B=32 multiplies intensity by roughly the batch size (weights are read once, amortised).
A 2048-context FlashAttention tile is above T4’s ridge. Real attention kernels hit high enough intensity to be compute-bound on inference GPUs.

s.check(
    "prefill_compute_bound_on_T4",
    lambda: WORKLOADS["prefill_B1_T1024_matmul_7B"] > DEVICES["T4"]["ridge"],
    msg=f"prefill AI={WORKLOADS['prefill_B1_T1024_matmul_7B']:.1f}  T4 ridge={DEVICES['T4']['ridge']:.1f}",
)
s.check(
    "prefill_compute_bound_on_A100",
    lambda: WORKLOADS["prefill_B8_T1024_matmul_7B"] > DEVICES["A100"]["ridge"],
    msg=f"prefill-B8 AI={WORKLOADS['prefill_B8_T1024_matmul_7B']:.1f}  A100 ridge={DEVICES['A100']['ridge']:.1f}",
)
s.check(
    "decode_bs1_memory_bound_everywhere",
    lambda: all(WORKLOADS["decode_B1_T1_kv2048"] < d["ridge"] for d in DEVICES.values()),
    msg=f"decode AI={WORKLOADS['decode_B1_T1_kv2048']:.1f}",
)
s.check(
    "batching_raises_decode_intensity",
    lambda: WORKLOADS["decode_B32_T1_kv2048"] > 2 * WORKLOADS["decode_B1_T1_kv2048"],
    msg=f"B=1 AI={WORKLOADS['decode_B1_T1_kv2048']:.1f}  B=32 AI={WORKLOADS['decode_B32_T1_kv2048']:.1f}",
)
s.check(
    "flash_attn_intensity_above_T4_ridge",
    lambda: WORKLOADS["flash_attn_B1_H32_T2048"] > DEVICES["T4"]["ridge"],
    msg=f"FA AI={WORKLOADS['flash_attn_B1_H32_T2048']:.1f}  T4 ridge={DEVICES['T4']['ridge']:.1f}",
)

One picture: workloads dropped on the A100 roof#

Same roofline, but now each workload is a dot at its measured intensity and the throughput it would achieve on A100. Dots to the left of the ridge ride the bandwidth slope; dots to the right are pinned at the compute ceiling. The prefill / decode split falls out of the geometry.

import matplotlib.pyplot as plt

dev = DEVICES["A100"]
x_ai = np.logspace(-1, 3.5, 400)
roof = np.minimum(dev["bw"] * 1e9 * x_ai / 1e12, np.full_like(x_ai, dev["compute"]))

fig, ax = plt.subplots(figsize=(7.2, 4.2))
ax.loglog(x_ai, roof, color="black", lw=1.5,
          label=f"A100 roof ({dev['compute']:.0f} TFLOPs, {dev['bw']:.0f} GB/s)")
ax.axvline(dev["ridge"], color="tab:gray", linestyle=":", label=f"ridge {dev['ridge']:.0f}")

colors = {"prefill": "tab:red", "decode": "tab:blue", "flash": "tab:green"}
for name, ai in WORKLOADS.items():
    kind = "prefill" if "prefill" in name else ("decode" if "decode" in name else "flash")
    y = min(dev["bw"] * 1e9 * ai / 1e12, dev["compute"])
    ax.scatter([ai], [y], s=80, color=colors[kind], zorder=5, edgecolor="black", lw=0.5)
    ax.annotate(name.replace("_", " "), (ai, y), xytext=(5, 5),
                textcoords="offset points", fontsize=7.5)

ax.set_xlabel("arithmetic intensity (FLOPs / byte)")
ax.set_ylabel("achievable throughput on A100 (TFLOPs)")
ax.set_title("workloads on the roof: decode stuck on the slope, prefill at the ceiling")
ax.legend(loc="lower right", fontsize=8)
ax.grid(True, which="both", alpha=0.3)
plt.tight_layout()
plt.show()

Exercises#

Your serving bill. Pick a model (say Llama-3-70B fp16 = 140 GB) and a GPU (say 8× H100 with 80 GB each - 640 GB total). Compute the minimum decode latency per token at batch 1, 8, 32. At what batch size does decode cross the ridge on H100?
Quantise the weights. Replace fp16 (2 bytes) with fp8 (1 byte) everywhere. Weight bytes halve, decode AI doubles. How does that move the line on the plot?
Short vs long context decode. Plot decode AI as a function of seq_k (1024, 2048, 4096, 8192) at B=1. Why does AI drop slightly as context grows? (Hint: the KV read scales with seq_k.)
Find the sweet-spot batch size. For a given GPU + model, what batch size just touches the ridge? Any batch smaller and you’re wasting compute; any bigger and the KV memory eats too much.

What’s next#

Notebook 02 in this track (02_kv_cache_variants_mha_gqa_mla) attacks the decode intensity number directly: multi-query, grouped-query, and latent attention all shrink the KV read, moving the decode point to the right on the roofline.

s.summary()
s.save()