GPU availability, pricing, and picking a fit for your model#
Track 08 - Production · Notebook 09 · Runtime: <5s · CPU-only
A practical reference for “what GPU should I rent and where?”. Cards in production today, indicative on-demand and spot prices across ten cloud providers (mid-2026), and a memory calculator that maps a model size to the smallest cluster that holds it.
from llm_systems_cookbook.nb import bootstrap
import pandas as pd
s = bootstrap("08_production_09_gpu_providers_pricing_and_model_fit")
pd.set_option("display.float_format", lambda x: f"{x:.2f}" if abs(x) < 1000 else f"{x:,.0f}")
GPU types in production today#
Memory governs which models fit; bf16 TFLOPS governs training and prefill throughput; fp8/fp4 TFLOPS governs decoding throughput on modern stacks. Numbers are vendor headline figures; real workloads see 30-60% of peak.
GPUS = pd.DataFrame([
# name vendor mem_GB bf16_TF fp8_TF year notes
("B200", "NVIDIA", 192, 2250, 4500, 2025, "Blackwell, HBM3e, NVL72 racks; FP4=9000 TFLOPS"),
("B100", "NVIDIA", 192, 1750, 3500, 2025, "Blackwell, lower-power air-cooled variant"),
("H200", "NVIDIA", 141, 989, 1979, 2024, "Hopper refresh, HBM3e, 4.8 TB/s bandwidth"),
("H100 SXM", "NVIDIA", 80, 989, 1979, 2023, "Hopper, mainstream training"),
("L40S", "NVIDIA", 48, 362, 733, 2023, "Inference workhorse, Ada Lovelace, GDDR6"),
("L4", "NVIDIA", 24, 121, 242, 2023, "Cheap inference, low TDP, FP8 support"),
("A100 80GB", "NVIDIA", 80, 312, 0, 2021, "Ampere, no FP8, still widely available"),
("A100 40GB", "NVIDIA", 40, 312, 0, 2020, "Older, plentiful spot inventory"),
("RTX 4090", "NVIDIA", 24, 165, 330, 2022, "Consumer Ada Lovelace, Vast/RunPod community"),
("MI325X", "AMD", 256, 1300, 2600, 2024, "ROCm 6.3+, 256 GB HBM3, vLLM + SGLang ready"),
("MI300X", "AMD", 192, 1300, 2600, 2024, "ROCm 6+, widely available on niche clouds"),
("Gaudi 3", "Intel", 128, 1835, 1835, 2024, "PyTorch via habana_frameworks; AWS DL2q"),
("TPU v5p", "Google", 95, 459, 0, 2024, "JAX/Pax native, no HF transformers"),
("Trainium2", "AWS", 96, 667, 1335, 2024, "Neuron SDK, optimum-neuron required"),
], columns=["gpu", "vendor", "mem_gb", "bf16_tflops", "fp8_tflops", "year", "notes"])
print(GPUS.to_string(index=False))
Where to rent them — on-demand $/hour for 1 GPU (mid-2026)#
Rough on-demand list prices for a single GPU in a single instance. Hyperscalers tend to be 2-3× the niche providers (Lambda, RunPod, Vast) but include enterprise networking, support contracts, and committed-use discounts of 30-60%. NaN means the SKU is not offered by that provider.
_NAN = float("nan")
# Indicative on-demand $/hr (mid-2026). AWS cut ML instance prices ~44% in June 2025;
# H100/H200 spot on niche clouds has converged to ~$1.20-2.50/hr.
PRICES = pd.DataFrame([
# provider B200 H200 H100 A100-80 L40S L4 4090
("AWS", 4.50, 4.00, 2.90, 1.74, 1.65, 0.78, _NAN),
("GCP", 4.40, 3.80, 2.65, 1.90, 1.60, 0.53, _NAN),
("Azure", 4.60, 4.10, 2.95, 1.95, 1.70, 0.90, _NAN),
("Oracle Cloud", 4.20, 3.50, 2.30, 1.60, 1.45, 0.85, _NAN),
("Lambda Labs", 3.80, 2.80, 1.99, 1.29, _NAN, 0.79, _NAN),
("CoreWeave", 4.10, 3.40, 2.39, 1.89, 1.49, _NAN, _NAN),
("Together AI", _NAN, _NAN, 1.49, 1.10, _NAN, _NAN, _NAN),
("Modal", _NAN, _NAN, 3.80, 2.70, 2.30, 0.90, _NAN),
("RunPod", 3.20, 2.49, 1.49, 1.09, 0.99, 0.39, 0.29),
("Vast.ai", _NAN, _NAN, 1.22, 0.90, 0.80, _NAN, 0.22),
], columns=["provider", "B200", "H200", "H100", "A100-80", "L40S", "L4", "4090"])
print(PRICES.to_string(index=False))
Spot / preemptible discounts#
For any cluster you can checkpoint or restart cheaply, spot is typically 40-70% off on-demand. Niche providers (RunPod, Vast) are already close to spot prices, so the spot premium there is small (10-20%). Hyperscaler spot instances can disappear in minutes; niche providers often guarantee a notice window.
SPOT_DISCOUNT = pd.DataFrame([
("AWS", 0.65), # ~35% of on-demand
("GCP", 0.40), # preemptible
("Azure", 0.45), # spot, very volatile
("Oracle Cloud", 0.50),
("Lambda Labs", 0.85), # reserved-only at the deepest discounts
("CoreWeave", 0.60),
("RunPod", 0.85), # community cloud is the cheap tier
("Vast.ai", 0.80), # already a marketplace; "spot" = interruptible
], columns=["provider", "spot_factor"])
spot_h100 = (
PRICES[["provider", "H100"]]
.merge(SPOT_DISCOUNT, on="provider")
.assign(spot_h100_per_hr=lambda d: d["H100"] * d["spot_factor"])
.sort_values("spot_h100_per_hr")
[["provider", "H100", "spot_factor", "spot_h100_per_hr"]]
)
print("H100 spot price ranking ($/hr):")
print(spot_h100.to_string(index=False))
Memory budget for a model#
For inference, the back-of-envelope is weights + kv_cache + activations + scratch. bytes_per_param is 2 (bf16/fp16), 1 (fp8/int8), 0.5 (int4/nf4). KV cache scales linearly with batch × seq_len; for long contexts it overtakes weight memory.
def vram_inference(
params_b: float,
weight_dtype_bytes: float,
layers: int, kv_heads: int, head_dim: int,
batch: int, seq_len: int,
kv_dtype_bytes: float = 2.0,
) -> dict:
"""Return a memory budget dict in GB. Numbers are theoretical floors."""
weights_gb = params_b * 1e9 * weight_dtype_bytes / 1024**3
kv_cache_gb = 2 * layers * kv_heads * head_dim * batch * seq_len * kv_dtype_bytes / 1024**3
activations_gb = max(0.5, params_b * 0.05) # ~5% of weights for hidden states & softmax buffers
overhead_gb = 1.5 # CUDA context, allocator slack, framework
total = weights_gb + kv_cache_gb + activations_gb + overhead_gb
return {
"weights_gb": round(weights_gb, 2),
"kv_cache_gb": round(kv_cache_gb, 2),
"activations_gb": round(activations_gb, 2),
"overhead_gb": round(overhead_gb, 2),
"total_gb": round(total, 2),
}
# Worked example: Llama-3.1-8B, fp16 weights, 8 KV heads, head_dim 128, 32 layers, batch 4, 4k ctx
budget = vram_inference(8.0, 2.0, layers=32, kv_heads=8, head_dim=128, batch=4, seq_len=4096)
print(json.dumps(budget, indent=2)) if (json := __import__("json")) else None
Picking the smallest cluster that fits#
Given a model and a deployment dtype, the recommendation is the cheapest GPU SKU whose aggregate memory exceeds the budget by 20% headroom. For weight_dtype_bytes < 2.0 we assume the model has been pre-quantised (GPTQ/AWQ/FP8/NF4); the runtime cost shifts from memory to a small dequant kernel hit.
# Representative open-weights models in 2026, including new Llama 4 and Qwen3 families.
# MoE models (Llama 4, DeepSeek-V3) list active params in params_b for KV/activation math;
# total params determine weight memory, noted in the "use" column.
MODELS = pd.DataFrame([
("SmolLM2-1.7B", 1.7, 24, 16, 64, "ultra-cheap edge / on-device"),
("Qwen3-1.7B", 1.7, 28, 8, 128, "Qwen3 with hybrid thinking mode"),
("Qwen2.5-7B", 7.6, 28, 8, 128, "agents, RAG, classification"),
("Llama-3.1-8B", 8.0, 32, 8, 128, "general workhorse (dense)"),
("Llama-4-Scout-17B", 17.0, 48, 8, 128, "MoE, 17B active / 109B total, 10M ctx"),
("Qwen2.5-14B", 14.7, 48, 8, 128, "stronger reasoning, dense"),
("Mistral-Small-22B", 22.2, 56, 8, 128, "long-context Q&A (dense)"),
("Qwen3-30B-A3B", 3.0, 48, 8, 128, "MoE, 3B active / 30B total"),
("Qwen2.5-32B", 32.5, 64, 8, 128, "frontier-on-budget (dense)"),
("Llama-3.3-70B", 70.0, 80, 8, 128, "frontier open weights (dense)"),
("Llama-4-Maverick-17B", 17.0, 48, 8, 128, "MoE, 17B active / 400B total"),
("Mixtral-8x22B", 141.0, 56, 8, 128, "MoE, 39B active"),
("Qwen3-235B-A22B", 22.0, 94, 64, 128, "MoE, 22B active / 235B total"),
("Llama-3.1-405B", 405.0, 126, 8, 128, "training-grade serving (dense)"),
("DeepSeek-V3", 671.0, 61, 128, 128, "MoE, 37B active, FP8 native"),
], columns=["model", "params_b", "layers", "kv_heads", "head_dim", "use"])
def recommend_cluster(
row: pd.Series, weight_dtype_bytes: float, batch: int, seq_len: int,
) -> tuple[str, int, float]:
budget = vram_inference(
row.params_b, weight_dtype_bytes,
layers=int(row.layers), kv_heads=int(row.kv_heads), head_dim=int(row.head_dim),
batch=batch, seq_len=seq_len,
)
needed_gb = budget["total_gb"] * 1.20 # headroom
candidates = [(g.gpu, g.mem_gb, PRICES[g.gpu].dropna().min())
for _, g in GPUS.iterrows() if g.gpu in PRICES.columns]
candidates.sort(key=lambda c: (c[2] if c[2] == c[2] else 1e9, c[1]))
for name, mem, hourly in candidates:
if mem >= needed_gb:
return name, 1, hourly
biggest = max(candidates, key=lambda c: c[1])
n = -(-int(needed_gb) // int(biggest[1])) # ceil divide
return biggest[0], n, biggest[2] * n
fits = []
for _, row in MODELS.iterrows():
fp16 = recommend_cluster(row, 2.0, batch=1, seq_len=4096)
fp8 = recommend_cluster(row, 1.0, batch=1, seq_len=4096)
fits.append({
"model": row.model,
"params_b": row.params_b,
"fp16_fit": f"{fp16[1]}× {fp16[0]}",
"fp16_$hr": round(fp16[2], 2),
"fp8_fit": f"{fp8[1]}× {fp8[0]}",
"fp8_$hr": round(fp8[2], 2),
"use": row.use,
})
fits_df = pd.DataFrame(fits)
print(fits_df.to_string(index=False))
Cost per million output tokens#
Once you know the GPU and a realistic decode throughput, the serving cost converts to $ per million tokens — the unit the rest of the industry quotes. A single H100 typically does 40-80 output tokens/sec at batch 1 on a 7-8B model in vLLM with FP8 KV cache; numbers below assume the middle of that range.
# Approximate decode tokens/sec per GPU for a 7-8B-class dense model in FP8.
# SGLang 0.4+ and vLLM 0.8+ results on H100/H200; Blackwell figures are early-access.
# Keys match PRICES column names so the join below picks up every SKU.
TOK_PER_S = {
"B200": 260, "H200": 135, "H100": 80,
"L40S": 42, "L4": 16,
"A100-80": 58, "4090": 28,
}
rows = []
for gpu, tok_s in TOK_PER_S.items():
hourly = PRICES[gpu].dropna().min() if gpu in PRICES.columns else None
if hourly is None:
continue
cost_per_million = hourly / 3600 / tok_s * 1_000_000
rows.append({"gpu": gpu, "tok_per_s": tok_s, "min_$_per_hr": hourly,
"$_per_million_output_tokens": round(cost_per_million, 2)})
cost_df = pd.DataFrame(rows).sort_values("$_per_million_output_tokens")
print(cost_df.to_string(index=False))
Visualisation#
import matplotlib.pyplot as plt
import numpy as np
fig, axes = plt.subplots(1, 3, figsize=(13, 3.6))
# 1: H100 $/hr by provider
h100 = PRICES[["provider", "H100"]].dropna().sort_values("H100")
axes[0].barh(h100["provider"], h100["H100"], color="tab:blue")
axes[0].set_xlabel("$/hour"); axes[0].set_title("H100 on-demand by provider")
axes[0].invert_yaxis()
for i, v in enumerate(h100["H100"]):
axes[0].text(v + 0.05, i, f"${v:.2f}", va="center", fontsize=8)
# 2: bf16 TFLOPS / $ per hour (effective compute per dollar)
compute = []
for _, g in GPUS.iterrows():
if g.gpu in PRICES.columns:
cheap = PRICES[g.gpu].dropna().min()
if cheap == cheap:
compute.append((g.gpu, g.bf16_tflops, cheap, g.bf16_tflops / cheap))
comp_df = pd.DataFrame(compute, columns=["gpu", "tflops", "hourly", "tflops_per_dollar"])
comp_df = comp_df.sort_values("tflops_per_dollar", ascending=False)
axes[1].barh(comp_df["gpu"], comp_df["tflops_per_dollar"], color="tab:green")
axes[1].set_xlabel("bf16 TFLOPS per $/hr")
axes[1].set_title("compute per dollar (cheapest provider)")
axes[1].invert_yaxis()
# 3: $/M output tokens at 7-8B-class throughput
cd = cost_df.sort_values("$_per_million_output_tokens")
axes[2].barh(cd["gpu"], cd["$_per_million_output_tokens"], color="tab:orange")
axes[2].set_xlabel("$ per 1M output tokens")
axes[2].set_title("serving 7-8B model · single-GPU decode")
axes[2].invert_yaxis()
for i, v in enumerate(cd["$_per_million_output_tokens"]):
axes[2].text(v + 0.5, i, f"${v:.2f}", va="center", fontsize=8)
fig.tight_layout(); plt.show()
Checks#
s.check(
"h100_vs_a100_memory_advantage_for_70b",
lambda: vram_inference(70.0, 2.0, 80, 8, 128, 1, 4096)["total_gb"] > 80,
msg="Llama-3.3-70B fp16 needs more than a single 80GB card",
)
s.check(
"fp8_halves_70b_memory_so_fits_h200_single",
lambda: vram_inference(70.0, 1.0, 80, 8, 128, 1, 4096)["total_gb"] < 141,
msg=f"fp8 70B = {vram_inference(70.0, 1.0, 80, 8, 128, 1, 4096)['total_gb']} GB (H200=141)",
)
s.check(
"8b_fits_single_l40s_at_fp16",
lambda: vram_inference(8.0, 2.0, 32, 8, 128, 1, 4096)["total_gb"] < 48,
msg="Llama-3.1-8B fp16 should fit a 48GB L40S",
)
s.check(
"vast_or_runpod_cheapest_h100",
lambda: PRICES.set_index("provider")["H100"].idxmin() in {"Vast.ai", "RunPod"},
msg=f"cheapest provider = {PRICES.set_index('provider')['H100'].idxmin()}",
)
s.check(
"blackwell_more_tflops_than_hopper",
lambda: GPUS.set_index("gpu").loc["B200", "bf16_tflops"]
> GPUS.set_index("gpu").loc["H100 SXM", "bf16_tflops"],
)
s.check(
"small_model_serving_under_15_per_million",
lambda: cost_df["$_per_million_output_tokens"].min() < 15.0,
msg=f"min = ${cost_df['$_per_million_output_tokens'].min():.2f} "
f"(reference: hosted APIs charge $0.10-1 per million)",
)
Notes for production#
Reservations beat spot for long jobs. 1-3 year committed-use discounts on AWS/GCP/Azure are 30-60% off list — better than spot for steady-state workloads. Lambda, CoreWeave, and RunPod have similar reserved tiers.
Spot is fine for training with checkpointing. Save weights every N steps and resume on preemption. FSDP2 + activation checkpointing handles this best.
AWS cut ML instance prices ~44% in June 2025. H100/H200 on hyperscalers are now much closer to niche-provider rates. Evaluate whether the enterprise SLAs and data-residency controls justify the remaining gap.
SGLang 0.4+ now competitive with vLLM. Benchmarks show SGLang is 3.1× faster than vLLM on DeepSeek-V3 traffic patterns; for workloads with heavy shared-prefix usage (system-prompt caching, RAG), SGLang’s RadixAttention scheduler consistently wins. HuggingFace TGI moved to maintenance mode in 2025; vLLM or SGLang are the recommended replacements.
Egress charges: hyperscalers bill outbound traffic at $0.05-0.12/GB. A model serving 1B tokens/day generates ~1-2 TB of egress; specialty providers (RunPod, Vast, Lambda) bundle or zero-rate it.
MIG and per-GPU partitioning: H100 supports up to 7 MIG partitions; useful for multi-tenant serving of small models. Most niche providers do not expose MIG; rent a smaller card instead.
Bin-packing across requests: vLLM and SGLang pack multiple concurrent requests into a single forward pass. Real production batch=32+ pushes 7-8B-class throughput to 300-600 tok/s on an H100 in FP8, significantly lowering the per-token cost vs single-request numbers.
AMD MI325X and MI300X are production-ready. ROCm 6.3 + vLLM + SGLang run Llama-3/Llama-4 and Qwen families with throughput within 5-15% of H100; MI325X’s 256 GB of HBM3 makes it especially attractive for large MoE models (DeepSeek-V3, Qwen3-235B).
Llama 4 Scout MoE (17B active / 109B total, 10M-token context): fits in a single H200 in FP8 (active-parameter-based KV, not full-weight KV). Maverick (400B total) requires 3-4× H100 SXMs.
Inference APIs vs self-host: Together AI, Fireworks, Anyscale, OpenRouter, and DeepInfra serve open models at $0.07-0.40 per million tokens — often cheaper than self-hosting unless you need a custom fine-tune, strict data residency, or sustained load above ~15 req/s. Cerebras CS-3 via AWS Bedrock offers ~5× token throughput vs standard GPU at comparable cost for supported model sizes.
FP4 quantization (NVIDIA Blackwell native): the B200 exposes 9000 FP4 TFLOPS. Microsoft DeepSpeed and NVIDIA TensorRT-LLM both support FP4 weight quantization. Practical gains for 7-8B models at FP4 vs FP8 are 1.3-1.6× in throughput with <0.5 PPL regression for most tasks — still maturing but worth tracking.
s.summary()
s.save()