Part of Index of Inference Engineering. Book: Ch 3 (Hardware) + Ch 4.1 (CUDA). You need operator-level GPU, not kernel-author level.
1. Sizing math (use daily)
- FP16 weights ≈ 2 GB per B params. Llama-3-8B ≈ 16GB, 70B ≈ 140GB. Plus KV cache headroom (grows with
max-model-len × concurrency). - Rule table: | Model | FP16 | AWQ-INT4 (~0.5GB/B) | Fits on | |-------|------|---------------------|---------| | 7-8B | ~16GB | ~4-6GB | 1x L4/A10G 24GB (FP16) or 1x T4 16GB (AWQ) | | 70B | ~140GB | ~40GB | TP across H100s (FP16) or 2x L4/A10G (AWQ) |
- Single-stream decode ceiling ≈
bandwidth ÷ weight_bytes(e.g. ~218 tok/s measured Llama-3.1-8B-AWQ on RTX 4090 1008GB/s). Batched is higher — that's vLLM's job.
2. Hardware map
- L4/A10G (24GB): cheap dev serving 7-8B. Start here.
- A100-80GB / H100/H200: 70B-class, FP8 (Hopper) near-lossless ~2x thrpt. FP8 kernels validated on Hopper; verify on Ada (L4/4090) per vLLM version.
- AMD MI300X: vLLM only real option (ROCm since v0.4). Triton+TRT-LLM is NVIDIA-only.
- MIG slices: hardware isolation for multi-tenant; serve via Deployment per slice, never DaemonSet.
3. Software stack (top-down)
Drivers (≥525) → CUDA (≥12.1) → Container Toolkit (≥1.14) → K8s GPU Operator (device plugin) → nvidia.com/gpu resource → runtime (vLLM) → NCCL (multi-GPU collectives)
- Multi-GPU: tensor parallelism (split weights, same node, NVLink,
--tensor-parallel-size= GPU count) vs pipeline parallelism (split layers, multi-node, TP×PP total) vs expert parallelism (MoE token routing). - NCCL all-reduce per layer is the TP tax. Keeping TP single-node avoids slow interconnect.
4. Do
# inside GPU pod
nvidia-smi dmon -s pucvmet -d 5
python3 -c "import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0))"
kubectl describe node | grep -A3 "nvidia.com/gpu"- Label nodes (
nvidia.com/gpu.product=H100-80GB), taint GPU pools, tolerate in serving Deployment. - Monitor with DCGM Exporter + Prometheus. Idle GPU = burning money.
5. Pitfalls
- CUDA container vs driver mismatch → check
nvidia-smion host first, pin NGC/container tag. - No
resources.limits: nvidia.com/gpu→ no GPU. Always set requests == limits. - OOM at startup ≠ mid-traffic OOM. Startup peak (weights + graph capture + warmup) is the risk; see 03-vLLM-Deep-Dive
--gpu-memory-utilization.
6. Architecture generations (Book §3.2)
| Gen | Examples | Inference note |
|---|---|---|
| Ada Lovelace | L4, L40S, RTX 4090 | cheap dev; HW FP8 exists but kernel support varies |
| Hopper | H100, H200 | FP8 + Transformer Engine, NVLink; the FP8 sweet spot |
| Blackwell | B200, GB200, NVL72 | FP4/FP6, rack-scale NVL72 for very large models |
| Rubin | (next gen) | announced/future; track only |
| Grace / Vera | CPU + GPU Superchips | host CPU tightly coupled to GPU |
- Instances: single-GPU, multi-GPU (NVLink/PCIe), and MIG (partition one GPU into isolated slices — hardware isolation for multi-tenant; serve one Deployment per slice).
- Other accelerators: AMD MI300X (ROCm, vLLM is the realistic path), plus TPU/Trainium — relevant only if a JD names them.
- Local inference (Book §3.5): desktop (llama.cpp/Ollama, GGUF) and mobile (edge/quantized) — different constraints (CPU offload, no batching); not the datacenter stack.
7. Interview one-liners
- "Hopper is the FP8 tier; Ada/L4 is the cheap AWQ dev tier; Blackwell adds FP4 and rack-scale NVL72."
- "MIG gives hardware isolation per tenant — one Deployment per slice, never a shared DaemonSet."