Part of Index of Inference Engineering. Book: Ch 5.5 (Disaggregation) + Ch 4.4 (NVIDIA Dynamo) + Ch 7.2.3 (Routing/Queueing).
Layers (add as you scale)
- L4 K8s Service (round-robin): fine for 1-2 pods. Destroys KV-hit rate at scale.
- L7 Ingress/Gateway (nginx/Istio/Kong/Envoy): route by path/header/model name.
- Inference-aware router: polls
/metrics, routes on affinity + queue + health + cost. Two hops total: LB→router (dumb transport), router→pod (smart placement, bypasses kube-proxy round-robin).
Options
- vLLM router (same binary, router mode, no model): KV-affinity (shared prefix → same pod), multi-model + aliases, session-ID routing, QPS/TTFT/pending per-engine metrics, K8s discovery + fault tolerance.
- LiteLLM proxy: register vLLM/TGI as OpenAI-compat models → routing + fallback + virtual keys + spend tracking across self-hosted + cloud. Fastest cost-aware front door.
- NVIDIA Dynamo: disaggregated prefill (compute-heavy) vs decode (bandwidth-heavy) onto different pods; max thrpt at scale.
- llm-d / Gateway Inference Extension: K8s-native disaggregation + KV-aware scoring (what RedHat hiring posts ask for).
Disaggregation (Book §5.5) — the "why"
- Prefill is compute-bound (big matmuls); decode is memory-bandwidth-bound (one token/step). On one GPU they fight: a long prefill stalls ongoing decodes.
- Disaggregated P/D puts prefill and decode on separate worker pools sized independently (DistServe, Splitwise). Prefill workers scale on prompt load; decode workers scale on active sequences.
- KV transfer is the tax: the KV cache must move from the prefill worker to the decode worker (NVLink/RDMA). Worth it at scale or with long prompts / strict TTFT SLOs.
- Dynamic disaggregation: NVIDIA Dynamo decides per-request, per-topology whether to disaggregate — the practical form of this in 2026.
- When NOT to: small scale, short prompts, single GPU — the KV transfer outweighs the win. Start monolithic (vLLM), add P/D only when measurements demand it.
- llm-d / Gateway Inference Extension: K8s-native KV-aware scoring + disaggregation (what Red Hat and similar hiring posts ask for).
Cost-aware policy (build this)
pick backend = min(cost) subject to p95 TTFT < SLO and quality ≥ floor; per-request token budget logged. Router + economics in same week — forces $ into code. See 09-Observability-Benchmarking-Cost.
Do
Start Service + vLLM. Add LiteLLM or vLLM-router when >2 pods or prefix-reuse matters. Measure hit-rate delta round-robin vs affinity on repeated-prompt workload.