shreyansh

07 Kubernetes GPU Production

Part of Index of Inference Engineering. Book: Ch 7.1-7.3 (Containerization, Autoscaling, Multi-Cloud). Written for platform engineers comfortable with K8s, new to LLM serving.

1. Pattern (use this, not DaemonSet)

Deployment (replicas scalable by KEDA) + ClusterIP + Cluster Autoscaler/Karpenter nodes + PVC model-cache. DaemonSet pins 1 pod/node and kills KEDA scaling — no mainstream stack uses it. StatefulSet/LeaderWorkerSet only for stable identity / multi-node replicas.

2. Minimal Deployment (Llama-3-8B, 1x L4)

  • Image: pin tag (vllm/vllm-openai:v0.28.0 as of Sep 2026 — verify current), never :latest.
  • Args: --model NousResearch/Meta-Llama-3-8B-Instruct --gpu-memory-utilization 0.90 --max-model-len 8192 (official meta-llama 403s until Meta approves; mirror avoids friction).
  • Env: HUGGING_FACE_HUB_TOKEN from Secret (gated models).
  • Resources: limits/requests: nvidia.com/gpu: "1". Toleration for GPU taint. emptyDir → swap for PVC to avoid re-download on restart.
  • Probes: startupProbe failureThreshold 40 periodSeconds 10 (~400s model load), then readiness/live on /health. Cold start ~6 min normal — don't mistake for crashloop.
  • Spread: topologySpreadConstraints hostname maxSkew 1.

3. Prod-stack Helm (reference)

git clone https://github.com/vllm-project/production-stack.git && cd production-stack
helm repo add vllm https://vllm-project.github.io/production-stack
helm install vllm vllm/vllm-stack -f tutorials/assets/values-01-minimal-example.yaml
# tutorials: 00-install-k8s-env → 01-minimal → custom configs → weights → multi-LLM → LMCache offload

Gives: serving engine + request router + Prometheus/Grafana dashboards (healthy instances, latency, TTFT, running/pending, KV usage/hit rate).

4. Autoscaling (GPU-aware)

Scale on queue depth / KV pressure / p95 latency, never CPU. Example: KEDA Prometheus scaler queue > 32 for 30s up, empty 5m down; HPA vllm:gpu_cache_usage_perc > 75. Over-provision 20-40% — pod start 30-120s (weights from disk/S3) can't catch bursts reactively.

5. Day-2

PVC RWX (NFS/Ceph/cloud) shared model-cache, VLLM_API_KEY via Secret env (never args — leaks in describe), runtimeClassName: nvidia, GPU Operator + KEDA + cert-manager prerequisites, per-model modelSpec (name/repository/tag/modelURL/replicaCount/CPU/mem/GPU/PVC).

Do

Ship P1 on K8s with dashboard screenshot + helm values + cold-start time + scale event log.

6. Cold starts, scale-to-zero, component scaling (Book §7.2)

  • Cold start = image pull + model download + weight load + graph capture + warmup. Budget it: pin images, cache weights on a PVC/NFS, keep nodes warm.
  • Scale to zero saves money on bursty consumer traffic but costs a cold start on the first request back — only use when the UX can tolerate it, or keep a warm floor.
  • Independent component scaling: scale the serving engine on GPU/queue signals, the router/gateway on RPS/latency, and the queue/workers separately. One component's bottleneck shouldn't size the whole stack.
  • Over-provision 20-40%: pod start of 30-120 s can't react to a burst; queue-based pre-scaling is the fix.

7. Multi-cloud capacity + operations (Book §7.3)

  • Capacity management: past a few hundred GPUs the problem is getting GPUs, not tuning pods. Unify capacity across providers into one pool; avoid per-team silos (one team starved while another idles).
  • Procurement: reserved vs on-demand vs spot; commitments for the baseline, spot/on-demand for bursts. Track effective $/GPU/hr including idle.
  • Geo-aware load balancing: run near end users to cut end-to-end latency; respect data-residency rules.
  • Reliability: multi-cloud/I-R failover, health-based routing, graceful degradation, validated backups of weights/config.
  • Security & compliance: data sovereignty, model/weights provenance, tenant isolation (MIG/network policy), secret handling, audit logs.

8. Interview one-liners

  • "At small scale the problem is autoscaling; past a few hundred GPUs it's capacity — you unify providers into one pool and stop building silos."
  • "Scale on queue depth / KV pressure / p95 latency, never CPU — and over-provision 20-40% because cold start can't catch a burst."