Part of Index of Inference Engineering. Book: Appendix A (Inference Glossary) for every term here, and Appendix B (Recommended Reading) for the full paper/book/blog list. Everything referenced, no dead theory. Versions checked Sep 2026.
Official docs / repos
- docs.vllm.ai — EngineArgs, SamplingParams, GuidedDecoding,
/metrics. - vllm-project/production-stack — tuts 00-install-k8s → 01-minimal (
values-01-minimal-example.yaml) → multi-LLM → LMCache. KEDA tutorial:tutorials/20-keda-autoscaling.md(supports scale-to-zero). - patchy631/time-to-first-token — 10-wk ship-one-service roadmap (follow verbatim).
- ai-hpc/ai-hardware-engineer-roadmap — theory spine (Track G).
- triton-inference-server/tutorials — vLLM-backend-in-Triton bridge (optional Hopper track).
- LMCache/LMCache — cross-instance KV cache offload.
Guides / benchmarks
- ScaleOps "vLLM on Kubernetes" — manifests, probes, KEDA, fractional GPU.
- SitePoint "vLLM Production Deployment: 2026 Guide" — Docker/compose, monitoring, practical defaults.
- Paralleliq "The Inference Stack" — routing vs serving layers (canonical framing).
- Paralleliq "LLM Inference Autoscaling Landscape" — KEDA vs llm-d vs Dynamo vs KServe.
- NomadX "vLLM vs TGI vs Triton on K8s Benchmark (2026)" — decision matrix.
- Inference.net "vLLM Advanced" — LLMEngine/AsyncLLMEngine, TP/PP/EP, APC+chunked+spec combo.
- kubernetes.recipes "Triton vs vLLM" — GPU Operator / K8s ops.
- quantized.uk "vLLM+AWQ tuning" — three-knob order, preemption fixes (domain resolves; locate the current article if the slug moved).
- SemiAnalysis InferenceX — live tok/s, perf/$, tokens/MW truth (cite over static numbers).
Papers (read abstract+fig+eval only unless hooked)
PagedAttention SOSP'23 · Orca OSDI'22 · SGLang NeurIPS'24 · SARATHI OSDI'24 · DistServe OSDI'24 · Splitwise ISCA'24 · Mooncake FAST'25 · StreamingLLM ICLR'24 · EAGLE spec · FlashAttention/2/3 · Medusa. Full citations + links in Book Appendix B (sections: Architecture, GPU Infrastructure, Inference Optimization Research).
Tools
vllm · sglang · guidellm · genai-perf · llmperf · LiteLLM · KEDA · DCGM Exporter · Prometheus/Grafana · RunPod/Lambda/Vast.ai (L4)
Version pins (refresh every ~1 month; book cutoff Jan 2026)
| Component | Pin | Note |
|---|---|---|
| vLLM | v0.28.0 (Aug 2026) |
PyPI latest as of Sep 2026; pin the exact release, never :latest |
| production-stack | v0.1.11+ |
integrated monitoring + KEDA since this version |
| CUDA | 12.1+ / 13.x on current NGC | match host driver; check nvidia-smi first |
Before every benchmark, re-check the current vLLM release and
vllm serve --helpfor renamed flags (KV-offload/connector args, chunked-prefill flags). The book is Jan 2026; treat any 2025-era flag as unverified.
Glossary (Book Appendix A expands each)
TTFT / TPOT-ITL / prefill / decode / KV cache / PagedAttention / continuous batching / chunked prefill / RadixAttention / spec decoding / AWQ-GPTQ-FP8 / TP-PP-EP / disaggregated P/D / prefix-aware routing / $/MTok / preemption (RECOMPUTE) / KEDA queue scaler / ops:byte / arithmetic intensity / MIG / NIM.