shreyansh

03 vLLM Deep Dive

Part of Index of Inference Engineering. Book: Ch 4.2-4.3 (Frameworks, Engines). vLLM is the default. Master this before anything else.

1. Quick start (rented L4)

pip install vllm
vllm serve Qwen/Qwen2.5-7B-Instruct-AWQ --host 127.0.0.1 --port 8000
# production-ish:
vllm serve hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4 \
  --served-model-name llama-3.1-8b --max-model-len 8192 \
  --quantization awq --gpu-memory-utilization 0.90 \
  --enable-prefix-caching --port 8000
curl localhost:8000/v1/models
curl -X POST localhost:8000/v1/completions -H 'Content-Type: application/json' \
  -d '{"model":"llama-3.1-8b","prompt":"Once upon a time,","max_tokens":10}'

2. Three knobs that matter (tune in order)

  1. --max-model-len — KV reserved for declared window. 128K default on traffic that sends 4K wastes all memory. Set to real p99 prompt+completion.
  2. --gpu-memory-utilization (fraction of total VRAM, not free) — 0.90 sane on dedicated card. Ceiling is init OOM (profiling+graph capture+warmup), not mid-traffic kill.
  3. --max-num-seqs — max batched requests. Direct lever on cache per request. Lower when preemptions climb.

Others: --max-num-batched-tokens (must exceed max-model-len if chunked prefill disabled), --tensor-parallel-size, --enforce-eager (debug only — skips CUDA graphs, kills steady-state speed), --enable-prefix-caching, --kv-offloading-backend lmcache.

Verify flags per release: vLLM's KV-offload/connector API has churned (--kv-transfer-config / connector args). Check vllm serve --help on the version you pin, don't trust an old blog. Same for the vllm/vllm-openai image tag — pin the current release, not a stale one.

3. API levels

  • LLM class: scripts/batch. Fine for prototypes.
  • LLMEngine / AsyncLLMEngine: production (Kafka/Redis queue, custom priority, streaming). add_request() + step() loop, EngineArgs holds every knob, SamplingParams (20+ fields — learn beyond temperature/top_p), GuidedDecodingParams (json/regex/choice/grammar for structured output).
  • Server: OpenAI-compat (/v1/completions, /v1/chat/completions, /health, /metrics).

4. Source to read (not rebuild)

block manager (paging) + V1 scheduler (single-controller event loop, chunked prefill default) + AsyncLLMEngine.generate() streaming path. Goal: explain why waste <4% and GPU never idles between requests.

5. Failure signatures

  • PreemptionMode.RECOMPUTE climbing → cache too small. Fix order: shorten max-model-len → lower max-num-seqs → raise gpu-memory-utilization.
  • Start-up OOM → something else on card (fraction is of whole card) or too-high utilization.
  • Crash with chunked-prefill disabled → max-num-batched-tokens < max-model-len.
  • Good latency, bad thrpt under conc → stray --enforce-eager.

6. Do

Expose /metrics, graph TTFT, ITL, queue_depth (num_requests_waiting), kv_usage, preemptions. Preemptions must stay 0 under real load. Save launch command + versions.