Part of Index of Inference Engineering. Book: Ch 5.2-5.4 (Speculative Decoding, Caching, Model Parallelism).
1. Batching
Continuous batching + chunked prefill are defaults in vLLM/SGLang — tune, don't rebuild. When num_requests_waiting climbs before target conc, you're preempting: tune max-num-seqs / gpu-memory-utilization / chunked prefill first. A run measured at preemption point is not publishable.
2. Caching (biggest free win)
- Prefix caching (
--enable-prefix-caching): reuse KV for shared system prompts. Cold 2K system prompt ~548ms TTFT → ~50ms on hit. - RadixAttention (SGLang): prefix-tree across requests; compare hit rate vs vLLM on your workload.
- Cross-instance (LMCache) + offload (
--kv-offloading-backend lmcache): share/warm KV across pods, wired in prod-stack. - Routing matters: same-prefix requests must land same pod (see 08-Routing-Router-LiteLLM-Dynamo). Round-robin destroys hit rate.
3. Speculative decoding
Draft model proposes N tokens, target verifies in parallel. 2-3x latency cut for structured/code/short-factual at low-medium conc; collapses when target saturated (draft eats memory+compute). Needs acceptance ≥0.7 (--speculative-model, --num-speculative-tokens). Combines with guided decoding (same GuidedDecodingParams passed through).
- Safe combo: prefix-cache + chunked prefill = +40-70% thrpt, zero accuracy cost.
- Add spec-decode only for latency-sensitive low-conc endpoints. Publish the crossover point, not just speedup.
- Methods (Book §5.2): draft-target (separate small model), Medusa (heads predict multiple tokens), EAGLE (feature-level draft model, often best acceptance), n-gram / lookahead (no extra model — great for code/structured/repetitive when the continuation is copyable). Try n-gram first (free), then EAGLE if you need more.
3b. Model parallelism (Book §5.4)
| Mode | Splits | Best for | Cost |
|---|---|---|---|
| Tensor parallel (TP) | each layer's weights across GPUs, same node | lower latency for big models | NCCL all-reduce per layer = the TP tax; keep single-node/NVLink |
| Pipeline parallel (PP) | layers across GPUs/nodes | models too big for one node | pipeline bubbles; multi-node |
| Expert parallel (EP) | MoE experts across GPUs | higher throughput on MoE | token routing/load-balance complexity |
- Rule: TP for latency, EP for throughput on MoE, PP for scale past a node.
TP × PP= total GPUs. - MoE note: a router activates a sparse subset of experts per token, so per-token FLOPs stay low while total params are huge — memory (all experts resident) and routing, not dense matmul, become the bottleneck.
4. Long context (128K)
KV scaling + YaRN + chunked prefill + prefix sharing. Bench FP16-KV vs FP8-KV at 128K; KV eviction (StreamingLLM sinks) as labeled variant including failures.
Do
Concurrency sweep past 500-1000 with two tools (e.g. guidellm + genai-perf), output p50/p95/p99 TTFT+ITL. Isolate each knob. Keep one graph where spec-decode lost — proves honest benchmarking.