Part of Index of Inference Engineering. Book: Ch 4.3-4.4 (Inference Engines, NVIDIA Dynamo). Triton/TensorRT-LLM content here is an optional Hopper track — skip until hired or chasing H100-scale roles.
Decision matrix
| Need | Pick |
|---|---|
| Fastest to prod, broadest models, AMD MI300X, OpenAI-compat | vLLM (default) |
| HF Hub-native, eval/safety tooling, low per-request latency (Rust core) | TGI |
| Max thrpt on H100/H200, stable model, platform team owns compile pipeline | Triton + TensorRT-LLM (+20-45%, sometimes 2x w/ FP8+spec) |
| Prefix-reuse shootout vs vLLM | SGLang (RadixAttention) |
| Client/edge, CPU offload | llama.cpp / Ollama / ONNX RT (not datacenter stack) |
Why not Triton first
TRT engine compile 30-60 min per model×GPU-family×version, engine locked to that GPU family, 8-16h YAML-to-serving vs ~1h vLLM, complex model-repo + ensemble + gRPC client. Worth it only past ~2000-5000 sustained RPS single model or hard latency SLO where GPU savings > 2 eng-weeks. Many teams run both: vLLM batch + TGI interactive, same weights.
Multi-GPU notes
- vLLM:
--tensor-parallel-size Nsingle-node transparent; multi-node via Ray (to 405B). - TGI: tensor-parallel 2-8 GPUs single node mature; multi-node rare.
- Triton/TRT-LLM: tensor + pipeline most mature for very large multi-node; needs OpenAI frontend adapter (
tensorrtllm_backend/examples/openai) or LiteLLM wrapper. - Multi-LoRA: vLLM
--enable-lora --max-loras Nsingle pod many adapters (winner); TGI ≥2.2 capped; Triton one-engine-per-adapter (heavy).
SGLang in one para
NeurIPS'24, RadixAttention = prefix tree reuse across requests. Stand up same 7B service in SGLang, compare prefix-hit rate vs vLLM on repeated-system-prompt workload. Keep whichever wins; report both.
Do
Deploy vLLM first. Only spike TGI if HF-enterprise, only spike Triton when you can name the RPS and $ saved. Log decision with numbers.
TensorRT-LLM depth (Book §4.3.3) — optional Hopper track
- What it is: NVIDIA's compile-then-serve stack. You build an engine (TensorRT plan) per model × GPU-family × TRT version, then serve it.
- Why it's fast: fused kernels, FP8/INT4/FP4 plugins, in-flight batching, and hardware-specific tuning — often +20-45%, up to ~2x with FP8 + speculation.
- Why it's costly: engine compile 30-60 min each; engines are not portable across GPU families or TRT versions; 8-16h YAML-to-serving vs ~1h for vLLM; model-repo + ensemble + gRPC client complexity.
- When worth it: a single stable model at ~2000-5000+ sustained RPS, or a hard latency SLO where GPU savings beat ~2 eng-weeks of build.
- Pattern many teams use: vLLM for batch/flexible models, TRT-LLM for one hot latency-critical model — same weights, two engines.
- NVIDIA Dynamo (Book §4.4): orchestrates disaggregated prefill/decode + smart routing across a fleet; the system layer above per-engine serving (see 08-Routing-Router-LiteLLM-Dynamo).