Part of Index of Inference Engineering. Book: Ch 0 (Inference) + Ch 1 (Prerequisites).
1. The three layers (memorize)
Every inference platform is three layers that must work together:
- Runtime — model performance techniques: batching, caching, quantization, speculation, parallelism, disaggregation.
- Infrastructure — autoscaling at small scale; at scale (few hundred GPUs+) it becomes capacity, multi-cloud, reliability, geo.
- Tooling — level of abstraction. Black box (give weights, get an API) vs full control. The right DX is in the middle: enough control for mission-critical inference without rebuilding everything.
Interview framing: "Inference engineering spans three layers — runtime, infrastructure, and tooling, from CUDA to Kubernetes."
2. App shape drives the optimization target
| Shape | Optimize for | Notes |
|---|---|---|
| Online / interactive | TTFT + ITL | latency-sensitive, streaming |
| Offline / batch | $/MTok + throughput | latency-tolerant, big batches |
| Consumer | bursty, scale-to-zero, cost | variable traffic |
| B2B | steady, SLA, regional, security | compliance, geo residency |
Fill this in for your real app: latency budget → expected concurrency → cost ceiling.
3. Model selection
- Open vs closed: open buys latency control, cost control, and customization; closed buys raw capability. Match to product, not leaderboard.
- Evaluate on your task. A 20-100 example eval set beats public benchmarks. Keep it for parity checks (05-Quantization-AWQ-GPTQ-FP8).
- Fine-tuning for domain quality: LoRA/QLoRA adapters — but prompt/RAG first. No training in this plan, only serving/tuning.
- Distillation: small model trained on a big model's outputs. Cheaper inference near the teacher's quality — a strong inference-cost lever.
4. Measuring latency and throughput
- TTFT = queue + prefill. ITL/TPOT = decode. Throughput = aggregate tok/s.
- Percentiles, never the mean: report p50/p95/p99. The tail defines UX and the SLO.
- End-to-end includes network + client rendering, not just server time.
- Perceived TPS is inversely related to ITL (2 ms ITL ≈ 500 TPS).
5. SLOs (define before building)
p95 TTFT < X ms, p99 ITL < Y ms, quality ≥ floor, $/MTok ≤ Z. Every optimization must move one metric without breaking the others — that tension is the whole job.
6. Do (no GPU)
- Write your app's SLOs: p50/p95/p99 TTFT + ITL +
$/MTok. - Pick one open model; justify vs one closed model on latency, cost, and control.
- Build a reusable 20-example eval set for every later parity check.
7. Interview one-liners
- "Online vs offline changes the objective: TTFT for interactive,
$/MTokfor batch." - "Open models buy latency, cost, and control; closed models buy capability. The choice is product-specific."