Part of Index of Inference Engineering. Book: all chapters — each project is the lab for a cluster of chapters (Ch 4-5 for P1/P2, Ch 5.4 + 7 for P3). Build in order. Each = resume bullet + interview story + benchmark post.
P1 — OpenAI-compat service on K8s-GPU (must-have, 2 wks)
- Serve Llama-3-8B or Qwen2.5-7B (AWQ on L4) via Docker → K8s Deployment+Service per 07-Kubernetes-GPU-Production.
- Grafana: TTFT, TPOT, thrpt, queue, KV hit, $/1M tok.
- Publish: launch cmd, Helm values, p95 numbers, cold-start time, pinned
vllm/vllm-openai:<current release>. - Resume: "Serve Llama-3-8B on 1xL4 via vLLM/K8s at p95 TTFT Xms, Y tok/s, preemptions 0."
P2 — Quant + router benchmark (differentiator, 2 wks)
- Same model FP16 vs AWQ-INT4 vs FP8 (if Hopper access) + LiteLLM/100-line FastAPI cost router (
cost/latency/quality, token budgets). - Table:
GB | tok/s | p95 TTFT | qualityΔ | $/MTok. Include one negative (INT4 loses quality → kept FP8). - Resume: "Cut $/MTok Z% via AWQ + prefix-aware routing with p95 SLO held."
P3 — Autoscaled multi-LoRA platform (closer, 2 wks)
vLLM --enable-lora --max-loras Nmulti-adapter + KEDA queue-depth autoscale + Spring gateway per 10-Java-Bridge-FastAPI-Gateway.- Prove: adapter isolation, scale-up on
queue>32/30s, scale-downempty/5m, auth+budget path. - Resume: "Platform serving N LoRA adapters from 1 pod, queue-driven autoscale, Java gateway (auth/ratelimit/Kafka→AsyncLLMEngine)."
Repo bar (all three)
README arch diagram + repro commands + pinned versions + Grafana screenshot + variants labeled + negative result + cost line. No :latest, no unreproducible numbers.