shreyansh

00 Roadmap and Study Plan

Part of Index of Inference Engineering. Goal: go from Java+K8s to hireable inference-platform engineer in 12 weeks. The book Inference Engineering (Philip Kiely, Baseten Books, 2026) is the theory spine; these notes are the lab manual. Read the week's book chapter → run the matching note's ## Do.

Constraints (you)

  • Stack: Java, Docker, K8s. Python: learn glue-only (FastAPI + tokenizer + prometheus client). No training/fine-tuning beyond understanding the concepts.
  • GPU: no local GPU needed. Rent 1x L4 (~0.6−0.8/hronRunPod/LambdaLabs/Vast.ai).Budget0.6-0.8/hr on RunPod / Lambda Labs / Vast.ai). Budget 30-50 total. Shut down after each session.
  • Time: ~45 min/day overall. Daily sessions = reading/notes/no-GPU work. Batch GPU work into 2-3 blocks/week of 90-120 min — L4 cold start (~6 min) + model download make 25 min/day wasteful.
  • Every GPU block ends with a command + metric, not just reading.

Reading method (per chapter)

  1. Skim (10 min): headings, figures, captions, bold → write the 5 key claims.
  2. Deep pass with note open: for each section write a concept → vLLM/SGLang knob → observed metric triple.
  3. Teach-back (90 s whiteboard): close the book, explain it out loud — this is the interview format.
  4. Lab: run the note's ## Do and log the number.

Read fully: Ch 0, 2, 4.1-4.3, 5.1-5.5. Skim: Ch 1, 3, 6, 7. Look up (reference): App A/B.

Weekly plan

Wk Book chapters Focus Deliverable Note
0 Ch 0 + Ch 1 Product + model selection SLOs + reusable eval set, no GPU 00b-Prerequisites-Product-Model-Selection
1 Ch 2.1-2.4 Mental model Derive prefill vs decode + ops:byte/roofline in own words 01-Fundamentals-Prefill-Decode-KV-Cache
2 Ch 3 + Ch 4.1 Hardware VRAM/bandwidth table + nvidia-smi dmon capture 02-GPU-CUDA-Basics-for-Backend-Devs
3-4 Ch 4.2-4.4 vLLM default OpenAI-compat endpoint on rented L4 + Grafana TTFT/ITL 03-vLLM-Deep-Dive
5 Ch 5.1 Quantization FP16 vs AWQ-INT4 vs FP8 table (GB, tok/s, p95 TTFT) 05-Quantization-AWQ-GPTQ-FP8
6 Ch 5.2-5.4 Batch/cache/spec Concurrency sweep 500-1000, prefix-cache win, spec-decode crossover 06-Batching-Caching-Spec-Decoding
7 Ch 5.5 + 4.4 Routing + disaggregation Round-robin vs prefix-aware hit-rate delta; Dynamo/llm-d notes 08-Routing-Router-LiteLLM-Dynamo
8-9 Ch 7.1-7.3 K8s production Deployment+KEDA prod-stack, autoscale on queue not CPU, multi-cloud/procurement notes 07-Kubernetes-GPU-Production
10 Ch 7.4-7.5 + 4.5 Cost + gateway Cost-aware router + $/MTok dashboard + Java gateway 09-Observability-Benchmarking-Cost + 10-Java-Bridge-FastAPI-Gateway
11 Ch 6 (skim) Modalities 1-page VLM/ASR/embedding/image notes + one demo 06b-Modalities
12 App A/B + consolidate Publish 3 repos + benchmark writeup + 1 upstream PR 11-Projects-P1-P2-P3

Triton/SGLang comparison and engine choice: 04-SGLang-TGI-Triton-When-Why (Triton = optional Hopper track, only after hired or chasing H100-scale roles). Positioning + interviews: 12-Interviews-Resume-Positioning. Start here. Every link, paper, repo: 13-Resources-Glossary.

Daily template (copy/paste)

  1. Build (GPU-block days only): run one command, save output.
  2. Read (15 min): one book section / paper section / doc page / source file.
  3. Log (5 min): metric + version pins (vllm:<current>, model id, flags) in notes.

Rules

  • Measurement before optimization. No tuning without baseline + dashboard.
  • Pin everything: image tag, model revision, flags. :latest is banned. Refresh stale pins per 13-Resources-Glossary.
  • Publish negative results (e.g. "INT4 saved 10GB but lost 1.5% quality → kept FP8"). Employers trust this.
  • vLLM-only depth first. FP8 and Triton+TensorRT-LLM are optional Hopper tracks (marked as such in 05-Quantization-AWQ-GPTQ-FP8 and 04-SGLang-TGI-Triton-When-Why) — add only after hired or chasing H100-scale roles (~2 weeks).
  • Keep a reusable 20-example eval set from Week 0; use it for every quantization/optimization parity check.

Exit criteria (job-ready)

  • P1 serving + dashboard live link/screenshot
  • P2 quant/router cost table
  • P3 autoscaled + Java gateway
  • Can whiteboard: request path → scheduler → KV cache → decode loop → metrics → $/MTok
  • Can derive ops:byte ratio and explain why decode is bandwidth-bound
  • Profile rewritten per 12-Interviews-Resume-Positioning