shreyansh

09 Observability Benchmarking Cost

Part of Index of Inference Engineering. Book: Ch 7.4 (Testing/Deployment/Cost/Observability) + Ch 4.5 (Benchmarking).

Dashboards (prod-stack gives these)

Healthy instances, request latency histogram, TTFT distribution, running/pending per instance, GPU KV usage %, KV hit rate. vLLM exposes Prometheus /metrics (tok/s, KV util, TTFT, queue depth, prefixcachehits_total). Scrape annotations + Grafana.

Tools

Tool Use
guidellm SLO-aware rate sweeps
genai-perf (NVIDIA) TTFT + ITL measurement
llmperf Hosted endpoint bench
production-stack K8s deploy + router metrics

Benchmark protocol (publishable)

  1. Pin: HW (e.g. 1x L4), model rev, vLLM tag, flags, token lengths (prompt/completion distribution).
  2. Warm up, then concurrency sweep to 500-1000 with two tools cross-checked. Report p50/p95/p99 TTFT + ITL + thrpt + preemptions.
  3. Variants labeled: FP16 base → FP8 → AWQ-INT4 → spec-decode → KV-evict. Isolate one knob per run.
  4. Quality proxy alongside speed (else numbers meaningless).
  5. Cost line: $/MTok = GPU $/hr ÷ (tok/s × 3.6M) adjusted for util. Compare configs on same HW. Never silently FX-convert non-USD salaries/costs — flag normalized explicitly.
    • Worked check: 1x L4 at $0.70/hr doing 500 tok/s → 0.70 / (500 × 3.6) = $0.00039/1K tok ≈ $0.39/MTok. Always state whether tok/s is single-stream or batched, and the utilization assumption — an idle GPU has infinite $/MTok.

Publish checklist (before posting numbers)

  • HW, model, exact token lens, conc sweep, p50/p95/p99 TTFT+ITL + thrpt
  • Pinned versions + repro commands
  • Quant/spec/evict variants labeled, negatives included
  • Cost/latency/quality router policy stated
  • Live cross-stack truth cited (SemiAnalysis InferenceX dashboard), lecture numbers marked as anchors

Weekly habit (post-roadmap)

Scan arXiv cs.DC + MLSys/OSDI/NeurIPS-efficiency titles weekly; read 1 abstract+method-fig+eval. Track vLLM/LMSys/Modal blogs. Papers that matter: PagedAttention (SOSP23), Orca (OSDI22), SGLang (NeurIPS24), SARATHI (OSDI24), DistServe (OSDI24), Splitwise (ISCA24), Mooncake (FAST25), StreamingLLM (ICLR24), EAGLE (spec).