shreyansh

05 Quantization AWQ GPTQ FP8

Part of Index of Inference Engineering. Book: Ch 5.1 (Quantization). Focus AWQ/GPTQ on L4; FP8 is a optional Hopper track.

Ladder (quality → size)

FP16/BF16 reference → FP8 (Hopper, near-lossless, ~2x thrpt) → AWQ-INT4 / GPTQ-INT4 (~4x smaller, 5-10% slower than FP8 due to dequant, tiny quality loss) → GGUF (CPU-offload only, skip for GPU prod)

  • AWQ (W4A16): activation-aware 4-bit, any CUDA GPU (A10/L4/T4 OK). Use pre-quantized Hub checkpoints (e.g. hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4) + --quantization awq. Qwen2.5-7B-AWQ ≈ 3.6GB weights, 4.2GB @4K → 5.9GB @32K (difference is all KV — that's max-model-len spending).
  • GPTQ: similar bits, different calibration; AWQ usually 5-15% faster in vLLM (verify per model/version).
  • FP8: best quality/perf on H100/H200 via Transformer Engine. Ada (L4/4090) has HW FP8 but kernel support varies — verify per vLLM release. Optional Hopper track: if you only have an L4, mark FP8 as "not tested" rather than guessing; keep the plan AWQ/INT4-first.
  • Never convert lossy→lossy (e.g. GPTQ→AWQ). No AWQ build = constraint, not a conversion task.

Book §5.1.1 number formats (FP32/TF32/FP16/BF16/FP8/INT8/INT4, exponent vs mantissa tradeoffs) and §5.1.2 approaches (PTQ, QAT, weight-only W4A16 vs weight+activation, per-channel vs per-group, calibration) explain why the ladder works. Read these before the parity protocol.

Composability

Quantization + TP just works (sharded quantized matrices). Standard: FP8 + TP=8 single H100 node for 70B max thrpt; AWQ + TP=4 for 70B on 4x L4 where FP16 won't fit.

Parity protocol (required per variant)

  1. FP16 baseline: thrpt + quality proxy (MMLU-subset / HumanEval-subset / your RAG eval).
  2. Candidate (FP8 / AWQ): same harness, pinned versions.
  3. Ship if quality delta ≤1-2% for thrpt/memory won. Else keep FP16/FP8 and document the negative result — interviewers love this.

Do

vllm serve <base-FP16> --max-model-len 8192 --gpu-memory-utilization 0.90
vllm serve <same-AWQ> --quantization awq --max-model-len 8192 --gpu-memory-utilization 0.90
# compare: weight GB, tok/s, p95 TTFT, preemptions, quality proxy

Record GB, tok/s, p95 TTFT, $/MTok. See 09-Observability-Benchmarking-Cost.