Part of Index of Inference Engineering Hands-on. Book: Inference Engineering (Philip Kiely), Ch 2 (Models), pp. 39–70.
Interactive lesson Steppable, animated diagrams for every section: Open Chapter 2 — Models →
Before this lesson The lesson assumes no prior knowledge — every term is defined on first use. These four ideas carry most of the weight, so they're worth a glance first:
- A neural network is a stack of layers of numbers (weights). Each layer multiplies and adds its input and passes the result on — we build this up in 2.1.
- Large language models (LLMs) and tokens: an LLM reads and writes tokens, where a token is roughly a word or part of one. A tokenizer converts text to token ids.
- Attention lets each token look back at earlier tokens to gather context. It is one of the most expensive operations in inference.
- The KV cache is the model's working memory: it stores what attention already computed so later steps never redo that work.
You do not need calculus or linear algebra — everything here is explained in terms of multiply, add, and store.
Learning objectives
- Explain the node → layer → hidden state mental model, and why non-linearities are required.
- Trace an LLM request: tokenize → prefill → decode, and say which phase is memory- vs compute-bound.
- Read a transformer block: attention + feed-forward + normalization → LMHead.
- Describe attention (Q/K/V), multi-head, self vs cross, and the causal mask; why the KV cache makes decode linear.
- Explain Mixture of Experts routing and sparse activation.
- Describe image/video generation as iterative denoising in latent space.
- Decide whether a workload is memory- or compute-bound from arithmetic intensity vs the hardware ops:byte ratio.
- Name the main attention optimizations: FlashAttention, PagedAttention, sliding-window/gated/linear/compressed/multi-latent, Mamba hybrids.
Section map
| § | Topic | One-line takeaway |
|---|---|---|
| 2.1 | Neural networks | Nodes form layers; hidden outputs are hidden states; encoders build representations, decoders use them. |
| 2.1.1 | Linear layers & matmul | A linear layer is y = W·x + b. |
| 2.1.2 | Activation functions | Non-linearities stop stacked matmuls collapsing into one. |
| 2.2 | LLM inference mechanics | Autoregressive token generation: prefill the prompt, then decode one token at a time. |
| 2.2.1 | LLM architecture | config.json + naming like Qwen3MoeForCausalLM; causal means next-token prediction. |
| 2.2.2 | Transformer blocks | Embedding → N × (attention + MLP + norm) → LMHead. |
| 2.2.3 | Attention | Q/K/V relate a token to prior tokens; quadratic, made linear at decode by the KV cache. |
| 2.2.4 | Mixture of Experts | A router activates a sparse subset of experts (e.g. 22B of 235B). |
| 2.3 | Image generation | Text encoder + denoiser + VAE; ~30–50 steps, 2 passes each. |
| 2.3.1 | Image architecture | Diffusion transformers operate on patches in latent space. |
| 2.3.2 | Few-step models | Latent consistency / distillation → 1–4 steps. |
| 2.3.3 | Video generation | Whole-video latent; framewise errors compound. |
| 2.4 | Inference bottlenecks | Compute vs memory bandwidth; find the limiting resource. |
| 2.4.1 | Ops:byte & arithmetic intensity | H100 FP16 ≈ 295 ops/byte; compare workload intensity to it. |
| 2.4.2 | LLM bottlenecks | Prefill compute-bound, decode memory-bound (worked example AI ≈ 62). |
| 2.4.3 | Media bottlenecks | Each denoising pass revisits the whole latent image or video, making media generation compute-bound. |
| 2.5 | Optimizing attention | FlashAttention, PagedAttention + new algorithms and Mamba. |
Key terms
- Hidden state — a layer's output vector; its dimensionality is the size of the internal representation.
- KV cache — stored key/value tensors for prior tokens; avoids recomputation so decode attention is linear.
- Logits → softmax → sampling — the LMHead scores the vocabulary; temperature / top-k / top-p shape the pick.
- Latent space — low-dimensional representation an image denoiser works in (~1% of pixel values).
- Arithmetic intensity — FLOPs per byte moved; below the hardware ops:byte ratio ⇒ memory-bound.
- FlashAttention / PagedAttention — fused kernels that cut HBM traffic / fragmented KV-cache pages.
Numbers to remember
- H100 FP16: 989 TFLOPS ÷ 3.35 TB/s ≈ 295 ops:byte.
- Worked decode attention example: intensity ≈ 62 < 295 ⇒ memory bound; a 4096×4096 matrix ≈ 32 MiB.
- MoE example: Qwen3-235B-A22B activates ~22B params per request.
- Image: 30–50 denoising steps × 2 forward passes ≈ ~100 passes.