How generative models actually compute — neural networks, the two inference styles (making text one piece at a time and refining images or video from noise), and where the bottlenecks hide.
You do not need calculus or linear algebra — everything here is explained in terms of multiply, add, and store.
Overview
Generative models are compositions of neural networks
Inference engineering is the practice of making generative models fast and cheap without
losing quality. That requires a working intuition for how models compute. Across modalities,
models work in two broad ways:
Text, one piece at a time
Start from the input and predict the most likely next piece, one at a time. This is how text models generate answers; §2.2 gives the method a formal name and explains its mechanics.
Images and video from noise
Start from noise and improve it repeatedly into the most likely image or video. §2.3 gives the method a formal name and explains its mechanics.
2.1
Neural networks
Nodes in a layer work independently, and each layer reads the previous layer's output. LLMs stack
dozens to hundreds of layers.
Signal flowing through a deep network a packet enters, and each layer lights up as the signal reaches it
0/–
Every layer reads the previous layer's output. Those intermediate vectors — the
hidden states — are what "dimensionality" refers to.
There are three layer roles — input, hidden, and
output. Hidden outputs are the hidden states, and their
dimensionality is the size of the internal representation.
Networks play two roles: an encoder builds an internal representation from an
input; a decoder uses that representation to generate an output. Networks are
composable — you can chain them into pipelines. Modern LLMs are decoder-only;
encoder–decoder models like Whisper handle audio → text.
2.1.1
Linear layers and matmul
A matrix is a grid of numbers. A matmul multiplies a vector through a matrix
to produce an output vector. A linear layer is the simplest matmul, applying a
weight matrix W and adding a bias b: y = W·x + b.
Linear layer: y = W·x + b each row of W dots the input vector to make one output number
0/–
Weights are fixed after training; at inference the same W is reused for every
token. Repeatedly loading those weights is a major decode cost, explored in §2.4.
2.1.2
Activation functions
Matmul is composable: multiplying through several matrices equals multiplying by their
product. A stack of purely linear layers therefore collapses into one. Activation
functions are non-linear, which prevents that collapse and makes depth worthwhile.
Non-linearity stops layers from collapsing without an activation, two matmuls are just one matmul
0/–
layer 1
W₁·x
→
layer 2
W₂·(W₁·x)
→
collapses to
(W₂W₁)·x
ReLU vs Swish
Because matmul composes, W₂(W₁x) = (W₂W₁)x. Non-linear activations break that
collapse, so depth actually buys representational power.
ReLU returns x when positive, else 0. Swish (SiLU) zeroes or
near-zeroes negatives while keeping positives. Because they are mostly differentiable, they
have usable gradients for training.
2.2
LLM inference mechanics
This section follows one journey: text first becomes token ids, then the model predicts each
new token from all previous tokens.
Subword tokenization
common words are one token; rare words split into several
Tokenization splits uncommon words.
Each token maps to an integer id. Fewer tokens per output ⇒ faster end-to-end inference.
A request has an input sequence (prompt/context), optionally a
reasoning sequence, and an output sequence. The
context window is the model's total-token budget across all three sequences.
Prefill (parallel) → Decode (one token at a time) the KV cache fills all at once, then grows one entry per new token
0/–
input tokens · generated tokens
Theskyisbluebecause│lightscattersintheair
KV cache · 0 entries
idle
Prefill computes attention for the whole prompt in one parallel pass and caches every K/V.
Decode then appends one token — and one cache entry — per forward pass.
After the controls narrow the candidate pool, the final token is drawn from the remaining probabilities.
Sampling: temperature, top-k, top-p
the output layer's scores become probabilities, then a token is drawn
Only tokens that survive top-k / top-p are re-normalised and sampled. Tighter settings ⇒ more
predictable output (good for structured output).
Lower temperature, top-k, or top-p makes output more predictable — useful when the response
must conform to a schema. These controls apply after each forward pass.
2.2.1
LLM architecture
A model directory ships a config.json describing the architecture. One
architecture can have multiple sizes (8B, 70B), multiple
variants, and LoRA adapters—small add-on modules that change behavior, not
architecture.
Reading an architecture name
hover or tap a segment of Qwen3MoeForCausalLM
tap a segment above
A causal language model predicts the next token from previous tokens — unlike
a masked model that fills a blank from both sides. All generative LLMs today are causal.
2.2.2
Transformer blocks
Put together, an LLM has an embedding input layer, N transformer blocks, and an output
language-modeling head (LMHead).
From tokens to next-token logits the signal steps through embedding → blocks → LMHead → sampled token
0/–
input
tokens
→
embedding
vectors
→
× N blocks
attn + MLP + norm
→
LMHead
logits
→
sample
token
A transformer block = attention + feed-forward network + normalization. The feed-forward
sublayers hold the majority of trainable weights; attention is second.
Each block is attention + an MLP (another stack of linear layers) + normalization. The
feed-forward sublayers hold the majority of trainable weights; attention is second-largest.
2.2.3
Attention
More precisely, the updating token supplies the query (Q); earlier tokens supply
the keys (K) and values (V).
Attention: Q · Kᵀ → softmax → · V the query scores against every key, then reads the values
0/–
query
Q · this token
×
keys
K · all prior
→
scores
S = QKᵀ (N×N)
softmax →
weights
P (N×N)
×
values
V
→
output
O = P·V
Attention sublayers are multi-head: several copies run this procedure in parallel,
each learning different relationships. Self-attention uses Q, K, and V from one
sequence; cross-attention takes Q from one sequence and K/V from another.
Causal mask: a token may only look back each row reveals which earlier tokens it is allowed to attend to
0/–
✓
×
×
×
×
×
✓
✓
×
×
×
×
✓
✓
✓
×
×
×
✓
✓
✓
✓
×
×
✓
✓
✓
✓
✓
×
✓
✓
✓
✓
✓
✓
green = allowed (attend to prior tokens) · hatched = masked (no look-ahead)
The mask is why causal LMs predict the next token from previous tokens only. Attention itself
is quadratic in sequence length; the KV cache makes decode linear.
Attention is quadratic in sequence length, but the KV cache
stores each prior token's K/V so decode runs in linear time. The cache lives
on GPU memory and dominates decode memory traffic.
2.2.4
Mixture of Experts
For example, in Qwen3-235B-A22B, about 22B of 235B parameters activate per request,
and different requests activate different experts.
Mixture of Experts: route each token to a few experts the router activates only 2 of 8 experts — the rest stay idle
0/–
token
x
→
router
gate scores
→
top-k
select k
→
combine
weighted sum
E1
E2
★E3
E4
E5
★E6
E7
E8
8 experts · 0 active
Expert Parallelism spreads experts across GPUs (here, two 4-GPU groups)
GPU group A
E1 E2 E3 E4
GPU group B
E5 E6 E7 E8
Sparse activation is why a Qwen3-235B-A22B can run like a ~22B model per request:
only the chosen experts compute. Best for 100B+ models; tiny models gain little.
MoE suits large models (100B+) and spreads experts across GPUs via Expert
Parallelism. Models under 32B — especially under 8B — gain little, since the whole
model is effectively one expert.
2.3
Image generation
A text-to-image foundation model therefore has a text encoder, a denoising model, and a VAE.
The denoiser works in latent space (a 128×128 latent ≈ 1% of pixel values) so
attention can run over the whole image. Each step runs two passes — conditioned and with the
negative prompt — so ~50 steps ≈ 100 forward passes.
Diffusion: noise → image in ~40 steps watch the latent resolve; each step costs two forward passes
0/–
step 0 / 40
forward passes 0
Each step: one pass conditioned on the prompt, one with the negative prompt.
A 50-step generation is really ~100 forward passes. Few-step models (latent consistency or
distillation) cut this to 1–4 steps for near-real-time output.
The main per-request arguments:
Argument
Effect
prompt
what the image should look like
negative prompt
styles or objects that should not appear
steps
speed vs quality (30–50 typical)
guidance scale
creativity vs prompt adherence (often ~4)
image size
fixed resolutions and aspect ratios
2.3.1
Image model architecture
Modern image models are diffusion transformers (DiT): like LLM transformers,
but they process image patches (small square tiles of the latent) instead of token embeddings.
SDXL is a classic base + refiner example; modern models use full LLMs as text encoders and
~20B-parameter denoisers.
Legacy (SDXL era)
CLIP-based text encoder · <4B denoiser · single-encoding VAE.
Modern
Full-LLM text encoder · ~20B denoiser · dual-encoding VAE.
The trend is blending LLM architecture into image generation — LLMs bring baked-in text
understanding and variable-length, single-pass generation. Models like HunyuanImage-3.0 do
this, and the same ideas extend to video.
2.3.2
Few-step image models
The most expensive part of image generation is the 30–50 denoising steps. Few-step
models target 1–4 steps:
Latent consistency
Predict the target latent directly, repeat 2–4 times to improve quality.
Distillation preferred
Adversarial and/or progressive distillation trains a small model to emulate a larger one in fewer steps.
2.3.3
Video generation
Video models resemble image models but encode 10–100× more information in latent space (two
spatial dimensions plus time). Generating frame-by-frame compounds errors; instead, modern
models hold the whole clip in latent space and update all of it every step.
Video: framewise drift vs whole-video latent the top row compounds error frame by frame; the bottom row holds the whole clip
0/–
framewise — errors accumulate
whole video in latent space — stable
Modern video models hold the entire clip in latent space and update all of it every forward
pass. Attention over that space is so expensive that they typically run at batch size 1.
Attention over that latent space is so costly that video models typically run at
batch size 1 — a full 8-GPU node per request. Techniques like
Self Forcing and added autoregressive components reduce the cost, but
attention remains dominant.
2.4
Calculating inference bottlenecks
GPUs have two resources: compute (FLOPs/s — floating-point operations per
second) and memory bandwidth (bytes/s). A bottleneck is an imbalance where one sits idle while the other
saturates. If a system is memory-bandwidth bound, compute optimization cannot help.
LLM decode is typically memory bound; image and video generation are compute bound.
2.4.1
Ops:byte ratio and arithmetic intensity
For an H100 in FP16, that comparison is
989 TFLOPS / 3.35 TB/s ≈ 295. The roofline model plots achievable
performance against workload intensity.
Roofline model (H100, FP16)
below the ridge → memory bound; above → compute bound
989 TFLOPS · 3.35 TB/s
A workload's arithmetic intensity (FLOPs per byte) compared with the hardware's ops:byte
ratio (≈295) tells you which resource to optimize. Decode sits far left — memory bound.
2.4.2
LLM inference bottlenecks
Prefill processes the whole input in parallel (weights loaded once → high
intensity). Decode generates one token at a time, reloading weights each step
(low intensity → memory bound).
A naive decode attention kernel (the small GPU program that runs the math) with N=4096 and d=128 forms
S, P at 4096×4096 (≈32 MiB each). Summing reads/writes and
compute gives an arithmetic intensity of about 62 — far below the H100's
295. Hence memory bound.
Batching raises arithmetic intensity: a batch does more compute for the same weight traffic.
This is why continuous batching is so effective.
2.4.3
Image-generation inference bottlenecks
Because every iteration revisits the whole latent image or video, image and video generation
are computationally demanding. Their performance problem is generally too little compute, not
too little memory movement; modality-specific behavior is covered in Sections 6.5 and 6.6.
Why repeated whole-object work strains computation each pass handles the whole latent object; dozens of passes repeat that math
0/–
whole object
latent image or video
→
each pass
attend everywhere at once
→
repeat
30–50 denoising passes
Repeating a large calculation once per step turns image and video generation into
computation-heavy work, unlike reloading model weights once per generated token.
2.5
Optimizing attention
Attention is the most expensive component across architectures, so optimization is delicate.
Two strategies: implementation improvements (better kernels) and
new algorithms (better complexity).
FlashAttention: cut the memory round-trips the naive kernel bounces S and P through HBM; the fused kernel keeps them on-chip
0/–
naive · extra HBM traffic
Q,K,V → HBMwrite S → HBMread Swrite P → HBMread Pwrite O → HBM
FlashAttention · fused
Q,K,V stay on-chipS computed in SRAMP used immediatelyO written once
Implementation improvements (FlashAttention, PagedAttention) squeeze the existing algorithm.
New algorithms (sliding-window, linear, compressed, multi-latent, Mamba) change its complexity.
FlashAttention
Fused kernel that eliminates excess HBM reads/writes (through B200). Basic attention is a handful of lines; FlashAttention is tens of thousands.
PagedAttention
Partitions the KV cache into blocks (“pages”) accessed via a lookup table, so it fits fragmented GPU memory instead of one contiguous block.
Sliding window
Attend to the previous w tokens only: quadratic → N·w. w is often 8K–32K.
Gated attention
Training-time layers approximate attention over context chunks in linear time per chunk.
Linear attention
Replaces the quadratic softmax equation with a linear-time approximation.
Compressed attention
Periodically compresses earlier context; attention reads compressed history plus recent tokens.
Multi-latent attention
Approximates attention in a low-dimensional latent space.
Mamba / hybrids
Selective state-space models scale linearly with sequence length; hybrid models mix Mamba and attention.
Recap
Test yourself
Twelve questions across the chapter. Click an answer to see the explanation.