Part of Index of Inference Engineering. Book: Ch 6 (Modalities). Many modalities reuse LLM architecture and tooling — same engines, different bottlenecks.
1. Vision Language Models (VLM)
- Image/video tokens injected alongside text; prefill cost scales with visual token count.
- Video = more frames = more tokens = prefill-bound. Levers: frame downsampling, visual prefix caching, chunked prefill.
- Omni-modal models share one LLM backbone across text/audio/vision.
2. Embedding models
- Encoder-only: single forward pass, no decode phase. Latency ≈ prefill only; work is throughput-bound.
- Batch aggressively, normalize + pool per model card. Used for RAG/retrieval — often the first model in a pipeline.
3. ASR (speech → text)
- Single-chunk (short utterance) vs long-file (chunk + stitch). Chunking trades latency for accuracy.
- VAD segments streams into speech chunks; diarization (who spoke when) is a separate stage.
4. TTS / speech-to-speech
- Streaming TTS: first-audio latency is the TTFT equivalent — the perceived-latency metric.
- Speech-to-speech = ASR + LLM + TTS pipeline, or a unified model; chaining adds end-to-end latency.
5. Image generation
- Diffusion/DiT: iterative denoising = many forward passes per image. Latency scales with steps.
- Levers: few-step distilled models, kernel/attention optimization, quantization. Batch size helps throughput.
6. Video generation
- Attention optimization + quantization; context parallelism across GPUs.
- Latency and memory dominate;
$ per clipis the business metric.
7. Do
- Run one small VLM (e.g. Qwen2.5-VL) and one embedding model on the L4; log TTFT + tok/s.
- Measure an image model's latency vs number of steps (or read a Diffusers benchmark).
8. Interview one-liners
- "VLM and video inference is prefill-bound — more visual tokens, not more decode."
- "Embeddings have no decode phase; optimize batching and throughput, not decode."