Messaging and Event Queues

Why Queues Exist in HLD

Synchronous RPC chains multiply failure: if service A calls B calls C, each with 99.9% availability, the chain is 99.7% — and one slow dependency blocks threads across the whole path. A queue breaks that coupling in time (producer and consumer need not be alive simultaneously), in rate (bursts get buffered instead of rejected), and in topology (one event can feed N consumers without the producer knowing them).

Say this in the interview: "I'll put a queue here so a 10x traffic spike or a 30-second downstream deploy doesn't page anyone — the queue absorbs it and consumers drain at their own pace." Interviewers promote candidates who reach for async boundaries at exactly the write path (order placed, payment captured, video uploaded) and keep reads synchronous.

Typical numbers to quote: SQS standard ~300 msgs/sec per queue API action (effectively unlimited with batching, but ~ms-to-hundreds-of-ms latency), RabbitMQ tens of thousands msgs/sec per queue with sub-millisecond-to-ms latency, Kafka single partition ~10–50 MB/s write, cluster scales linearly to millions of msgs/sec with ~5–10 ms p99 produce latency (batch linger dependent).

The Three Models: Queue vs Pub-Sub vs Stream

Point-to-point queue

One message → exactly one consumer. Competing consumers on the same queue give horizontal scale-out of work: add consumers, drain faster. Examples: SQS standard/FIFO, RabbitMQ queue, Celery/Redis-backed task queues.

Use for work distribution: send-email jobs, image-thumbnail tasks, order-fulfillment commands. Each message is a unit of work that must be done once (or at-least-once with dedup).

Pub-sub (topic + subscriptions)

One message → all active subscribers, each with its own cursor. The broker fans out; publishers never know who consumes. Examples: SNS → SQS fanout, Redis pub-sub, RabbitMQ fanout/topic exchanges, Google Pub/Sub.

Use for notifications and decoupling: "order.created" feeds the warehouse service, the email service, and the analytics pipeline independently. The classic AWS pattern is SNS topic → N SQS queues so each subscriber gets durable buffering (pure SNS/Redis pub-sub drops messages for offline subscribers; the SQS leg fixes that).

Partitioned log / stream

One append-only, durably stored log split into partitions; consumers track offsets and can replay history. Examples: Kafka, Redpanda, Kinesis, Pulsar. Retention is time/size-based (days to forever), not "delete on ack" — the log is the source of truth, which is why this model powers event sourcing and the Stream Processing note's pipelines.

Use when you need replay (rebuild a cache, onboard a new consumer, re-run a buggy job over last week's events), per-key ordering at scale, or multiple independent consumers reading at their own pace.

Positioning detail lives in Kafka vs RabbitMQ — link it, don't re-derive it: RabbitMQ is a smart-broker task router (per-message acks, rich routing, low latency at moderate volume); Kafka is a dumb-broker durable log (sequential I/O, zero-copy, throughput scales with partitions). SQS is the "no-ops, integrates with Lambda/autoscaling, but 256 KB message cap and no ordering unless FIFO (300 TPS per message-group ID)" option.

Delivery Semantics: At-Least-Once Is the Default

• At-most-once: fire and forget. Consumer crashes after receiving but before processing = message lost. Only for metrics, heartbeats, anything loss-tolerant.
• At-least-once: broker redelivers until acked (visibility timeout expiry in SQS, unacked redelivery in RabbitMQ, offset not committed in Kafka). Duplicates are guaranteed in failure windows — a crash between processing and ack always double-processes. This is the default every interviewer expects you to design for.
• Exactly-once (EOS): possible only end-to-end inside one transactional system — Kafka's idempotent producer (sequence numbers, enable.idempotence=true) + transactions (transactional.id, consume-transform-produce atomically) + a transactional sink (Kafka Streams, Flink with checkpointing). Costs: ~10–20% throughput hit, transaction coordinator as a chokepoint, and it still breaks the moment you call a non-transactional side effect (send an email, charge a card). So the standard answer: at-least-once delivery + idempotent consumers — dedupe keys (message ID / idempotency key stored with a TTL in Redis/DB with unique constraint), and make handlers re-runnable (upserts, not increments; state-machine transitions guarded on current state).

SQS specifics worth quoting: standard queues = at-least-once, best-effort ordering, near-unlimited throughput; FIFO = exactly-once processing per message-group (content-based dedup ID, 5-min dedup window), 300 TPS per group without batching / 3,000 with batching, ordering guaranteed within a group. RabbitMQ: at-least-once with manual acks + publisher confirms; quorum queues (Raft-based replication) over classic mirrored queues for durability. Kafka: per-record acks (acks=all, min.insync.replicas=2 for survive-one-broker-loss), consumer offsets committed only after processing.

Ordering: Per-Key, Never Global

Global ordering at scale is a contradiction — one total order means one writer, one partition, one throughput ceiling. Real systems order per key: Kafka partition key (user-id, order-id), SQS FIFO message-group ID, RabbitMQ single-queue single-consumer. Consequences to state explicitly:

1. Pick a cardinality that matches scale: partitioning by user_id with 100 partitions means one hot celebrity key still caps at one partition's throughput, and adding partitions later reorders history (key→partition mapping changes unless you over-partition upfront — e.g. 10x current need).
2. Ordering + competing consumers conflict: to preserve order you must route one key to one consumer (sticky assignment), which reintroduces hot-partition skew.
3. Cross-aggregate ordering (deduct inventory AND notify seller in order) cannot come from the broker — use sequence numbers/version vectors at the consumer (see Version Vector) or a saga (see Microservices Patterns).

Failure Handling: Retries, Backoff, DLQ

The interview answer has four named parts:

1. Retry with exponential backoff + jitter (e.g. 1s → 2s → 4s … capped, plus ±20% jitter to avoid thundering herds when a dependency recovers). Bounded — 3–5 attempts, not infinite.
2. Dead-letter queue (DLQ): after max attempts, the message moves to a DLQ (SQS redrive policy maxReceiveCount, RabbitMQ dead-letter exchange, Kafka dedicated *.DLT topic via framework or manual produce). DLQs need their own alarms (depth > 0 for > N minutes pages) and a replay path (redrive button / replay job) — a DLQ nobody monitors is just a second trash can.
3. Poison messages: a malformed message that crashes every consumer causes infinite redelivery and head-of-line blocking. Defenses: try/catch → immediate DLQ on deserialization errors (don't retry what will never succeed), and per-message timeouts.
4. Visibility/lease discipline: SQS visibility timeout must exceed the p99 processing time or a slow-but-healthy consumer loses its message to a duplicate worker; extend via heartbeat for long jobs. Same idea as consumer-group session timeouts in Kafka (max.poll.interval.ms).

Backpressure: What Happens When Consumers Can't Keep Up

Unbounded buffering just moves the crash from the service to the broker (or OOMs the consumer). Name one mechanism per interview:

• Bounded queues + explicit overflow policy: fixed-size in-memory queue; on full, block (slows producer — natural backpressure), drop-oldest/newest (metrics, live video frames), or spill to disk/Kafka (durable but slower).
• Pull over push: Kafka/Kinesis consumers pull at their own pace — the broker never overwhelbs them; lag (consumer offset vs log end) becomes the single scaling metric ("add partitions/consumers when p99 lag > 60s"). RabbitMQ push with prefetch count (basic.qos) is the same idea: prefetch 10–50, not unlimited.
• Load shedding + autoscaling: shed low-priority traffic first (drop recommendation refreshes, keep checkouts), autoscale consumers on queue depth/lag, and page on sustained growth (queue depth growing for 15 min = capacity incident, not a blip).
• Circuit breaking upstream callers so a saturated downstream fails fast instead of queueing timeouts — see Microservices Patterns.

Size the buffer with Little's Law thinking: 10k msgs/sec × 30s of tolerated downstream outage = 300k messages ≈ at 1 KB each, ~300 MB — fits in Kafka trivially, needs explicit SQS depth alarms, and will OOM an in-memory queue.

Trade-offs

Interview One-Liners

• Q: At-least-once or exactly-once for an order pipeline? A: At-least-once delivery with idempotent consumers — dedupe on order-id unique constraint; exactly-once only inside a single Kafka-transactional pipeline and it dies at the first external API call.
• Q: How do you guarantee ordering across two microservices? A: You don't get it from the broker — order per key in one partition, and sequence cross-aggregate effects with version numbers or a saga, accepting no global total order.
• Q: Kafka or RabbitMQ for this design? A: RabbitMQ for routed task queues under ~100k msgs/sec with per-message acks and complex routing; Kafka for durable replayable streams, per-key ordering, and anything feeding stream processing — full breakdown in the Kafka vs RabbitMQ note.
• Q: What happens to messages that keep failing? A: Bounded retries with exponential backoff + jitter, then DLQ with depth alarms and a redrive/replay path; deserialization failures go straight to DLQ without retry.
• Q: Consumers are falling behind — how do you know, and what do you do? A: Alert on consumer lag / queue depth slope, not just depth; autoscale consumers on lag, shed low-priority traffic, and check for a poison message blocking the partition head.
• Q: SQS standard or FIFO? A: Standard for throughput and simplicity when duplicates/out-of-order are tolerable; FIFO per message-group when ordering + dedup matter, accepting 300–3,000 TPS per group.
• Q: How big should the queue buffer be? A: Throughput × tolerated outage duration × message size via Little's Law — e.g. 10k rps × 30s × 1 KB ≈ 300 MB — and overflow policy (block/drop/spill) must be explicit.

Related

• Kafka vs RabbitMQ — deep broker comparison (linked, not duplicated)
• Stream Processing — what consumes the log downstream
• Microservices Patterns — saga/outbox patterns that publish to these queues
• CAP, PACELC and Consistency Models — why the queue is AP-flavored buffering in front of CP state
• Version Vector — consumer-side ordering across aggregates