shreyansh

WhatsApp Chat

TL;DR — 30-Second Glance

  • Stateful connection gateways hold 100–200k persistent TLS/WebSocket conns each; any gateway can serve any user, with a user → gateway routing table in Redis for delivery fan-out.
  • Store-and-forward: every message is persisted before ack; receipts (sent → delivered → read) are separate idempotent events keyed by client_msg_id.
  • Ordering: per-sender sequence for 1:1 chats; server-assigned sequence per group (single-writer partition by group_id) for total order in groups.
  • Presence at scale works only via deltas (publish on change, subscribe to online contacts) — never broadcast heartbeats.
  • E2E encryption (Signal): the server is a blind router; spam/abuse must use metadata + user reports, never content.

1. Requirements

Functional

  • 1:1 messaging: text, media (image/video/audio/doc), replies, deletes ("delete for everyone" within a window).
  • Group messaging: up to ~1024 members; join/leave, admin controls, group metadata.
  • Delivery receipts: single tick (sent), double tick (delivered), blue ticks (read).
  • Online presence + last-seen + typing indicators.
  • Multi-device: phone + up to 4 companion devices, each with independent connection.
  • Offline queue: messages held for offline users (up to 30 days), then expired.
  • Media upload/download via resumable URLs.

Non-functional

  • Low send latency: p99 < 300 ms in-region for online recipients.
  • High availability: chat must survive single-DC failure; no SPOF on the send path.
  • Ordering: causally sensible order per conversation on every device.
  • Durability: a message acked as "sent" is never lost.
  • Confidentiality: E2E encrypted; server cannot read message bodies.

Out of scope

  • Status/Stories feed, payments, business API analytics, full-text server-side search of bodies (impossible under E2E).

2. Capacity Estimation

Assumptions (WhatsApp-scale, rounded for interviews):

Quantity Assumption
Registered users 2 B
DAU ~1 B
Messages/day 100 B (public figure, ~2020)
Avg message record (text + metadata) ~500 B
Fraction with media attachment ~5%, avg media ~300 KB
Peak concurrency ~20% of DAU online → 200 M concurrent conns
Peak/avg traffic ratio ~4×

Derivations:

  • Avg send rate = 100e9 / 86,400 ≈ 1.16 M msgs/s; peak ≈ 4.6 M msgs/s.
  • Chat-record ingress (text path): 1.16 M/s × 500 B ≈ 580 MB/s avg (~2.3 GB/s peak). Delivered messages are deleted, so steady-state chat storage is small (offline + sync windows only).
  • Media ingress: 5% × 100 B = 5 B media/day × 300 KB ≈ 1.5 PB/day; with 30-day retention for undelivered/unfetched media ≈ 45 PB before erasure coding.
  • Connections: 200 M concurrent / 200k conns per gateway box ≈ 1,000 boxes + 30% headroom ≈ 1,300 gateways. At ~50 KB RAM per conn (socket + TLS + buffers), one box holds ~10 GB just for conn state — this is why gateways are stateful but disposable.
  • Presence heartbeats: naive 200 M users × 1 ping/60 s ≈ 3.3 M pings/s — handled as cheap UDP/TCP keepalives at the gateway layer, not as presence publishes. Publishes happen only on online/offline transitions (~2 per session), cutting presence write load by ~1000×.

3. API Design

Transport: persistent WebSocket (or custom TCP + TLS) for send/receive/presence; HTTPS REST for registration, media, and history sync.

# Session
CONNECT wss://chat.edge/whatsapp?device_id=<d>&token=<auth>
  -> { user_id, server_time, resume_token }

# 1:1 / group send (same primitive; group has conv_id of the group)
SEND_MESSAGE { conv_id, client_msg_id (uuid, idempotency key),
               sender_seq (per-sender counter), ciphertext, msg_type,
               media_blob_id?, reply_to? }
  -> ACK_SENT { client_msg_id, server_seq, server_ts }

# Receipts (idempotent, last-write-wins per message per device)
SEND_RECEIPT { conv_id, server_seq, receipt: DELIVERED | READ, device_id }
FETCH_OFFLINE { since_usn (user sequence number) }
  -> { messages[], receipts[], next_usn }

# Groups
CREATE_GROUP { title, member_ids[] } -> { group_id }
ADD_MEMBERS { group_id, member_ids[], group_version } -> { group_version+1 }
LEAVE_GROUP { group_id }

# Presence
SET_PRESENCE { status: ONLINE | OFFLINE }      # usually implicit from socket
SUBSCRIBE_PRESENCE { user_ids[] }              # only for visible chats
  -> PRESENCE_EVENT { user_id, status, last_seen }   # deltas only

# Media (REST)
POST /v1/media:init { byte_size, sha256 } -> { blob_id, upload_urls[] }
GET  /v1/media/{blob_id} -> 302 to CDN/object-store URL

Key design points: client_msg_id makes retries safe (server dedups); sender_seq enables gap detection for 1:1 ordering; resume_token/since_usn enables cheap reconnect without full sync.

4. Data Model & Storage

Entity Store Key / layout
Users, devices, keys SQL (sharded by user_id) users(user_id, phone_hash, …), devices(device_id, user_id, key_bundle)
Conversations + members SQL (sharded by conv_id) conversations(conv_id, type, group_version, …)
Messages (recent window) Wide-column (Cassandra/Scylla), partition by conv_id, clustering by server_seq messages(conv_id, server_seq, sender_id, client_msg_id, blob_ref, ts) with TTL after delivery + sync window
Per-user sync log (offline queue) Log/KV sharded by user_id: user_seq → event backs FETCH_OFFLINE; trimmed on ack; 30-day expiry
Presence Redis Cluster, sharded by user_id presence:{user_id} = {status, last_seen, version} (short TTL, refreshed by gateway)
Routing table Redis Cluster route:{user_id} = {gateway_id, connected_at}
Media blobs Object storage + CDN content-addressed by sha256 → natural dedup of viral forwards

Why this split: hot paths (routing, presence) are tiny KV ops; message history is append-only wide rows; the per-user sync log is the durability primitive for offline/multi-device fan-out. See Relational vs Non-Relational and MongoDB vs Cassandra for the SQL-vs-wide-column split.

5. High-Level Design

Flow: client → L4/L7 edge LB → connection gateway (holds socket, owns keepalive) → chat service (auth, sequencing, fan-out) → message store + per-recipient sync logs → recipient gateway (via routing table) or offline queue; presence service on the side; media service off the text path.

loading diagram…

Send path (online 1:1): sender gateway → chat service persists message (partition by conv_id) → appends to recipient's sync log → looks up recipient gateway in routing table → pushes; recipient acks → receipt event flows back. If recipient is offline, the push step is skipped and the sync log waits. Group send: chat service assigns the next server_seq for the group, persists once, then fans out to N sync logs (see Deep Dive B for large groups).

6. Deep Dives

A. Connection management with no SPOF

  • Gateways are stateful but disposable: all durable state lives in the message store/sync logs, so any gateway may die — clients reconnect (jittered backoff, e.g. 1s → 2s → … capped at 60s + jitter) to a healthy gateway via DNS/anycast + LB health checks, then resume with resume_token/since_usn.
  • Routing table (route:{user_id} in Redis, TTL ~2× keepalive interval) maps user → current gateway. A stale entry only causes one failed push, after which the sender re-resolves — delivery itself never depends on gateway memory.
  • Scale knobs: ~200k conns/box needs kernel tuning (file descriptors, tcp_tw_reuse, busy-poll), TLS session resumption, and compact per-conn structs. Gateways are sharded by nothing — pure horizontal scale behind the LB; capacity planning is "concurrent conns ÷ conns-per-box + headroom" from §2.
  • Reconnect storms (e.g. network flap affecting millions) are damped by client-side jitter + server-side staged sync (receipts first, then messages paginated).

B. Single vs group messaging and ordering

  • 1:1: each sender numbers its messages (sender_seq). The receiver buffers out-of-order arrivals and fills gaps via FETCH_OFFLINE. No server coordination needed — cheap and correct.
  • Groups: N senders ⇒ sender sequences are incomparable. The chat service partition owning group_id (see Partitioning) acts as single-writer sequencer, stamping a monotonic server_seq. This gives total order per group; conflicts (concurrent membership changes) are serialized through a group_version compare-and-swap.
  • Large-group fan-out: writing N=1024 sync-log entries per message is fine (≈1k KV writes, batchable), but for very large/announcement-style groups flip to pull: persist once, bump a group high-watermark, and let members fetch deltas. Rule of thumb: push for normal groups, pull past a fan-out threshold.
  • Multi-device: each device has its own sync cursor over the user's log; a per-user sequence number (USN) keeps all devices converged. Ordering across devices uses Lamport clocks-style reasoning; strict global order is not linearizable — and doesn't need to be.

C. Presence without melting the system

  • Publish transitions only (offline→online, online→offline, with debounce ~30–60s to avoid flapping writes).
  • Subscribe narrowly: a client receives presence only for users in currently visible chats, not its whole contact list. Fan-out per transition ≈ online subscribers (tens), not contacts (hundreds).
  • Presence records carry a version; last-writer-wins with version compare. Gateways refresh a short-TTL key; expiry ⇒ implicitly offline, so crashed gateways fail safe.
  • last_seen is served from Redis (see cache-aside patterns); privacy settings ("nobody") are enforced in the presence service before fan-out.

7. Bottlenecks, Trade-offs & Monitoring

Approach Pros Cons Use when
Stateful gateways + Redis routing 200k conns/box, cheap push Stale routes, reconnect logic on client Always for chat; push is the product
Server-sequenced groups Total order, simple clients Hot partition for mega-groups Groups ≤ ~1k; pull-based past that
Store-and-forward + sync log Durable, multi-device falls out Storage for offline window Always; trim aggressively on ack
E2E (Signal) True privacy, smaller compliance surface No server search/mod tools, E2E multi-device is harder Consumer chat; see note below

Bottlenecks: gateway RAM/FDs (mitigate: more boxes, lean conn structs); hot-group partitions (split sequencer from fan-out workers); presence storms (debounce + jitter); media egress cost (CDN caching, client-side dedup by hash before upload).

Monitoring: p99 send latency, gateway conn count/FD usage, offline-queue depth and age, receipt lag (sent→delivered), presence publish rate, reconnect rate, media upload success.

Interview one-liners

  • Q: Why not a pure stateless design? A: Push needs a socket home; statelessness is recovered by keeping all durable state outside the gateway.
  • Q: How do you order group messages? A: Single-writer sequencer per group_id stamps server_seq; clients render in that order.
  • Q: What breaks at 10× scale? A: Gateway fleet size and presence fan-out — both scale horizontally, but presence needs delta-only discipline.
  • Q: E2E encryption note? A: Signal protocol, server is a blind router; keys via a key directory with safety numbers. Consequences: no server-side content moderation/search; abuse handled via metadata heuristics + user reports; backups need separate encrypted-backup keys.