Twitter Timeline

1. Requirements

Functional:

1. Post tweet (280 chars + up to 4 media refs + poll/quote refs).
2. GET /home-timeline — ranked mix of followed authors, ads, who-to-follow (cursor pages).
3. GET /user-timeline/{user} — reverse-chron author archive (public, CDN-cacheable).
4. Follow/unfollow, like, retweet/repost, reply threading.
5. Delete cascades from timelines within ~1 min.

Non-functional:

Out of scope:

• Full-text/keyword search index (separate search stack).
• DMs, Spaces/live audio, long-form articles.
• Counter-abuse / spam classifiers beyond hooks.

2. Capacity Estimation

Assumptions: 400M users, 200M DAU; 500M tweets/day; avg 200 followers; 20% tweets carry media (avg 1 MB); peak 4× (sport finals); home-timeline cache 800 ids × 64 B ≈ 50 KB/user.

3. API Design

POST /api/v2/tweets
{ "text": "...", "media_ids": ["m_1","m_2"],
  "reply_to": null, "quote_of": "t_88" }
201 { "tweet_id": "t_71xQ2", "created_at": "...", "fanout": "pending" }

GET /api/v2/home-timeline?limit=20&cursor=...
200 { "items": [{ "tweet_id": "t_71xQ2", "text": "...", "author": {...},
                 "entities": {"media": [...], "poll": null},
                 "like_count": 1204, "rank_reason": "followed:elonmusk" }],
      "next_cursor": "..." }

GET /api/v2/users/{id}/tweets?limit=20&cursor=...   // user timeline, CDN-cacheable
POST /api/v2/tweets/{id}/like | /retweet | /delete // 204; counts via Kafka

4. Data Model & Storage

-- tweets_by_id (tweet_id TIMEUUID PK, author_id, text, media_ids, reply_to, created_at)
-- tweets_by_author ((author_id), created_at DESC, tweet_id)  -- user-timeline range scan
-- home:{user_id} ZSET score=rank member=tweet_id, MAXLEN 800
-- celebrity outbox: celeb:{author_id} LIST (capped 3000)

• Bodies immutable except delete (tombstone → async timeline scrubber removes ids within ~1 min).
• Retweet = new tweet row with retweet_of pointer (keeps author-timeline semantics clean; dedup at rank time).

5. High-Level Design

Write path: tweet → API → tweet store → fan-out decider → normal: Kafka → workers push to followers' home zsets (+ author timeline row) / celebrity: celebrity-outbox only. Read path: home-timeline → Redis zset → hydrate (tweet-cache → Cassandra) → merge celebrity outboxes + ads + WTF → rank/filter → page. User-timeline: CDN → author head-cache → tweets_by_author scan (pure pull, no fan-out).

• Ingest service: auth, validation, dedup (client mutation-id), persist, enqueue — p99 budget owned here.
• Fan-out service: the News Feed hybrid, sharded by follower-range; early-rank pre-scores at write time (author affinity buckets) so reads merge cheaply.
• Timeline service (read): zset paging + hydration + mixing; owns cursor codec and seen-filter.
• Media pipeline: upload → transcode/thumbnail/NSFW → CDN (async; tweet renders progressively).
• Graph service: follow edges + WTF candidate generation input.

6. Deep Dives

6.1 Hybrid fan-out at Twitter scale + peak events

Directly extends News Feed case §6.1 with two Twitter-specific twists. (1) Threshold is dynamic: follower-count cutoff auto-lowers when fan-out lag exceeds SLA (a mid-size account going viral mid-match gets promoted to pull-path within minutes). (2) Early ranking at ingest: each pushed id carries a precomputed partial score (author-affinity bucket, media boost, language match); read-time ranker merges rather than scores from scratch — keeps the 200 ms p99 while merging 3 sources + ads. World-Cup-final playbook: pre-lower thresholds, pre-scale workers on lag (not CPU), shed WTF/ads mixing first (timelines stay correct, just less monetised), serve stale-while-revalidate past 30 s lag.

6.2 Media and graph objects (the non-text half)

• Media objects: client uploads bytes direct to object storage via presigned URL (origin never proxies bytes); async workers produce variants (thumb/small/large, video HLS renditions) + blurhash placeholder + NSFW/safety labels; tweet row references media_ids with per-variant CDN URLs. Tweet-accept only waits for the metadata row, never transcode — media renders progressively. Hot media (viral video) is CDN-cached at edge; origin shield + signed URLs for sensitive content.
• Graph objects: follow edges in a dedicated adjacency store (FlockDB-style: sharded by (user_id, edge_type)), because fan-out reads follower lists at 1M+/s and a general RDBMS buckles. Forward edges (who I follow) feed the pull path; reverse edges (my followers) feed push sharding. WTF and ads candidates come from graph walks (2-hop) + offline embeddings, injected at mix time — which is why they're the first thing shed under load (§6.1).

6.3 Home vs user timeline: why two systems

Consequence: profile views of a celebrity cost ~zero origin (CDN), while home timelines of their followers cost a bounded merge. Deletes: tombstone in tweet store → scrubber removes id from home zsets (sampled scan, ~1 min) + purge CDN user-timeline key. Say in the interview: "user timeline is a cacheable archive; home timeline is a per-viewer computation — different SLOs, different stores."

7. Bottlenecks, Trade-offs & Monitoring

First bottleneck at 10x: fan-out worker lag during global events → dynamic celebrity thresholds + lag-based autoscaling + shed mixing (ads/WTF) before correctness. Next: home-zset Redis memory → 800→400 cap + 7 d→3 d TTL for low-activity users. Next: graph-store reverse-edge reads for mega-accounts → precomputed follower-shard snapshots.

Interview one-liners

• Q: How is this different from News Feed? / A: Same hybrid core, plus dynamic celebrity thresholds, early-rank at ingest, and split home (per-viewer compute) vs user (cacheable archive) timelines.
• Q: World Cup final — what breaks first? / A: Fan-out lag; playbook is lower thresholds, lag-based worker scale, shed ads/WTF mixing first.
• Q: Why doesn't tweet-post wait for video transcode? / A: Accept persists metadata + enqueues; media renders progressively from blurhash to variants.

Related

• News Feed case — prerequisite; hybrid fan-out, ranking, cache tiers
• Instagram case — media pipeline sibling (heavier video/photo focus)
• Cache Patterns and Eviction — zset + CDN layering, stampede guards
• Partitioning — timeline/follower-range sharding
• Stream Processing — fan-out queue, counters, early-rank stream
• MongoDB vs Cassandra — tweet store choice
• Kafka vs RabbitMQ — fan-out transport choice