YouTube Video Platform

1. Requirements

Functional

• Upload (resumable, up to hours of video), progress + retry per chunk.
• Transcode to multiple resolutions/bitrates (240p → 4K) + thumbnails + previews.
• Playback with adaptive bitrate (ABR), seek, captions, playback-speed.
• Metadata: title, description, tags, visibility, likes, comments (separate subsystem).
• View counts, watch-time analytics for creators.
• Search/discovery hooks (index metadata + captions; ranking out of scope).

Non-functional

• Upload success: resumable to byte-level; tolerate client disconnects for days.
• Time-to-first-rendition: popular path (720p) ready in minutes, not hours.
• Playback: startup < 2 s, rebuffer ratio < 0.5%, global p99 segment fetch < 500 ms.
• Durability 11×9s on originals; availability via CDN even if origin region fails.
• Scale: 500+ hours uploaded/min, billions of playbacks/day.

Out of scope

• Live streaming (separate ingest path), recommendation engine internals, comment moderation, DRM license server internals (Widevine/FairPlay integration noted only).

2. Capacity Estimation

Derivations:

• Raw ingress: 30,000 min/min × 60 MB ≈ 1.8 TB/min ≈ 2.6 PB/day. With renditions (1.5×) + erasure coding (1.4×): ≈ 5.5 PB/day of new stored bytes. (This is why per-title encoding — fewer renditions for talking-head content — and lifecycle tiers for unwatched videos are first-class design, not polish.)
• Transcode compute: 1 min of 1080p ≈ ~1–5 CPU-min (codec-dependent); 30,000 content-min/min ⇒ 30–150 k vCPUs sustained — a dedicated elastic worker fleet (spot/preemptible) is mandatory.
• Playback egress: 1 B hrs/day × ~1 GB/hr (720p-ish avg) ≈ 1 EB/day — hence CDN offload ≥ 95%; origin serves only misses + manifests. Edge cache math: top 1% of videos serve ~80%+ of views ⇒ a few PB of hot edge cache covers the planet.
• Metadata: uploads 30k videos/min ⇒ ~43 M videos/day?? No — 500 hrs/min at ~10 min avg video = 50 videos/min ≈ 72 k/day; each metadata row ~10 KB ⇒ ~700 MB/day, trivial. (State the avg-duration assumption explicitly in interviews.)

3. API Design

# Resumable upload (sessions survive for days)
POST /v1/uploads:init { filename, byte_size, sha256, mime }
  -> { upload_id, chunk_size: 8MB, urls[] (signed, per chunk) }
PUT /v1/uploads/{upload_id}/chunks/{index}  (binary, Content-Range)
  -> { received: true }                       # idempotent per chunk hash
POST /v1/uploads/{upload_id}:complete { chunk_hashes[] }
  -> { video_id, status: TRANSCODING }

# Metadata & publishing
PATCH /v1/videos/{video_id} { title, description, visibility, thumbnails }
GET   /v1/videos/{video_id} -> { metadata, renditions[], status, manifests{} }

# Playback (ABR client picks rendition per segment)
GET /v1/videos/{video_id}/manifest.m3u8   # HLS master: BANDWIDTH-tagged variants
GET /v1/videos/{video_id}/hls/720p/seg_{n}.ts
GET /v1/videos/{video_id}/dash/manifest.mpd

# View beacon (rate-limited, deduped server-side)
POST /v1/videos/{video_id}:view { session_id, watch_ms, player_events }
  -> 204

Chunk hashes make re-uploads idempotent; the manifest is generated once per rendition and cached at edge with long TTL; segments are immutable ⇒ Cache-Control: immutable, max-age=1y.

4. Data Model & Storage

Bytes and metadata never mix: the metadata row is kilobytes; segments stream straight from edge. See Object storage and CDNs.

5. High-Level Design

Flow: client → upload API (session + signed chunk URLs) → object store assembles original → transcode orchestrator (DAG) → renditions back to object store → manifest publish → metadata READY → CDN warms → playback; view beacons → counter pipeline → warehouse.

6. Deep Dives

A. Transcoding DAG

• Split: the original is split at GOP (keyframe) boundaries into ~5–10 s shards. GOP-aligned splits transcode independently with no quality seam — this is the key insight that makes the DAG embarrassingly parallel.
• Map: worker fleet pulls shard tasks from a priority queue (new uploads from subscribed/large channels first; backlog scavenged by spot workers). Each task: decode → scale → encode per rendition ladder (per-title: static talking-head gets fewer rungs than sports).
• Reduce/package: mux shards per rendition, emit HLS .ts/fMP4 + DASH segments, write master manifests, record rendition set in metadata, flip status READY progressively (720p published before 4K finishes — progressive readiness is a product feature).
• Failure handling: shard tasks are idempotent (deterministic output keys); dead workers just requeue; poison shards (corrupt GOP) quarantine after N attempts with operator tooling. Spot/preemptible workers cut the 100 k-vCPU bill by ~70%.

B. Chunked streaming and CDN

• Why segments: 4–6 s segments over HTTPS need no special servers, cache perfectly (immutable), and let the client adapt bitrate every few seconds based on measured throughput. Seeking = fetch the segment index for that timestamp — no server work.
• Tiered caching: edge PoP → regional shield → origin. A miss fetches from shield, not origin, collapsing 100 edge misses into 1 origin fetch (origin shield). Hot videos are pushed (pre-warmed) to edges on publish for big channels.
• Long tail: unwatched videos live in cheap object tiers; first view pays an origin-fill latency cost (acceptable — cold content has no audience waiting). Manifests get short-TTL edge caching so re-transcodes propagate in minutes.

C. View counting at scale

• Write-behind counters: edge/player beacons → regional aggregators bump in-memory views:{video_id}; flush to durable store every ~60 s. A video with 1 M views/hour generates ~17 flushes/min, not 1 M writes — a 1000×+ reduction.
• Fraud/dedup: count a view once per (user/session, video, window) with autoplay < N seconds excluded; naive loop-replay filtered by velocity rules. Exactness is explicitly traded: public counts are eventually consistent (±flush interval), money-facing creator analytics reconcile from the warehouse log.
• Uniques: HyperLogLog per video/day (~12 KB for 2% error) instead of set-of-viewers; watch-time sums via the same flush path. Audit trail = raw beacon log in cold storage, re-aggregatable.

7. Bottlenecks, Trade-offs & Monitoring

Bottlenecks: transcode fleet sizing at upload spikes (autoscale on DAG depth + SLA per priority class); origin egress on viral misses (shield + pre-warm); metadata DB read fan-out on homepages (cache-aside video cards, see cache-aside); beacon ingestion (sample aggressively for mega-viral videos).

Monitoring: time-to-720p-ready, transcode queue depth by priority, rebuffer ratio, startup latency, CDN hit ratio by tier, origin egress, view-flush lag, fraud-filter rate.

Interview one-liners

• Q: Why is upload resumable at chunk level? A: Hour-long uploads on mobile networks will fail; 8 MB idempotent chunks make retry cost constant.
• Q: How do segments enable ABR? A: Each rendition is independent 5 s files; the client re-picks bandwidth every segment.
• Q: Exact view counts? A: No — write-behind + HLL; exactness lives in the warehouse, freshness on the page.
• Q: Biggest cost? A: Rendition storage + egress; per-title ladders and 95%+ edge offload.

Related

• CDNs — tiered cache, origin shield, pre-warm
• Object storage — originals, segments, lifecycle tiers
• cache-aside — metadata and manifest caching
• Message queues — transcode task DAG
• Stream Processing — beacon aggregation pipeline