Notification System

1. Requirements

Functional

• Multi-channel send: mobile push (APNs/FCM), SMS, email; extensible to WhatsApp/voice.
• Templating: versioned templates with variables, locale fallback, per-channel rendering (e.g. SMS ≤ 160 GSM chars/segment).
• Targeting: direct user send + topic/segment broadcast.
• Priority: P0 (OTP, fraud alert) → P1 (transactional) → P2 (engagement) → P3 (marketing).
• Preferences: opt-in/out per channel/category, quiet hours, frequency caps.
• Tracking: sent/delivered/opened/clicked/failed events; per-message status query.
• Retries with backoff; dead-letter queue with operator replay.

Non-functional

• P0 latency: p99 < 10 s end-to-end (OTP usability threshold).
• Throughput: absorb 10× bursts (flash sales) without losing P0 latency.
• At-least-once delivery with dedup so retries never double-send an OTP charge.
• Provider resilience: survive a vendor outage by failing over.
• Cost awareness: SMS ~$0.005–0.05/segment dominates; prefer push/email when allowed.

Out of scope

• Building push/SMS networks themselves; message composition UI; full marketing-automation journey builder.

2. Capacity Estimation

Derivations:

• Queue ingress at peak: 60 k msgs/s × 2 KB ≈ 120 MB/s — trivially handled by a partitioned log (see Message queues).
• Dedup store: 3 B keys in window × ~100 B/key (hash + TTL) ≈ 300 GB in Redis — sharded, TTL-evicted; OTP-only shorter window (15 min) shrinks the hot set.
• SMS cost sanity: 150 M SMS/day × $0.01 ≈ **$1.5 M/day** — this is why preference checks, push-first routing, and SMS segment collapsing are architectural (not cosmetic) decisions.
• Push fan-out: one broadcast to 50 M devices is fanned out by workers into provider-batched calls (FCM multicast ≤ 500 tokens/call ⇒ 100 k calls), paced by the provider rate limiter.

3. API Design

# Core send — idempotency_key is REQUIRED
POST /v1/notifications
{ "idempotency_key": "evt_9f2… (uuid)",
  "user_id": "u_123",                       # or topic/segment selector
  "channel": "auto | push | sms | email",   # auto = push-first fallback chain
  "template_id": "otp_login",
  "template_version": "7",                  # pinned; else latest stable
  "locale": "en-IN",
  "params": { "code": "482913" },
  "priority": "P0",
  "ttl_seconds": 300,                       # OTP expires; do not retry past it
  "dedupe_window_hours": 24 }
-> 202 { "notification_id": "n_abc", "status": "queued", "deduped": false }

GET /v1/notifications/{id} -> { status, channel_used, attempts[], events[] }

# Templates (versioned, reviewed like code)
POST /v1/templates { name, channel_variants: { push:{title,body}, sms:{body}, email:{subject,html} } }
POST /v1/templates/{id}:publish { version }

# Preferences & caps
PUT /v1/users/{id}/preferences { push:{marketing:false}, quiet_hours:{…}, caps:{sms_per_day:5} }

# Provider delivery callbacks
POST /webhooks/delivery/{provider} { provider_msg_id, status, … }  # signature-verified

Fallback chain example (channel: auto): push → if unregistered/bounced, SMS → if SMS fails and TTL allows, email. Each hop is a new attempt on the same notification_id, preserving dedup.

4. Data Model & Storage

5. High-Level Design

Flow: client → intake API → preference/cap check → dedup → template render → router (channel + priority) → per-channel priority queues → dispatcher workers → provider gateway (rate limiter → vendor A/B) → webhooks update status; scheduler handles delayed retries; DLQ + replay console.

Physically: intake and dispatch scale independently; queues (Kafka topics or SQS-like per channel×priority: 3 channels × 4 priorities = 12 logical queues) absorb bursts; the provider gateway is the only component that talks to vendors, centralizing throttling, signing, and failover.

6. Deep Dives

A. Priority queues and starvation-free scheduling

• Why separate queues: a single FIFO lets a 50 M-recipient marketing blast head-of-line-block OTPs for minutes. Isolation by (channel, priority) bounds the blast radius.
• Scheduling: dispatchers use weighted-fair sharing (e.g. weights P0:P1:P2:P3 = 16:8:2:1) with strict preemption for P0 — idle P3 capacity is borrowed, but P0 never waits behind P3. Within a queue, FIFO with per-user pacing so one user's 1 k-event burst doesn't delay others (deficit round-robin by user_id hash).
• Scheduled/quiet-hour holds: messages whose user is in quiet hours park in a delay queue (checked pre-send, not pre-queue, so preference flips apply) — except P0, which overrides quiet hours by policy.
• Backpressure: if a provider slows, its queue grows; autoscale dispatchers on queue depth/age, and shed P3 first (drop with SLO_SHED status + metric, never silently).

B. Per-provider rate limits

• Each vendor publishes different limits (e.g. SMS aggregator: 100 msg/s per long-code, 1 k/s per short-code; email ESP: hourly/daily quotas; FCM/APNs: per-project throughputs and per-connection pacing). The gateway holds a token bucket per (provider, account/sender-id), sized from the contract, not from wishful thinking.
• Adaptive limits: on 429/5xx, multiplicative-decrease the bucket and retry with backoff; on sustained success, additive-increase back toward contract. This tracks vendor-side dynamic throttling that static configs miss.
• Failover: ordered vendor list per channel with health scores (error rate, p99 latency over 5 min). Failover is per-message with a retry budget, and failback is gradual (5% → 50% → 100%) to avoid flapping.
• Circuit breaker per vendor: trip after N consecutive failures, half-open probe after cooldown — prevents retry storms from hammering a dying vendor (and your wallet).
• Related building block: Rate limiting and backpressure.

C. Dedup and retries with backoff

• Dedup: SETNX idempotency_key at intake returns the existing notification_id on replay — the client retries safely, and the status API is the read-your-write path. Keys cover (event, user, channel-class); a payment event and its reminder use different keys by construction.
• Retry policy matrix: transient (timeout, 5xx, 429) → retry; permanent (invalid token, unsubscribed, template error) → DLQ immediately; expired (now > created + ttl) → drop with TTL_EXPIRED. OTP: max ~3 attempts within 5 min; email: up to ~8 attempts over 24 h.
• Backoff: exponential (1s, 2s, 4s, …) with full jitter, capped; jitter is load-bearing — without it, a vendor blip resubmits as a synchronized thundering herd. Delayed retries live in a delay queue / scheduled topic, not in worker memory.
• DLQ + replay: poison messages (bad template var, revoked vendor creds) park with error class; operators fix and replay selectively. Redrive rate is itself throttled.

7. Bottlenecks, Trade-offs & Monitoring

Bottlenecks: vendor quotas (mitigate: multi-vendor + pre-warmed quotas before big campaigns); dedup Redis memory (shorter windows for low-risk classes); template-render CPU at 60 k/s (precompile + cache rendered fragments); webhook ingestion spikes (async ack + queue).

Monitoring: per (channel, priority, vendor): queue depth and age (age catches slow-drain), send→deliver latency, delivery rate, 429 rate, failover events, DLQ depth by error class, SMS segment count (cost), preference-filter rate.

Interview one-liners

• Q: How do OTPs survive a marketing blast? A: Separate queues + weighted-fair scheduling with P0 preemption; P3 sheds first.
• Q: Exactly-once? A: No — at-least-once delivery + at-most-once effects via idempotency keys at intake and provider-side dedup where supported.
• Q: Vendor is down — what happens? A: Health-scored failover to vendor B within the retry budget; circuit breaker stops the bleeding; DLQ catches the rest.
• Q: Biggest cost lever? A: Push-first routing + preference/cap checks before queueing; SMS is ~1000× the cost of push.

Related

• Message queues — queue topology, delay queues, DLQ
• Rate limiting — token buckets, adaptive limits, circuit breaking
• cache-aside — preferences/caps caching
• Stream Processing — delivery-event pipelines and analytics
• Kafka vs RabbitMQ — choosing the queue substrate