shreyansh

Observability and Operations

30-Second Glance

  • Observability = metrics (what's happening), logs (why, per request), traces (where time went across services) — interviews expect all three, each with its sampling/retention budget stated.
  • RED (services) + USE (infrastructure) tell you what to measure; SLOs/SLIs + error budgets tell you when to act — alert on burn rate, not on raw thresholds.
  • Capacity planning is arithmetic: peak QPS × cost-per-request ÷ per-host headroom, plus headroom for one AZ failure (N+1) and growth runway.
  • Every HLD answer ends with "how I'd know it's broken": the 3–5 golden-signal dashboards and the two pages (burn-rate, saturation) I'd wire on day one.

The Three Pillars (and Their Budgets)

Metrics: numbers over time

Counters (requests, errors), gauges (queue depth, connections), histograms/summaries (latency distributions — never average latency alone; quote p50/p95/p99). Prometheus pull-model + Grafana is the default answer; CloudWatch/Datadog for managed. Cardinality discipline is the senior point: a user_id label on a metric creates millions of series and kills the TSDB — aggregate by endpoint/region/status, keep high-cardinality to traces/logs. Retention: ~15s scrape, 15-day raw + downsampled years (Thanos/Cortex/Mimir or managed equivalents).

Logs: discrete events with context

Structured JSON (timestamp, trace-id, service, level, message, key fields) so they're queryable in ELK/Loki/CloudWatch Insights — never free-text-only. Every log line carries the trace/request ID injected at the gateway (see Microservices Patterns). Cost control: INFO for business events, DEBUG off in prod except per-request dynamic sampling, and retention tiers (hot 7–14 days, cold archive to S3/GCS for compliance). A service emitting >10 MB/s of logs per host has a logging problem, not a traffic problem.

Traces: the request's journey

Distributed tracing (OpenTelemetry → Jaeger/Tempo/X-Ray) stitches spans across service hops so a 900 ms checkout decomposes into gateway 5 ms → auth 20 ms → inventory 700 ms (found it). Sampling is the budget: head-based 1–5% plus tail-based retention of all errors/slow traces (keep 100% of failures, 1% of successes). Trace context (W3C traceparent) must propagate through queues too — the outbox/saga events from the Microservices Patterns note carry trace IDs or cross-service debugging goes blind at the broker (same for Messaging and Event Queues lag attribution).

loading diagram…

RED and USE: What to Measure

  • RED (for request-driven services): Rate (req/s), Errors (fraction 5xx + business failures), Duration (p50/p95/p99 per endpoint). One RED dashboard per service, with the top-5 endpoints split out.
  • USE (for resources/infra): Utilization (% busy), Saturation (queue depth / waiting work — the leading indicator), Errors (dropped packets, disk errors, OOM kills). Apply per resource: CPU, memory, disk, NIC, connection pools, Kafka consumer lag (saturation of the pipeline), DB replicas lag.
  • The interview synthesis: "RED tells me users are hurting; USE tells me which resource will hurt them next — I alert on RED burn and capacity-plan on USE saturation."

Golden signals (Google SRE's overlapping set: latency, traffic, errors, saturation) map 1:1 — cite either acronym, they're interchangeable currency.

SLOs, SLIs, Error Budgets

  • SLI (indicator): the measured ratio — e.g. "fraction of checkout requests < 500 ms over 30 days" or "queue-drain freshness: events processed within 60s."
  • SLO (objective): the target — 99.9% ("three nines" = 43 min downtime/month ≈ 8.6s/day budget), 99.95%, 99.99% (52 min/year — needs multi-region, no single-AZ design qualifies).
  • Error budget: 1 − SLO — the allowed failure rate (0.1% for 99.9%). Policy hook: budget healthy → ship features fast; budget burning → freeze deploys, prioritize reliability. This is how you answer "how do you balance velocity vs reliability" — with the budget number, not philosophy.
  • SLA is the contractual cousin (with penalties/credits); never conflate — SLOs are internal, SLAs are promises to customers, and SLOs should be stricter than SLAs.

Numbers to memorize: 99% = 7.2h/mo; 99.9% = 43m/mo; 99.95% = 22m/mo; 99.99% = 4.3m/mo. Multi-AZ single-region caps around 99.95%; 99.99%+ needs multi-region active-active (and the consistency compromises in CAP, PACELC and Consistency Models).

Alerting: Burn Rate, Not Thresholds

Bad alerting pages on CPU > 80% (noisy, user-invisible). Good alerting pages on error-budget burn rate:

  • Fast burn: 2–5% of monthly budget in ≤1 hour (e.g. 99.9% SLO burning at 10x) → page immediately.
  • Slow burn: budget exhausting over days → ticket, investigate in work hours.
  • Multi-window pairs (1h + 6h windows) to catch both spikes and inevitable exhaustions without flapping.

Supporting rules: alert on symptoms (latency/error/freshness SLIs), not causes (disk 90% is a ticket + auto-remediation, not a 3am page); every alert names its runbook link and its SLO; alert fatigue gets fixed by tuning, never by ignoring ("if it pages and nobody acts, delete or demote it"). Consumers of async designs alert on lag/depth slope (see the messaging note); scheduled jobs alert on missing-success (dead-man's switch), not just failure.

Capacity Planning: Arithmetic, Not Vibes

The five-step interview calc:

  1. Peak load: p99/peak QPS (not average — provision for peak + growth; e.g. 10k avg → 50k peak event traffic).
  2. Cost per request: profiled CPU/RAM/IO per request type (e.g. 20 ms CPU on 8-core box ≈ 400 rps/core theoretical, derate to ~50% headroom → ~1.5k rps/host).
  3. Hosts = peak / per-host-at-target-utilization: target ≤60–70% steady-state so bursts and rolling deploys fit.
  4. N+1 (or N+2): survive one AZ/rack failure — 3 AZs, each able to absorb another's load (~50% headroom minimum for single-AZ-loss survival).
  5. Stateful systems separately: DB/shard counts from data size × replication factor ÷ per-node limit (e.g. 10 TB × RF3 ÷ 2 TB/node ≈ 15 nodes + headroom), Kafka partitions from throughput ÷ ~10 MB/s/partition, connection pools from (hosts × pool size) < DB max connections (the classic pool-explosion outage).

Add growth runway (3–6 months at current growth) and a load-test gate (Gameday/stress to 2x peak before the launch review). Autoscaling (HPA on custom metrics like queue lag, not raw CPU) covers diurnal shape; the plan covers the floor.

Operational extras to name-drop in closing minutes: deploy safety (rolling + canary 1%→10%→100% with automatic rollback on SLO burn), chaos/Gameday drills, blameless postmortems with action-item tracking, and on-call rotation sized so nobody carries >25% of weeks.

Trade-offs

Approach Pros Cons Use when
Metrics-first (RED/USE dashboards) Cheap, real-time, alertable No per-request causality; cardinality traps Always — the baseline
Verbose unstructured logging Easy to add, human-readable Unqueryable at scale, cost explosion, PII leakage Never in prod paths — structure it or sample it
Structured logs + trace IDs Per-request causality, cheap grep-to-cause Still no cross-service timing Every service, day one
Full distributed tracing (OTel) End-to-end latency decomposition Instrumentation + sampling + storage cost Any request crossing ≥2 services — i.e. all microservices
100% trace/log retention Never miss evidence 10–100x cost for 1% of value; slows queries Compliance-mandated paths only; sample the rest
Threshold alerts (CPU, disk) Simple to write Noisy, user-invisible, fatigue Capacity tickets + auto-remediation, never pages
Burn-rate SLO alerts Pages iff users hurt; self-tuning to SLO Needs well-defined SLIs first The paging path — always

Interview One-Liners

  • Q: Metrics, logs, traces — do I need all three? A: Metrics for real-time symptoms, traces for where time went across services, logs for why on a specific request — sampled and budgeted so the bill doesn't exceed the value.
  • Q: What do you measure on a new service? A: RED per endpoint plus USE per resource, each log line carrying the trace ID and each event crossing the queue keeping it.
  • Q: How do you set an SLO? A: From user tolerance backwards (checkout p99 < 500 ms at 99.9%), SLO stricter than any SLA, error budget gating deploy velocity — burn the budget, freeze features.
  • Q: What pages you at 3am? A: Only fast-burn-rate SLO alerts and saturation-with-no-headroom — everything else is a ticket; any page without a runbook gets fixed or deleted.
  • Q: How many hosts for 50k peak rps? A: Per-host rps at 60–70% target utilization, divided into peak, plus N+1 across 3 AZs and growth runway — autoscaling shapes diurnal, the plan sets the floor.
  • Q: Average latency looks fine but users complain? A: Averages hide tails — check p95/p99 per endpoint in traces; one slow dependency (find it by span breakdown) poisons the mean while most requests stay fast.
  • Q: How do you deploy safely? A: Canary 1%→10%→100% with automatic rollback on SLO burn, gated by load tests at 2x peak and verified in Gameday drills.