30-Second Glance
- Scale up (bigger box) until economics or a hard ceiling stops you; scale out (more boxes behind a load balancer) for anything that must survive a box dying.
- L4 balances TCP/UDP connections (~100k+ rps per HAProxy/NLB node, no payload inspection); L7 routes on HTTP semantics (path, header, cookie) at ~1–5 ms added latency for TLS termination + parsing.
- No single algorithm wins: round-robin for uniform stateless workers, least-connections for uneven request costs, consistent-hash when the backend itself holds state (caches, shards).
- Health checks (active probes + passive outlier ejection) are what turn "N boxes" into actual fault tolerance — a balancer that keeps sending traffic to a dead node is just a fancy SPOF.
- LB tiers stack: DNS/GSLB at the edge → L7 edge LB → service-level LB / sidecar mesh inside the cluster.
Vertical vs Horizontal Scaling
Vertical scaling (scale-up) means a bigger machine: more cores, more RAM, faster NVMe. A single well-tuned box today can serve surprising load — nginx pushes ~50k+ static rps on modest hardware, a single Postgres primary can do ~10–30k simple-read TPS. Scale-up is operationally trivial: no distributed state, no coordination, linearizability for free.
It fails three ways, and interviewers expect all three named:
- Hard ceiling. You cannot buy infinite cores (~100s of vCPUs on cloud instances), RAM (~10s of TB), or single-NIC bandwidth (~100–200 Gbps). One beefy box also shares one failure domain.
- Superlinear cost. Cloud pricing for the top-end instance shapes is disproportionate — two 32-vCPU boxes are usually cheaper than one 64-vCPU box, and give you 2 availability zones.
- Blast radius. One box = one SPOF unless you add a standby, at which point you are doing distributed systems anyway.
Horizontal scaling (scale-out) adds identical, preferably stateless nodes behind a load balancer. Capacity math becomes linear-ish: if one app server sustains 2k rps at p99 < 200 ms with 60% CPU headroom, 10 servers ≈ 20k rps, minus ~5–10% balancer + coordination overhead. The price is everything state-related: sessions, caches, and connections must live somewhere shared (see Cache Patterns and Eviction), and deploys/autoscaling become real subsystems.
The standard interview answer: scale up while one box comfortably holds the load with headroom (rule of thumb: peak < 50–60% of a single node's tested capacity), scale out the stateless tiers first, and push state into purpose-built stores (DB, Redis) rather than keeping it on app servers.
L4 vs L7 Load Balancing
| L4 (transport) | L7 (application) | |
|---|---|---|
| Inspects | IP + TCP/UDP ports | Full HTTP/gRPC: path, headers, cookies, body hints |
| Routing power | Per-connection only | Path/host routing, canary by header, auth-aware, retries |
| Cost | Very cheap: ~10–100 µs added, 1M+ concurrent conns per node | 1–5 ms added (TLS termination + parsing); terminates TLS so backends stay plain HTTP |
| TLS | Pass-through (can't see inside) | Terminates TLS (cert lives on LB), re-encrypts or not to backends |
| Protocol limits | Anything TCP/UDP (DBs, Redis, custom) | HTTP-family only |
| Examples | AWS NLB, HAProxy in TCP mode, Maglev, IPVS | AWS ALB, nginx, HAProxy HTTP mode, Envoy, Traefik |
Practical numbers: a single HAProxy node does ~100k+ L4 rps or ~40–60k L7 TLS-terminated rps; AWS ALB scales in Load Balancer Capacity Units (~3.5k new conns/s, 1 Gbps per LCU, auto-scales with minutes of lag — pre-warm before known spikes). Rule: L4 when you need raw throughput or non-HTTP protocols; L7 the moment routing depends on anything inside the request.
Balancing Algorithms
| Algorithm | How it picks | Pros | Cons / Use when |
|---|---|---|---|
| Round-robin | Cycles evenly | Trivial, fair when requests cost the same | Fails with heterogeneous nodes or mixed cheap/expensive endpoints |
| Weighted RR | Weights ∝ capacity | Handles mixed instance sizes during migrations | Weights go stale; not load-aware |
| Least-connections | Fewest open conns | Best default for long-lived/variable-cost requests (WebSockets, streaming) | Needs accurate conn accounting; poor for very short requests (all counts ≈ 0) |
| Least response time / latency-aware | Fastest recent p99 (e.g. Envoy, Finagle-style) | Self-corrects for slow nodes, GC pauses, noisy neighbors | Herding risk — damp with jitter/subsetting |
| Consistent-hash (on key/client IP) | hash(key) → ring → node; only K/N keys move on membership change |
Backend-state affinity: caches stay warm, shards stable | Hot keys stay hot — pair with bounded loads / virtual nodes; see Partitioning |
| Sticky sessions (cookie/IP-hash) | Same client → same server | Lets a stateful legacy app scale out without rework | Breaks even spreading; resubmits on failover; prefer stateless + shared session store instead |
Notes interviewers probe: consistent hashing uses O(100–200) virtual nodes per physical node to smooth the ring; on node add/remove only ~K/N keys remap vs ~all keys with mod-N hashing. Sticky sessions are a migration crutch, not a design — they couple availability (server dies → sessions die) to routing.
Health Checks and Failure Detection
- Active checks: LB probes
/healthzevery 5–30 s; mark down after 2–3 consecutive failures, mark up after 2–3 successes. Separate liveness (restart me) from readiness (stop sending traffic, e.g. during deploy warmup or dependency outage). - Passive checks / outlier ejection: track real-traffic errors and latency (Envoy's defaults: eject after 5 consecutive 5xx, eject for 30 s, max 10% of hosts). This catches gray failures active probes miss.
- Drain on deploy: deregister → finish in-flight (deregistration delay, typically 30–300 s) → terminate. Skipping drains is the #1 cause of deploy-time 502s.
- Budget the detection window explicitly:
interval × unhealthy_thresholdis your added MTTR floor (e.g. 10 s × 3 = 30 s of partial blackholing before ejection).
Where the LB Sits
- Global (GSLB / DNS): Route53 latency-based or GeoDNS, Cloudflare/Akamai anycast. Steers users to a region/PoP. Coarse (DNS TTL caching, typically 30–300 s to shift traffic) but global. First answer to "multi-region failover."
- Edge / ingress L7: ALB / Cloudflare / nginx ingress terminates TLS, enforces WAF + Rate Limiting, routes by host/path to services.
- Service-level / east-west: kube-proxy / ClusterIP, Envoy sidecars in a mesh (Istio/Linkerd) — per-service balancing, retries, circuit breaking, mTLS. Handles the internal fan-out the edge LB never sees.
- Client-side: smart clients with a membership list (gRPC xDS, Cassandra drivers). Zero-hop balancing; cost is client complexity and thundering config pushes.
Between tiers sit the cross-cutting concerns: API Gateways own edge policy (auth, composition), while CDN/Edge absorbs static and cacheable load before it ever reaches your LBs. Async absorb-spikes paths go through Stream Processing and Kafka vs RabbitMQ rather than synchronous scale-out.
Trade-offs
| Approach | Pros | Cons | Use when |
|---|---|---|---|
| Vertical scale-up | Zero architecture change; strong consistency free | Cost curve; hard ceiling; SPOF | Early stage; stateful primaries (DB leader) sized with headroom |
| Horizontal scale-out, stateless | Linear capacity; AZ/region fault tolerance; cheap units | Needs shared state stores; deploy/autoscale machinery | App tiers, API workers — the default past product-market fit |
| L4 LB | Max throughput; protocol-agnostic; cheap | No content routing; TLS opaque | DB/Redis fronting, extreme-rps TCP, DSR return paths |
| L7 LB | Smart routing, TLS offload, WAF/auth hooks | Added latency; must scale the LB tier itself | Any HTTP service needing path/header routing or canary |
| Sticky sessions | No shared session store needed | Uneven load; failover loses state | Legacy migration only; plan the exit to stateless + Redis |
| Client-side balancing | No LB hop; scales with clients | Stale membership; per-language client work | High-throughput internal RPC (gRPC/xDS), DB drivers |
Interview One-Liners
- Q: One server handles 2k rps; you need 50k rps. How many boxes?
A: Not 25 — target ≤60% utilization plus N+1/AZ redundancy: ~25 / 0.6 ≈ 42, rounded to span 3 AZs. Always state headroom and redundancy assumptions. - Q: L4 or L7 for a WebSocket chat tier?
A: L4 or L7 with least-connections (connections are long-lived; RR spreads count but not load). Sticky only if session state can't move; otherwise externalize presence to Redis. - Q: How do you deploy without 502s?
A: Readiness gates + connection draining (30–300 s deregistration delay) + health-check thresholds; rolling with maxUnavailable small. - Q: Why is consistent hashing better than mod-N for cache nodes?
A: Node add/remove remaps ~K/N keys instead of nearly all; virtual nodes smooth the distribution. Same math as in Rebalancing. - Q: ALB latency spiked during a flash sale — what happened?
A: LCU auto-scale lag (minutes); fix is pre-warming the ALB plus edge caching and queue-based admission for the spike. - Q: When would you skip the LB entirely?
A: Client-side balancing for hotspot internal RPC, or DNS/anycast steering at global tier — every hop must justify its latency and cost.
Related
- Partitioning — same hashing math applied to data instead of traffic
- Rebalancing — membership change without full remap
- Cache Patterns and Eviction — where shared state lives once app servers go stateless
- Content Delivery and Edge — shedding load before it reaches the LB tier
- API Design and Gateways — edge policy (auth, routing, composition) above the LB
- Rate Limiting and Throttling — admission control in front of scaled-out tiers