30-Second Glance
- JSON: universal, human-readable, schemaless — 2–5× bigger and slower; right for public APIs, configs, debugging.
- Thrift/Protobuf: IDL + codegen, compact binary (3–10× smaller than JSON), fast; right for internal RPC (Twitter/Finagle, gRPC).
- Avro: schema with the data (or registry), best-in-class schema evolution for storage/streams (Hadoop, Kafka + Schema Registry).
- Compatibility rules decide upgrades: never reuse field IDs, only add optional/defaulted fields, gate wire changes behind registry checks.
The Comparison Matrix
| Dimension | JSON (+Schema/Avro-JSON) | Apache Thrift | Protocol Buffers (proto3) | Apache Avro |
|---|---|---|---|---|
| Format | Text, schemaless (JSON Schema optional) | Compact binary (+ JSON/simplejson transports) | Compact binary (varint, packed repeated) | Compact binary + schema (file header or registry ref) |
| Schema language | None built-in | IDL (.thrift), rich types + services + exceptions |
IDL (.proto), messages + services, weak enums |
JSON schema; schema is data |
| Codegen | No (any parser) | Yes (most languages, incl. unions/exceptions) | Yes (official + gRPC first-class) | Optional — generic/datum readers common |
| Size vs JSON | 1× (baseline) | ~3–6× smaller | ~3–10× smaller | ~3–8× smaller |
| Encode/decode speed | Slowest (text parse, float formatting) | Fast (binary, simple framing) | Fastest-typical (varint + packed + arena alloc) | Fast reads (no field IDs on wire); schema resolution cost |
| Schema evolution | Ad-hoc (ignore unknown keys by convention) | Field IDs; optional/default; explicit compat discipline | Field numbers; optional/defaults; unknown-field retention |
Reader/writer schema resolution: add/remove with defaults, aliases, promotion |
| RPC story | REST (OpenAPI) | Mature end-to-end RPC (TBinary/TCompact, multiplexed, Finagle) | gRPC (HTTP/2, streaming, deadlines, load balancing) | None native — pairs with Kafka/REST + Schema Registry |
| Storage story | Logs, configs, documents | Less common | Possible (length-delimited) but awkward | Best: OCF files carry schema; splittable, compressible |
| Canonical users | Public APIs, configs, MongoDB/Elastic docs | Twitter, Facebook internal, Evernote, Pinterest | Google, gRPC ecosystem, Envoy/xDS, K8s etcd | Hadoop, Kafka (Confluent/Apicurio registry), data lakes |
loading diagram…
Encoding Details Worth Quoting
- Protobuf varints: small ints cost 1 byte (
1→0x01); negativeint32withoutsint32costs 10 bytes (sign-extension trap — ZigZagsint32/sint64fixes it).packedrepeated numerics drop per-element tags (~2× win on arrays). - Thrift TCompact: varint + field-id deltas + bit-packed booleans; TBinary is the debuggable 1:1 layout. Service IDL (methods, exceptions,
oneway) is richer than proto services — no streaming, but excellent versioning via field IDs. - Avro wire: no field IDs on the wire at all — values in schema order, so payloads are tiny, but reader and writer schemas must be reconciled (registry fetch + resolution). Aliases + defaults + type promotion (
int→long→float→double) are the evolution toolkit. - JSON cost: field names on every record (
"user_id":12345≈ 12 bytes overhead per field per record), string escaping, float round-trips. At 100K msgs/s this is Mbps of names alone — the standard justification for binary internally.
Sizes on a typical 10-field event: JSON ~250–400 bytes → Protobuf/Thrift ~40–90 bytes → Avro ~35–80 bytes (plus one-time schema). At 1M events/s Kafka ingest, JSON-vs-binary is ~2–3 Gbps vs ~0.5 Gbps — a whole network tier.
Schema Evolution: The Rules That Prevent Outages
- Never reuse a field number/ID. Deleted
field 5stays retired forever; reuse silently corrupts old data into new semantics. - Only add optional (defaulted) fields. New readers handle missing (default); old readers ignore unknown (Protobuf retains, Thrift skips, Avro resolves). Required additions break old writers.
- Never change a field's type incompatibly (
int32 → string,enumrenumbering). Prefer a new field + deprecate the old. - Gate with a registry in streams/storage. Kafka + Schema Registry (Avro/Protobuf/JSON-Schema) enforces BACKWARD/FORWARD/FULL compatibility per subject before a producer deploys — this check has prevented more outages than any codec choice.
- Enums need an unknown/default arm. Every client must handle unseen enum values (proto3 open enums, Thrift explicit default, Avro aliases) or the first new value crashes old fleets.
Choosing in an HLD Answer
| Context | Pick | One-line justification |
|---|---|---|
| Public/external API, webhooks, configs | JSON (+ OpenAPI) | Debuggability and universality beat bytes; compress with gzip |
| Internal RPC, polyglot services | gRPC + Protobuf (or Thrift if legacy/exceptions-heavy) | Codegen, streaming, deadlines, ecosystem |
| Kafka/event log / lakehouse storage | Avro (+ Schema Registry) | Schema evolution + compact files, unmatched for long-lived data |
| Cache payloads, feature flags at edge | Protobuf/Thrift or MessagePack | Bytes × QPS = real money; keep JSON at the boundary only |
| ML features / analytics interchange | Parquet (columnar files) + Avro/Proto events upstream | Columns for scans (see Column-Oriented Storage), rows for streams |
Trade-offs
| Approach | Pros | Cons | Use when |
|---|---|---|---|
| JSON | Zero tooling, human-readable, universal | 2–5× size, slow parse, ad-hoc evolution | Boundaries, configs, debugging, public APIs |
| Thrift | Compact + full RPC IDL (services, exceptions) | Smaller ecosystem than gRPC today; fewer streaming stories | Internal RPC in Thrift shops, exception-rich APIs |
| Protobuf | Compact + fastest typical + gRPC/HTTP2 streaming | Opaque bytes; proto2/proto3 migration quirks; weak nullability story | Default internal RPC/service mesh |
| Avro | Best evolution + self-describing storage | Needs registry discipline; no RPC; resolution cost | Kafka, Hadoop, lakehouse — data at rest and in streams |
Interview One-Liners
- Q: Why not JSON for everything? A: Field names per record + text parsing cost 2–5× size and the slowest encode/decode — fine at the edge, wasteful at 1M msgs/s internally.
- Q: Protobuf vs Thrift in one line? A: Both IDL+codegen binary; Protobuf pairs with gRPC streaming and the larger ecosystem, Thrift has richer service/exception IDL — pick by org standard.
- Q: Why is Avro the Kafka default? A: No field IDs on the wire (tiny), reader/writer schema resolution with defaults/aliases/promotion, and registry-enforced compatibility before deploy.
- Q: What breaks when you reuse a field number? A: Old bytes decode under new semantics — silent corruption across versions, the worst kind. Retire numbers forever.
- Q: How do you add a field safely? A: Optional with a default, never required; new readers default it, old readers skip/retain it — then enforce BACKWARD/FULL in the registry.
- Q: Negative ints blowing up Protobuf size — why?
A: Plain
int32sign-extends negatives to 10-byte varints; usesint32/sint64(ZigZag) for possibly-negative values. - Q: Where does Parquet fit vs these four? A: Parquet is columnar file storage for analytics (see columnar note), not a row/stream codec — events arrive as Avro/Proto, land as Parquet.