Serialization formats: JSON, Protobuf, and Avro

How it works

A producer maps an application object into an agreed representation; a consumer decodes those bytes into usable fields. For example, a location update might contain driver_id, latitude, longitude, and recorded_at. Both sides must agree on field meanings, including coordinate units and timestamp precision, not just the encoding.

JSON represents data as text with field names, such as {"driver_id":"d17","latitude":40.7}. It is easy to inspect and widely supported, making it a practical default for public HTTP APIs. JSON does not require a separate schema, but production APIs still need a documented, validated contract; large integer identifiers can also lose precision in clients that represent all numbers as floating point.

Protocol Buffers, often called Protobuf, defines messages in .proto files and commonly generates code for producers and consumers. Its binary encoding identifies fields by numeric tags rather than repeating their names. For example, driver_id might use field number 1; removing that field should reserve its number and name so a later field cannot accidentally reuse them.

Avro defines schemas separately from its compact binary records. Decoding requires the writer's schema, and schema resolution lets a reader use a compatible, different schema. Avro container files include their schema; an event-stream integration commonly carries a schema identifier and retrieves the schema from a registry instead.

Compatibility matters because deployments and stored data outlive individual processes. A new Protobuf field can usually be ignored by an older binary reader, but that reader will not understand its meaning. In Avro, adding a reader field generally needs a default so the reader can process older records that lack it. Test both old-reader/new-writer and new-reader/old-writer combinations.

The format does not choose your transport or delivery guarantees: Protobuf is not gRPC, and Avro is not Kafka. An HTTP endpoint or message queue can carry different encodings. Define framing, content type, schema identification, maximum message size, and validation alongside the format so receivers can safely interpret each payload.

A worked example

Choosing an encoding for an analytics ingestion stream

Suppose an analytics collector receives 100,000 events per second. In a representative benchmark, assume JSON averages 600 bytes per event and Avro averages 240 bytes, before compression. That is 60 MB/s versus 24 MB/s of payload entering the stream, excluding protocol overhead and replication. Keep JSON at the browser-facing endpoint for client simplicity, then validate and encode records as Avro internally. Attach a schema identifier and retain every referenced writer schema for replay. Before committing, compare compressed batches and parsing CPU: compression may narrow the size gap. Add a nullable campaign_id with a default of null so new readers can process older records.

Trade-offs to say out loud

Readability versus compactness: JSON is convenient to inspect directly, while Protobuf and Avro often reduce payload size but require schema-aware tools for debugging.
Explicit contracts versus development overhead: Schemas and generated code catch structural mistakes earlier but add build steps, compatibility checks, and coordination across languages.
Performance depends on the workload: Measure encoded size, encode/decode CPU, allocations, and compression on representative messages rather than assuming binary is always faster or smaller.
Schema infrastructure becomes a dependency: Registry-based decoding needs schema caching, durable schema retention, and a policy for registry outages so replay does not depend on an unavailable lookup.
Wire compatibility is weaker than semantic compatibility: Changing a timestamp from milliseconds to seconds can preserve its numeric encoding while silently breaking every consumer's interpretation.
Long retention raises migration costs: Durable events and cached payloads may require readers to support historical encodings and schemas long after producers have upgraded.

When to reach for it

A high-volume ingestion or fanout path makes bytes per message and parsing CPU significant capacity inputs.
Services written in different languages need a clear, shared data contract.
Producers and consumers deploy independently, including mobile clients that may remain on older versions.
Events are retained for replay, analytics, or rebuilding derived state, so historical schemas must remain readable.
An external API needs broad client support and easy inspection, making JSON a strong starting point.

How to bring it up in the interview

Choose per boundary rather than declaring one format for the entire system: JSON for an external API and Protobuf or Avro internally can be reasonable if translation costs are justified.
Tie the choice to a constraint: estimate message rate times encoded size, then propose a benchmark that includes compression and parsing CPU; interviewers may ask whether network or compute is actually the bottleneck.
Walk through one schema change, such as adding an optional device type, and explain how old readers, new readers, and historical records behave.
For durable streams, explain how schema identifiers are stored, how schemas remain available during replay, and what happens to undecodable records; expect probes about registry outages and poison messages.
State what serialization does not solve: a schema cannot provide ordering, deduplication, authentication, or business-rule validation.

Common mistakes

Calling JSON schemaless and treating field names, types, and semantics as an informal agreement.
Reusing Protobuf field numbers or assuming any field-type change is safe because the payload still decodes.
Assuming Avro records always embed their full schema, or that a schema registry automatically makes every change compatible.
Choosing a binary format solely from generic performance claims without testing realistic payloads, batching, and compression.
Treating successful decoding as valid input instead of enforcing message-size limits, field bounds, authorization, and application invariants.

Where it comes up

Interview questions where serialization formats is part of a strong answer, and why.

Questions people ask

What is the difference between JSON, Protobuf, and Avro?

JSON is a text format that carries field names and is easy to inspect. Protobuf uses a declared message schema and a binary encoding with numeric field tags. Avro uses a writer schema to decode compact binary records and supports resolution against a compatible reader schema.

Should I use Protobuf or Avro for Kafka messages?

Both can work; Kafka transports bytes and does not require either format. Protobuf is a reasonable fit when teams already use generated message contracts, while Avro is useful when writer/reader schema resolution fits the data pipeline. The more important design decisions are compatibility rules, schema retention, and reliable decoding during replay.

Is Protobuf always faster and smaller than JSON?

No universal result applies to every payload, library, or runtime. Protobuf often uses fewer bytes for structured messages, but compression, field types, batching, and implementation quality affect the outcome. Benchmark representative traffic and include encode/decode CPU rather than comparing payload size alone.