How it works
Start by identifying readers, writers, and how long each version can remain active. A database migration might need to support two application versions during a rolling deployment; an event stream may need to support consumers reading several years of historical records.
Define compatibility from the reader's perspective. Backward compatibility means a new reader can read data written with an old schema. Forward compatibility means an old reader can read data written with a new schema. Full compatibility requires both; compatibility with only the immediately preceding version may not cover older retained data.
Prefer additive changes with explicit missing-field behavior. For example, adding an optional campaign_id to a click event lets new consumers handle older events where it is absent. Old consumers must tolerate the extra field, and the serialization format and validation rules must actually permit that behavior.
For database changes, use expand–migrate–contract. To rename customer_id to account_id, first add account_id, then deploy code that maintains both fields and tolerates rows not yet migrated. Backfill existing rows, verify consistency, switch reads, and remove customer_id only after old code and the rollback window are gone.
Treat wire-format rules separately from business meaning. Protobuf field numbers must not be reused, while Avro schema resolution depends on writer and reader schemas and their defaults. Even a structurally valid change can be unsafe: changing an amount from dollars to cents silently changes its meaning.
Test the reader/writer combinations that will actually coexist, including rollback versions and historical records. A schema registry can enforce configured structural compatibility rules, but it cannot prove business correctness. Monitor decode failures, missing-field fallbacks, and migration progress before retiring the old representation.
A worked example
Renaming a field without stopping writes
A service has 80 million customer rows and must rename customer_id to account_id without changing its meaning. First, add a nullable account_id column. Deploy code that writes both columns in the same transaction and reads account_id with a fallback to customer_id. Wait until all old writers have drained before starting the backfill. Copy values in resumable batches, updating only rows where account_id is still null. Throttle batches based on database load and replication lag, then verify that the columns agree. Switch reads to account_id while continuing to maintain customer_id for rollback. After the rollback window closes and dependent jobs have migrated, stop writing the old column and remove it in a later release.
Trade-offs to say out loud
- Independent deployments require compatibility work
- Supporting mixed versions avoids coordinated releases but adds defaults, fallback paths, and tests that must eventually be removed.
- Additive changes postpone cleanup
- Keeping deprecated fields preserves compatibility but increases payload size, storage use, and ambiguity about which representation is authoritative.
- Backfills compete with production traffic
- Large rewrites consume database capacity and can cause replication lag, so they need throttling, checkpoints, and concurrency-safe updates.
- Versioning adds operational overhead
- A new API or event version isolates breaking changes but creates parallel contracts, adapters, and a retirement process.
- Replay extends the support horizon
- Long-lived events require readers or translation code that understands historical schemas, even after every live producer has upgraded.
- Rollback becomes constrained
- Once new writers produce data that old code cannot interpret, rolling back application binaries alone may no longer be safe.
When to reach for it
- Services or mobile clients deploy independently and exchange structured payloads.
- A database field must be renamed, split, retyped, or made mandatory without downtime.
- Consumers replay retained events or rebuild derived views from historical data.
- A rolling upgrade must keep old and new servers, workers, or agents interoperable.
- A migration changes data representation while writes continue.
How to bring it up in the interview
- Name the compatibility boundary first: database rows, public API responses, event payloads, or persisted workflow state. Interviewers will probe which versions coexist and for how long.
- Explain the rollout as ordered steps rather than saying 'make it backward compatible.' Identify when readers change, when writers change, and what permits removal of the old field.
- Separate structural compatibility from semantic compatibility. Be ready to explain missing versus null, unknown enum values, units, and whether a default is actually correct.
- For a backfill, explain how concurrent writes remain correct: use atomic updates where possible, conditional writes or version checks, and resumable batches.
- State the rollback boundary explicitly. Expect follow-ups about old clients, delayed messages, replay, and what happens if the rollout stops halfway.
Common mistakes
- Assuming that adding a field is always safe, even when old consumers reject unknown fields or new readers require it to be present.
- Renaming or changing a field's type in place while old readers and writers still exist.
- Treating a schema version number or registry check as proof that the business meaning remains compatible.
- Backfilling rows without protecting newer values from stale overwrites.
- Dropping old fields as soon as the new deployment finishes, ignoring rollback, offline clients, queued work, and replay.
Where it comes up
Interview questions where schema evolution is part of a strong answer, and why.
- Design Google AnalyticsCollectors and aggregation jobs must interpret events from independently upgraded SDKs and replay historical data without silently changing metric meaning.
- Design an A/B testing platformExposure and outcome contracts must remain interpretable across SDK versions and replay so schema changes do not corrupt attribution.
- Design a control plane for a distributed databaseRolling upgrades require compatible control-plane messages and persisted workflow state while old and new components coexist.
Questions people ask
What is a backward-compatible schema change?
It is a change that allows a new reader to interpret data written with the old schema. Adding a field can qualify if the new reader handles its absence correctly. Whether old readers can handle new data is a separate question: forward compatibility.
How do you rename a database column without downtime?
Add the replacement column, deploy code that keeps both representations consistent, backfill existing rows, and switch reads after verification. Remove the old column only when no supported code depends on it. The backfill must account for concurrent writes rather than blindly copying a stale snapshot.
How do you handle schema evolution in event-driven systems?
Define reader/writer compatibility rules, validate changes before publication, and test consumers against retained event versions. Use compatible additions when possible; use a new event type or version when meaning changes incompatibly. Historical events generally remain unchanged, so replay needs version-aware readers or explicit translation code.