How it works
In leader-follower replication, one node accepts writes and sends an ordered stream of changes to followers. Followers apply those changes to maintain copies of the data. For example, a profile update goes to the leader, while profile reads may go to either the leader or a follower.
A new follower usually starts with a consistent snapshot, then consumes changes from the corresponding log position. Replication may ship physical storage changes or logical changes such as inserted and updated rows. The source must retain enough history for the follower to catch up; otherwise, it needs a new snapshot.
With asynchronous replication, the leader acknowledges a write without waiting for followers. With synchronous replication, acknowledgment waits for a configured set of replicas to reach a specified stage, such as durable log storage. Receiving, durably storing, and applying a change are different milestones; a durable change may not yet be visible to follower reads.
Replication lag is the gap between the leader's progress and a follower's progress. If a user updates a profile and immediately reads from a lagging follower, they may see the old value. For read-your-writes behavior, route that user's reads to the leader or require a follower to apply at least the write's log position before serving the read.
If the leader fails, a failover mechanism chooses an eligible follower and promotes it. Safe failover must preserve the writes promised by the acknowledgment policy and prevent the old leader from accepting writes, typically through terms, leases, or fencing. Promoting a follower that lacks an asynchronously acknowledged write can lose that write.
Leader-follower is not the only model: multi-leader systems accept writes at several leaders, while leaderless systems coordinate reads and writes across replicas. These models require explicit conflict and consistency rules. Replication copies data; sharding divides data into partitions, each of which may have its own replicas.
A worked example
Replicated seat inventory with safe acknowledgments
Suppose a booking service uses three database nodes, one in each of three availability zones. The leader handles reservation transactions and acknowledges success only after its log and one follower's log are durably stored. A follower is currently 200 milliseconds behind in applying changes, so availability pages may show stale seats; the actual reservation still runs an atomic transaction on the leader. For failover, use a protocol whose election rules preserve committed writes and fence the old leader, rather than promoting any reachable copy. If the system cannot safely elect a leader or meet its acknowledgment policy, reservations pause. This trades some availability for protecting acknowledged bookings.
Trade-offs to say out loud
- Read capacity versus freshness
- Followers can offload reads, but asynchronous follower reads may be stale and need explicit routing rules for freshness-sensitive requests.
- Write latency versus durability
- Waiting for remote replicas adds network and storage latency but can protect acknowledged writes against the failures covered by the replication policy.
- Write availability versus safety
- When required replicas are unreachable, preserving the synchronous durability promise may require rejecting writes rather than silently weakening it.
- More copies do not mean more write capacity
- Each follower must process the replicated changes, so adding followers does not remove a single leader's write bottleneck.
- Geographic resilience versus coordination cost
- Replicas in separate regions reduce shared failure exposure, but synchronous cross-region writes add latency while asynchronous replication leaves a potential data-loss window.
- Redundancy versus operational cost
- Replicas consume storage, bandwidth, and recovery capacity, and they do not replace backups because accidental deletes and corruption can propagate.
When to reach for it
- The database must keep serving after a node or availability-zone failure.
- Reads dominate the workload, and a meaningful share can tolerate stale data.
- The design needs regional read locality or a defined disaster-recovery strategy.
- A requirement such as “never lose an acknowledged reservation” forces you to define which replicas must persist a write before success.
- Maintenance, upgrades, or migrations require a second copy that can catch up and take over safely.
How to bring it up in the interview
- Start with the required guarantees: acceptable read staleness, recovery time objective, and recovery point objective, the acceptable amount of data loss. Choose replication and placement from those requirements rather than simply drawing three database boxes.
- State the acknowledgment contract precisely: for example, success means the leader and one follower in another zone have durably stored the log record. Expect the interviewer to ask whether that follower has also applied the record and whether it can serve a fresh read.
- Separate read routing by operation. Public profile reads might use followers, while inventory decisions and authorization checks need a sufficiently current, correctly coordinated path; replication alone does not make check-then-write operations atomic.
- Walk through a partition, not just a crash. Explain who can promote a leader, how the old leader is fenced, and what happens when no safe candidate is reachable.
- Describe lag monitoring and overload behavior. Track replay progress and lag over time, remove unsuitable replicas from freshness-sensitive read pools, and account for catch-up traffic competing with normal queries.
Common mistakes
- Claiming that synchronous replication automatically provides linearizable reads; read routing, leader authority, and failover rules still matter.
- Treating observed replication lag as a hard bound or using a fixed sleep to guarantee read-your-writes.
- Promoting any healthy follower without checking whether it contains the required committed history or fencing the previous leader.
- Using replicas to solve a write-throughput bottleneck that actually requires partitioning, batching, or a different data model.
- Treating replicas as backups or assuming that three copies in one failure domain protect against a regional outage.
Where it comes up
Interview questions where database replication is part of a strong answer, and why.
- Design a distributed key-value storeReplication policy determines durable-write guarantees, behavior during partitions, and how replicas converge after failures.
- Design a distributed message log like KafkaReplicated partition logs require precise acknowledgment and leader-election rules to preserve accepted records through failover.
- Design a distributed message queue like RabbitMQReplicating messages and queue state is central to preserving accepted work and recovering safely after broker failures.
- Design a control plane for a distributed databaseSafe database failover and rolling maintenance require replication-aware promotion, catch-up checks, and fencing.
- Design a flash giveaway: 6 million burgers in an hourProtecting successful claims through database failover requires an explicit replication acknowledgment policy and safe leader promotion.
- Design a large data migration to the cloudSnapshot transfer followed by change replication keeps the destination current until a verified, fenced cutover.
Questions people ask
What is the difference between database replication and sharding?
Replication keeps copies of the same data on multiple nodes. Sharding splits a dataset across nodes to distribute storage and work. A common design uses multiple shards, each with a leader and followers.
What causes replication lag, and how do you handle it?
Lag can come from network delays, write bursts, slow disks, expensive replay, or resource contention on followers. Monitor both log progress and apply progress, then address the bottleneck or reduce competing work. For correctness-sensitive reads, use an authoritative read path or wait for a required replication position rather than assuming the follower is current.
Can leader-follower replication lose data?
Yes: with asynchronous replication, the leader can acknowledge a write and fail before a follower receives it. Synchronous replication can protect against specified failures, but only when durable acknowledgment and failover rules preserve the promised writes. Neither approach replaces backups or protects against every correlated failure.