How it works
Prepare a replacement before failure. Stateless servers may already share traffic, while a database standby must replicate the primary's data before it can take over. Place replicas across the failure boundaries you need to survive, such as separate availability zones.
Detect a suspected failure using heartbeats, request failures, or health checks. A timeout does not prove that a server is dead: it may be slow or separated by a network partition. Detection thresholds trade faster recovery against unnecessary failovers.
Decide who has authority to promote a replacement. Automatic database failover usually relies on a database election protocol or a high-availability controller backed by quorum-based coordination. Two independent monitors must not each promote a different primary.
Choose an eligible replica and prevent the old primary from continuing authoritative writes. Election terms, enforced leases, fencing tokens, or infrastructure fencing can provide this protection, depending on the system. Updating a routing record alone does not stop an isolated primary from accepting writes.
Reconnect clients to the new owner through a proxy, service discovery, or a database driver that understands topology changes. Existing connections may need to be discarded; DNS updates do not move established connections. Retry ambiguous requests with stable idempotency keys where duplicate effects would matter.
Restore redundancy after service recovers. For example, if primary A fails and replica B takes over, A should rejoin only after its state is reconciled with B, often as a replica. Moving leadership back to A is a separate, controlled operation rather than an automatic requirement.
A worked example
Automatic database failover across three zones
Suppose a consensus-backed database has three replicas, one per zone, and acknowledges writes only after a majority durably records them. The target RTO is 15 seconds, with no acknowledged-write loss for a single-zone failure. Primary A becomes unreachable. After a configured 5-second election timeout, B and C can elect a leader using rules that preserve committed entries; an isolated A cannot commit new writes without a majority. Assume election and recovery take 2 seconds and client reconnection takes another 3: observed recovery is about 10 seconds, not a guarantee. Clients retry uncertain writes with idempotency keys. When A returns, it catches up as a replica.
Trade-offs to say out loud
- Recovery speed versus false positives
- Short detection timeouts reduce outage time but can turn temporary pauses or network problems into disruptive leadership changes.
- Write latency versus data-loss exposure
- Waiting for durable replication increases write latency but can protect acknowledged writes during supported failures, whereas asynchronous replication can leave the promoted replica missing recent writes.
- Consistency versus availability during partitions
- A single-writer system may need to reject writes without a quorum or safe fencing rather than risk two writable primaries.
- Automation versus operational risk
- Automatic failover reduces dependence on human response but needs conservative promotion rules, reliable coordination, and tested recovery paths.
- Standby capacity and failure independence
- Replicas cost resources, must handle the promoted workload, and offer limited protection if they share the same failure domain.
- Restored availability versus full recovery
- Serving traffic from a new primary does not mean redundancy, replication lag, connection pools, and background workloads have all recovered.
When to reach for it
- The availability target requires recovering from a server or availability-zone failure faster than manual repair allows.
- A replicated database, message partition, or document owner has one authoritative writer whose failure would otherwise stop progress.
- Acknowledged writes must survive primary failure, so the design needs explicit replication, promotion, and fencing rules.
- A regional outage requires transferring ownership elsewhere without allowing both regions to act as the sole authority.
- The interviewer asks what happens to in-flight requests, connections, or committed data when a node disappears.
How to bring it up in the interview
- Start with the failure scope and recovery objectives: recovery time objective (RTO) is the target time to restore service, and recovery point objective (RPO) is the acceptable data-loss window.
- Walk through detection, authority transfer, replica eligibility, fencing, client reconnection, and restoring redundancy; saying 'promote a replica' skips most of the design.
- State exactly what a successful write acknowledgment guarantees and whether every eligible replacement must contain that write; expect probes about replication lag and crashes immediately after acknowledgment.
- Explain the network-partition case separately from a clean crash: identify which side can continue writing and how the other side is prevented from doing so.
- Describe how you would test failover under load, including stale connections, ambiguous retries, insufficient standby capacity, and failure of the failover controller itself.
Common mistakes
- Treating replication as automatic failover: a replica provides another copy, but the system still needs safe promotion and client routing.
- Assuming a missed heartbeat proves the old primary is dead, then promoting a replacement without preventing split-brain.
- Promising zero data loss with asynchronous replication, or assuming synchronous replication alone makes every promotion safe.
- Claiming that a DNS or load-balancer update instantly repairs existing connections and in-flight requests.
- Retrying every failed write as a new operation, even though the old primary may have committed it before the response was lost.
Where it comes up
Interview questions where failover is part of a strong answer, and why.
- Design a control plane for a distributed databaseSafe automatic promotion requires durable failover workflows, authoritative coordination, and fencing against competing controllers or primaries.
- Design a distributed message log like KafkaPartition leader failover determines whether acknowledged records survive and whether ordering remains valid.
- Design a distributed cacheReplica promotion must coordinate ownership and routing while handling stale cache state and increased misses.
- Design a distributed key-value storeFailure recovery requires explicit rules for replica ownership, write availability, and reconciling divergent data.
- Design Google DocsDocument owner failover must preserve acknowledged edits and let reconnecting clients resume without losing or duplicating updates.
- Design Netflix's concurrent screen limitRegional failover must transfer admission authority without allowing old and new regions to allocate conflicting playback slots.
- Design a flash giveaway: 6 million burgers in an hourFailover must preserve acknowledged claims so that lost allocation state does not cause stock to be awarded twice.
Questions people ask
What is automatic database failover?
Automatic database failover detects an unavailable primary, safely promotes an eligible replica, and enables clients to connect to it without an operator initiating each step. It requires more than health checks: the system must control write authority and define which replicated state is safe to preserve.
Can database failover cause data loss?
Yes. With asynchronous replication, a primary can acknowledge writes that have not reached the replacement when it fails. Durable synchronous replication and safe election rules can protect acknowledged writes within a specified failure model, but do not protect against every correlated failure or logical deletion.
What is the difference between failover, switchover, and failback?
Failover transfers responsibility because the current component is unavailable or suspected to be unhealthy. A switchover is a planned transfer, usually with time to drain work and catch up replication. Failback moves responsibility back to the original location after recovery and should happen only when it is safe and useful.