How it works
Start with one leader and two replicas. The leader becomes isolated from the replicas but remains reachable by some clients. If the replicas promote a replacement while the old leader keeps accepting writes, two nodes now claim authority over the same data.
A missed heartbeat means a node is unreachable from the observer, not necessarily dead. Network partitions, long process pauses, and delayed messages can all trigger failover while the original process is still alive. A timeout can start a recovery attempt; it cannot prove that the old leader has stopped.
A consensus protocol can require a majority to elect a leader and commit writes. In a fixed three-voter group, two votes are required, so the isolated node cannot commit new writes. Two nodes may temporarily believe they are leader, but a correct protocol prevents both from committing conflicting histories. Majority counting alone is not enough; election and log rules matter too.
External side effects need protection as well. A paused worker may resume after its lease expires and continue writing to shared storage. Give each new owner a monotonically increasing fencing token, and have the protected resource atomically reject obsolete tokens. Establish the new fence before allowing the replacement to act; merely attaching a token to requests does nothing unless something enforces it.
Choose what happens when authority cannot be established. A consistency-first system rejects or pauses affected writes without a quorum. An availability-first, multi-writer system may accept writes on both sides, but it needs explicit conflict semantics. Intentional multi-writer replication is not itself split brain; the problem is violating the system's promised ownership or consistency rules.
When connectivity returns, select the authoritative history before restoring normal traffic. In a consensus-backed log, uncommitted divergent entries can be replaced. If both sides already acknowledged conflicting writes, recovery may require application-level reconciliation; choosing a winner does not undo an oversold seat or an external payment.
A worked example
Failing over a three-replica message log
A log partition has three voting replicas: A, B, and C. A leads in term 12, and a write is acknowledged only after replication to a majority. A network partition isolates A, while B and C can still communicate. Under the consensus protocol's election rules, B becomes leader in term 13 with two votes. A may still receive client requests, but it cannot obtain a majority and must not acknowledge new writes as committed. B can commit with C. When A reconnects, it observes the higher term and steps down; any conflicting uncommitted suffix is replaced. Clients retry uncertain requests using stable request IDs so retries do not create duplicate logical records.
Trade-offs to say out loud
- Safety can reduce availability
- Requiring a quorum stops a minority partition from committing writes, even when its nodes are healthy and reachable by clients.
- Safer failover can take longer
- Waiting for lease expiration or confirmed isolation of the old writer increases recovery time but avoids granting overlapping authority.
- Replication adds coordination latency
- Committing through a quorum requires network round trips, which can be costly when voters span distant regions.
- Fencing requires enforcement
- The storage system or side-effect boundary must validate ownership, and some external APIs cannot enforce fencing tokens.
- Available writes create reconciliation work
- Accepting conflicting writes during a partition requires merge rules that may not preserve scarce-inventory or financial invariants.
- Forced recovery can sacrifice data
- Promoting a stale replica or forcing a new quorum can discard acknowledged writes unless durability and recovery procedures explicitly prevent it.
When to reach for it
- You introduce automatic primary failover for replicated storage, a message log, or a cache shard.
- A lease or distributed lock gives one worker exclusive permission to modify a resource.
- A multi-region design must preserve a hard limit, such as available inventory or concurrent playback slots.
- A control plane promotes database replicas, changes ownership, or cuts writes over to a replacement system.
How to bring it up in the interview
- Name the invariant first: for example, only one authority may commit writes for a partition, or no more than two playback slots may be allocated per account.
- Walk through an asymmetric failure: the controller cannot reach the primary, but some clients still can. Interviewers will ask what prevents that primary from continuing to write.
- Separate leader selection from enforcement. Explain who grants authority, where epochs or fencing tokens are checked, and what happens to requests from an old owner.
- State the partition policy explicitly: which operations stop, which reads may be stale, and whether any writes remain available with weaker guarantees.
- Cover recovery and durability. Expect questions about acknowledged writes on the old leader, stale replicas, membership changes, and an operator forcing promotion.
Common mistakes
- Treating a heartbeat timeout as proof that the old leader is dead.
- Assuming a lock or expiring lease stops a paused process from resuming work after ownership changes.
- Saying 'use a quorum' without specifying the consensus protocol, commit rule, and safe membership-change procedure.
- Using DNS or load-balancer updates as fencing, even though existing connections or direct clients can still reach the old writer.
- Claiming last-write-wins repairs every conflict, including double bookings, duplicate side effects, or lost financial updates.
Where it comes up
Interview questions where split brain is part of a strong answer, and why.
- Design a control plane for a distributed databaseSafe database promotion requires proving that the old primary can no longer act as an authorized writer.
- Design a distributed message log like KafkaPartition leadership and acknowledgment rules must prevent competing leaders from committing conflicting log histories.
- Design a distributed key-value storePartition handling requires an explicit choice between exclusive write authority and available writes with conflict resolution.
- Design a distributed cacheReplica failover and ownership changes must define what happens when the old shard owner remains reachable.
- Design Netflix's concurrent screen limitRegional failover must not let independent authorities allocate the same account's playback capacity twice.
- Design a large data migration to the cloudA safe cutover must fence the source writer before the destination becomes authoritative.
Questions people ask
What is the difference between split brain and a network partition?
A network partition prevents some nodes from communicating. Split brain is a failure of authority: multiple nodes or groups act as the exclusive owner of the same resource. A partition can trigger split brain, but a correctly designed system can instead stop unsafe operations.
Does a quorum prevent split brain?
A correctly implemented consensus protocol uses intersecting quorums, election rules, and commit rules to prevent conflicting committed histories. It does not guarantee that an isolated node immediately stops believing it is leader. External writes and side effects still need fencing or another mechanism that rejects obsolete authority.
How do you recover from split brain?
First stop competing writers and establish one authoritative owner. Then compare histories, preserve committed data according to the replication protocol, and reconcile any conflicting acknowledged operations. Do not blindly merge logs or restart both sides: some business effects need compensation or manual review.