# Split brain > Split brain happens when multiple nodes or groups act as the exclusive authority for the same resource, often after losing contact with each other. In system design interviews, it explains why detecting a failed leader and promoting a replacement is not enough: the old leader may still be running and accepting writes. Source: https://sysround.com/concepts/split-brain Updated: 2026-11-05 ## How split brain works Start with one leader and two replicas. The leader becomes isolated from the replicas but remains reachable by some clients. If the replicas promote a replacement while the old leader keeps accepting writes, two nodes now claim authority over the same data. A missed heartbeat means a node is unreachable from the observer, not necessarily dead. Network partitions, long process pauses, and delayed messages can all trigger failover while the original process is still alive. A timeout can start a recovery attempt; it cannot prove that the old leader has stopped. A consensus protocol can require a majority to elect a leader and commit writes. In a fixed three-voter group, two votes are required, so the isolated node cannot commit new writes. Two nodes may temporarily believe they are leader, but a correct protocol prevents both from committing conflicting histories. Majority counting alone is not enough; election and log rules matter too. External side effects need protection as well. A paused worker may resume after its lease expires and continue writing to shared storage. Give each new owner a monotonically increasing fencing token, and have the protected resource atomically reject obsolete tokens. Establish the new fence before allowing the replacement to act; merely attaching a token to requests does nothing unless something enforces it. Choose what happens when authority cannot be established. A consistency-first system rejects or pauses affected writes without a quorum. An availability-first, multi-writer system may accept writes on both sides, but it needs explicit conflict semantics. Intentional multi-writer replication is not itself split brain; the problem is violating the system's promised ownership or consistency rules. When connectivity returns, select the authoritative history before restoring normal traffic. In a consensus-backed log, uncommitted divergent entries can be replaced. If both sides already acknowledged conflicting writes, recovery may require application-level reconciliation; choosing a winner does not undo an oversold seat or an external payment. ## Worked example: Failing over a three-replica message log A log partition has three voting replicas: A, B, and C. A leads in term 12, and a write is acknowledged only after replication to a majority. A network partition isolates A, while B and C can still communicate. Under the consensus protocol's election rules, B becomes leader in term 13 with two votes. A may still receive client requests, but it cannot obtain a majority and must not acknowledge new writes as committed. B can commit with C. When A reconnects, it observes the higher term and steps down; any conflicting uncommitted suffix is replaced. Clients retry uncertain requests using stable request IDs so retries do not create duplicate logical records. ## Trade-offs - **Safety can reduce availability.** Requiring a quorum stops a minority partition from committing writes, even when its nodes are healthy and reachable by clients. - **Safer failover can take longer.** Waiting for lease expiration or confirmed isolation of the old writer increases recovery time but avoids granting overlapping authority. - **Replication adds coordination latency.** Committing through a quorum requires network round trips, which can be costly when voters span distant regions. - **Fencing requires enforcement.** The storage system or side-effect boundary must validate ownership, and some external APIs cannot enforce fencing tokens. - **Available writes create reconciliation work.** Accepting conflicting writes during a partition requires merge rules that may not preserve scarce-inventory or financial invariants. - **Forced recovery can sacrifice data.** Promoting a stale replica or forcing a new quorum can discard acknowledged writes unless durability and recovery procedures explicitly prevent it. ## When to use it in a system design interview - You introduce automatic primary failover for replicated storage, a message log, or a cache shard. - A lease or distributed lock gives one worker exclusive permission to modify a resource. - A multi-region design must preserve a hard limit, such as available inventory or concurrent playback slots. - A control plane promotes database replicas, changes ownership, or cuts writes over to a replacement system. ## How to bring it up in the interview - Name the invariant first: for example, only one authority may commit writes for a partition, or no more than two playback slots may be allocated per account. - Walk through an asymmetric failure: the controller cannot reach the primary, but some clients still can. Interviewers will ask what prevents that primary from continuing to write. - Separate leader selection from enforcement. Explain who grants authority, where epochs or fencing tokens are checked, and what happens to requests from an old owner. - State the partition policy explicitly: which operations stop, which reads may be stale, and whether any writes remain available with weaker guarantees. - Cover recovery and durability. Expect questions about acknowledged writes on the old leader, stale replicas, membership changes, and an operator forcing promotion. ## Common mistakes - Treating a heartbeat timeout as proof that the old leader is dead. - Assuming a lock or expiring lease stops a paused process from resuming work after ownership changes. - Saying 'use a quorum' without specifying the consensus protocol, commit rule, and safe membership-change procedure. - Using DNS or load-balancer updates as fencing, even though existing connections or direct clients can still reach the old writer. - Claiming last-write-wins repairs every conflict, including double bookings, duplicate side effects, or lost financial updates. ## Interview questions where it comes up - [Design a control plane for a distributed database](https://sysround.com/questions/design-database-control-plane): Safe database promotion requires proving that the old primary can no longer act as an authorized writer. - [Design a distributed message log like Kafka](https://sysround.com/questions/design-kafka): Partition leadership and acknowledgment rules must prevent competing leaders from committing conflicting log histories. - [Design a distributed key-value store](https://sysround.com/questions/design-key-value-store): Partition handling requires an explicit choice between exclusive write authority and available writes with conflict resolution. - [Design a distributed cache](https://sysround.com/questions/design-distributed-cache): Replica failover and ownership changes must define what happens when the old shard owner remains reachable. - [Design Netflix's concurrent screen limit](https://sysround.com/questions/design-netflix-screen-limit): Regional failover must not let independent authorities allocate the same account's playback capacity twice. - [Design a large data migration to the cloud](https://sysround.com/questions/design-cloud-data-migration): A safe cutover must fence the source writer before the destination becomes authoritative. ## FAQ ### What is the difference between split brain and a network partition? A network partition prevents some nodes from communicating. Split brain is a failure of authority: multiple nodes or groups act as the exclusive owner of the same resource. A partition can trigger split brain, but a correctly designed system can instead stop unsafe operations. ### Does a quorum prevent split brain? A correctly implemented consensus protocol uses intersecting quorums, election rules, and commit rules to prevent conflicting committed histories. It does not guarantee that an isolated node immediately stops believing it is leader. External writes and side effects still need fencing or another mechanism that rejects obsolete authority. ### How do you recover from split brain? First stop competing writers and establish one authoritative owner. Then compare histories, preserve committed data according to the replication protocol, and reconcile any conflicting acknowledged operations. Do not blindly merge logs or restart both sides: some business effects need compensation or manual review. ## Related concepts - [Leader election](https://sysround.com/concepts/leader-election) - [Quorums](https://sysround.com/concepts/quorums) - [Distributed locks and fencing tokens](https://sysround.com/concepts/distributed-locks) - [Consensus algorithms](https://sysround.com/concepts/consensus) --- SysRound (https://sysround.com) is an AI practice tool for system design interviews: a voice or typed AI interviewer, a shared whiteboard, then feedback on every skill and a model answer. Its feedback is AI-generated and can be wrong.