How it works
Start with the decision that must not conflict: which node owns a database shard, or which command occupies the next position in a replicated log. Replicas must not decide different values for that position, even when they cannot all communicate.
Raft and Paxos are common crash-fault consensus algorithms: they handle nodes stopping or becoming unreachable, not nodes deliberately lying. Raft organizes replication around an elected leader; Multi-Paxos commonly uses a stable leader too. Byzantine consensus addresses a different failure model.
In Raft, the leader appends a command to its log and sends it to followers. For a new entry from its current term, durable replication to a majority allows it to commit that entry and preceding entries. Replicas then apply committed commands in log order to a deterministic state machine.
A five-voter group needs three votes and can tolerate two unavailable voters. Any two majorities overlap, but overlap alone is not enough: election and log rules ensure a replacement leader preserves committed history. An isolated old leader cannot commit new commands without a majority.
During a network partition, only a side with a majority can keep committing updates. Safety means nodes do not decide conflicting values; liveness means they eventually make progress. Raft and Paxos preserve safety during arbitrary message delays, but progress requires enough reachable nodes and sufficiently stable communication.
An agreed write log does not automatically make every read current. Linearizable reads need a safe read protocol, such as confirming the leader's authority with a quorum before reading applied state. External actions also need protection: a consensus-backed ownership record should produce a fencing token that the target resource checks.
A worked example
Safe failover in a five-replica message log
A log partition uses five Raft voters: A, B, C, D, and E. Leader A receives a publish request and durably replicates its current-term entry at index 420 to A, B, and C. It commits the entry and acknowledges the client. A partition then separates A and B from C, D, and E. The three-node side can elect an eligible leader; Raft's election rules prevent a candidate missing the committed entry from winning. A and B cannot commit new publishes. If the original acknowledgment was lost, the client may retry against the new leader, so the service still needs request deduplication. Consensus preserves committed history; it does not make retries unique.
Trade-offs to say out loud
- Consistency versus partition availability
- A group without a reachable majority must stop committing updates rather than accept conflicting histories.
- Write latency
- Committing a write requires quorum communication and durable log persistence, so replica placement directly affects latency.
- Fault tolerance versus replication cost
- Tolerating f unavailable voters generally requires at least 2f + 1 voters, increasing storage, network traffic, and operational work.
- Ordered writes versus throughput
- A leader-based consensus group has finite write capacity, so larger systems typically partition data into multiple independent groups.
- Operational complexity
- Membership changes, snapshot recovery, and election tuning require protocol-aware handling rather than simply adding or removing replicas.
- Limited scope of agreement
- Consensus orders state changes within its group but does not by itself provide cross-shard transactions or exactly-once external side effects.
When to reach for it
- Acknowledged writes must survive replica failover without creating conflicting histories.
- A control plane needs one authoritative record of shard ownership, configuration, or failover decisions.
- A replicated queue or log needs a well-defined committed prefix after its leader crashes.
- A strict invariant must remain true during regional partitions, and rejecting operations without a majority is acceptable.
How to bring it up in the interview
- Name the invariant first: for example, 'A committed log entry must never disappear after failover.' Then explain why asynchronous replication alone does not guarantee it.
- Describe the consensus boundary: one group per shard, one metadata group, or another explicit scope; do not imply one global group must order every operation.
- Walk through a partition with concrete voter counts and state which side can acknowledge writes; expect follow-up questions about an isolated old leader and a lost client response.
- Separate commit, application, and reads: explain when success is returned, how replicas apply commands, and how reads avoid stale leaders.
- Prefer an established consensus-backed store or proven implementation over writing the protocol yourself; expect probes about membership changes, recovery, fencing, and retry deduplication.
Common mistakes
- Calling any majority-based replication scheme consensus without explaining how elections preserve previously committed decisions.
- Claiming consensus guarantees progress during every network partition or that a timeout proves a node has failed.
- Assuming electing a new leader immediately stops the old leader from issuing external writes; downstream resources still need fencing.
- Treating consensus as a replacement for transactions, client-request deduplication, or reconciliation with external systems.
- Using global consensus for data that can safely converge asynchronously, adding coordination latency without a required invariant.
Where it comes up
Interview questions where consensus algorithms is part of a strong answer, and why.
- Design a distributed message log like KafkaConsensus provides a way to define the committed log prefix and preserve acknowledged entries across leader failover.
- Design a distributed key-value storeConsensus-backed replication is a central design option when the store must preserve a single authoritative history through failures.
- Design a distributed message queue like RabbitMQReplicated queue state needs an agreement mechanism to preserve durable enqueues and ownership transitions during failover.
- Design a control plane for a distributed databaseAuthoritative failover and ownership decisions require agreement so competing controllers cannot independently authorize conflicting primaries.
Questions people ask
What is the difference between consensus and replication?
Replication copies data across nodes. Consensus establishes agreed decisions, which a replicated state machine can use to maintain one ordered history despite failures. Asynchronous replication alone can lose acknowledged writes during failover if the replacement replica is behind.
What is the difference between Raft and Paxos?
Both solve crash-fault consensus under their stated assumptions. Raft explicitly structures the protocol around leader election, log replication, and safety rules; basic Paxos decides one value, while Multi-Paxos supports a sequence of decisions. For interviews, explaining quorum safety and failover behavior usually matters more than reciting protocol messages.
Does distributed consensus guarantee strong consistency?
Consensus can underpin linearizable operations, but the surrounding service must implement them correctly. Reads from a stale follower or an isolated former leader can still return old state. The service also needs correct command application, read protocols, and transaction semantics for its advertised guarantees.