# Distributed systems interview questions > Distributed systems interview questions test how you preserve correctness when machines communicate over unreliable networks. Prepare to explain consistency, replication, partitioning, message delivery, and recovery using concrete failure scenarios, not just definitions. Strong answers state an invariant, trace the normal path, and explain what happens when an operation times out or a node fails. Source: https://sysround.com/guides/distributed-systems-interview-questions Updated: 2026-11-05 ## How to structure a distributed systems answer Start with the behavior the system must guarantee. “Never allocate the same seat twice” is more useful than “use strong consistency.” Then identify which component owns that decision and what durable state proves it happened. For each design choice, explain the normal path, a failure between two steps, and recovery. Distinguish a rejected operation from an operation whose outcome is unknown to the caller. - Define the invariant: no duplicate charge, no lost acknowledged message, or no stale permission grant. - Set the scope: one key, one partition, multiple records, or multiple regions. - Name the required guarantee: ordering, durability, atomicity, freshness, or availability. - Trace a failure: the server commits, but its response never reaches the client. - Explain recovery: retry identity, durable status lookup, replay, or reconciliation. - State the cost: additional coordination, latency, storage, or reduced availability. ## Consistency and CAP interview questions Choose consistency per operation rather than labeling the entire application. A product description may tolerate stale reads, while allocating the last unit of inventory needs an atomic decision. - What does CAP actually say? During a network partition, a system cannot guarantee both linearizability and availability for every request. CAP availability is not the same as an uptime percentage, and partition tolerance is not a feature you can simply disable on an unreliable network. - How does linearizability differ from serializability? Linearizability makes operations appear atomic and respects real-time order. Serializability makes concurrent transactions equivalent to some serial execution, without necessarily respecting real-time order. Strict serializability combines both properties. - When is eventual consistency acceptable? When temporary divergence does not violate the product's correctness rules and there is a defined convergence mechanism. Explain conflict resolution and how the user experiences stale data. - How do you provide read-your-writes? Route reads to an authoritative replica, carry a committed version and wait for a replica to catch up, or use a session-aware routing scheme. Follow-up: what happens when that replica or region is unavailable? ## Replication, quorums, and consensus questions Separate replication from agreement. Copying data to replicas does not by itself establish which writes are committed or which node may lead after a failure. - When should a replicated write be acknowledged? Acknowledging before replication reduces latency but can lose acknowledged writes during failover. Waiting for durable replication improves durability at the cost of latency and availability when replicas are unreachable. - Does R + W > N guarantee strong consistency? It provides read/write quorum overlap for a fixed replica set. Linearizable behavior still depends on the protocol, version handling, concurrent writes, membership changes, and which versions reads may return. - What problem does consensus solve? It lets participating nodes agree on decisions, commonly an ordered log, under a stated fault model. A majority-based protocol can stop making progress when it cannot reach a majority rather than allow conflicting decisions. - How do you prevent split-brain writes? Use an authority mechanism that prevents competing leaders from committing conflicting operations. For side effects outside the replicated system, use fencing tokens that the destination validates. - How does a replica recover? Restore a snapshot, replay the remaining log, and verify it has caught up before serving operations that require current state. Follow-up: what if the required log entries have already been deleted? ## Partitioning, rebalancing, and hot-key questions Partitioning increases capacity only when the workload can spread across partitions. Explain the access pattern before choosing a partition key, and distinguish a hot partition from a single hot key. - How do you choose a partition key? Favor keys that distribute load while keeping important operations local. Hash partitioning spreads keys; range partitioning supports ordered scans but can concentrate writes at an active range boundary. - What does consistent hashing solve? It limits ownership changes when nodes join or leave. It does not automatically solve hot keys, replication safety, or uneven object sizes. - How do you rebalance without losing writes? Copy a snapshot, capture concurrent changes, catch up, and switch ownership through versioned routing or another coordinated handoff. Fence the old owner so stale clients cannot keep writing there. - How do you handle a hot key? Replicate or cache reads, coalesce requests, and limit demand. Splitting writes requires a data model that supports merging partial results; an exact account balance cannot be sharded like an approximate view counter. - What changes when a query crosses partitions? Fan-out increases work and exposes the request to slow or failed partitions. Discuss bounded concurrency, partial-result policy, and whether an index or different data model avoids the scatter-gather query. ## Message delivery, ordering, and exactly-once questions Describe delivery guarantees separately from business effects. A broker may deliver a message repeatedly while the application ensures that one payment or notification intent is recorded. - Why does at-least-once delivery create duplicates? A worker can finish its work and crash before acknowledging the message. The broker cannot distinguish that case from an unfinished attempt, so it redelivers. - How do you make a consumer idempotent? Give each logical operation a stable identifier and atomically record that identifier with the local state change. A separate deduplication check followed by a write leaves a race. - Can you guarantee exactly-once processing? State the boundary. A transaction may atomically commit an output and a consumer offset within a supported system, but an external email or payment call needs its own idempotency or reconciliation mechanism. - How do you preserve ordering? Route related events to the same ordered partition and control consumer concurrency. Follow-up: how do retries, poison messages, and partition-count changes affect that ordering? - What happens when consumers fall behind? Measure lag and processing capacity, apply backpressure, and scale where ordering allows. A dead-letter queue isolates problematic messages but needs an inspection and safe replay process. ## Distributed transaction and dual-write questions First ask whether the invariant can fit inside one database transaction. Crossing a service boundary adds failure states, so explain why that boundary is necessary before introducing a distributed protocol. - Why is writing to a database and then publishing an event unsafe? The database commit can succeed while publication fails. Reversing the order can publish an event for a database change that never commits. - How does the outbox pattern help? Commit the business change and an outbox record in one local transaction, then publish the record asynchronously. Publication can repeat, so consumers still need duplicate handling. - When would you use two-phase commit? When participants support it and atomic commitment across them is required. Explain prepared state, retained locks or resources, and how failures can leave transactions waiting for a decision. - When would you use a saga? When a workflow can use local transactions with explicit compensating actions. Compensation is a new business action, not a guarantee that every external effect can be perfectly undone. - What if a payment provider times out? Treat the result as unknown, not failed. Retry with the same provider-supported idempotency key, query status, and reconcile before attempting another independent charge. ## Clocks, leases, and concurrency questions A timestamp is not automatically a reliable ordering mechanism. Machine clocks can differ or move, and a process can pause long enough to lose a lease while still believing it owns a resource. - Can wall-clock timestamps order distributed events? Not reliably by themselves. Clock skew can reverse apparent order; last-write-wins based on timestamps can discard a logically later update. - What do Lamport clocks tell you? If one event causally precedes another, its Lamport timestamp is smaller. The reverse implication does not hold, so Lamport clocks alone cannot identify concurrent events. - How do vector clocks help? They can distinguish causal order from concurrency for tracked participants. Discuss metadata growth and how the application resolves concurrent versions. - Is a distributed lock enough to protect a resource? Not if an expired holder can still modify it. Issue monotonically increasing fencing tokens and require the protected resource to reject stale tokens. - How do you prevent two workers from completing the same job? Use an atomic claim or lease to coordinate work, but also protect the final effect with idempotency or a conditional write. Lease expiration alone does not guarantee single execution. ## Timeouts, retries, and regional failure questions A timeout is evidence that a response did not arrive in time, not proof that the remote operation failed. Recovery design must handle both incomplete operations and completed operations with lost responses. - How should retries work? Retry only appropriate failures, keep the logical operation identity stable, and use bounded exponential backoff with jitter. Propagate deadlines and enforce a retry budget. - Why can retries cause an outage? Retried requests add load to a system that may already be saturated. Retries at multiple layers multiply attempts, so choose a deliberate retry owner and shed excess work. - How do circuit breakers and bulkheads differ? A circuit breaker limits calls to a failing dependency. Bulkheads isolate resources so one dependency, tenant, or workload cannot consume all shared capacity. - How do you detect a failed node? Heartbeats and timeouts produce suspicion, not certainty. Safe ownership changes need coordination because a slow or partitioned node may still be running. - What should a regional failover plan specify? Define recovery point and recovery time objectives, replication lag, promotion authority, traffic routing, and how the old writer is fenced. Test failback and data reconciliation as well as failover. ## Practice questions that combine these topics Use full designs to connect the concepts. For each exercise, draw the state transitions and interrupt the request after every durable write, network call, and acknowledgment. A useful answer can trace one operation through success, an ambiguous timeout, a retry, and recovery. Add components only when you can explain which requirement or failure they address. - Design a distributed key-value store: choose partition ownership, write acknowledgment rules, consistency guarantees, and convergence after a partition. - Design a distributed message log: explain per-partition ordering, replication, safe leader replacement, consumer offsets, and retention. - Design a distributed queue: handle visibility leases, worker crashes, duplicate delivery, poison messages, and hot queues. - Design a payment system: preserve ledger invariants, deduplicate requests, handle unknown provider outcomes, and reconcile discrepancies. - Design a distributed cache: cover ownership changes, replica lag, eviction, expiration, and hot-key protection. - Design a distributed job scheduler: separate scheduled occurrences from execution attempts and prevent retries from repeating business effects. ## Questions to practice - [Design a distributed key-value store](https://sysround.com/questions/design-key-value-store): Partition ownership, replication, consistency choices, and convergence make this a direct exercise in distributed systems fundamentals. - [Design a distributed message log like Kafka](https://sysround.com/questions/design-kafka): Partition ordering and durable replication require explaining commit rules, leader replacement, and recovery. - [Design a distributed message queue like RabbitMQ](https://sysround.com/questions/design-message-queue): Visibility leases and worker failures expose the difference between message delivery and retry-safe business effects. - [Design a payment system](https://sysround.com/questions/design-payment-system): External timeouts and concurrent money movement require idempotency, transactional invariants, and reconciliation. - [Design a distributed cache](https://sysround.com/questions/design-distributed-cache): Ownership changes, stale replicas, and hot keys test partitioning and failure-handling trade-offs. - [Design a distributed job scheduler](https://sysround.com/questions/design-job-scheduler): Durable scheduling and worker leases require distinguishing logical job occurrences from potentially duplicated execution attempts. ## FAQ ### How do I prepare for distributed systems interview questions? Practice explaining consistency, replication, partitioning, messaging, and failure recovery through concrete designs. For every operation, identify its invariant, durable commit point, and retry behavior. Then rehearse failures such as a lost response, an expired lease, or an unavailable region. ### Do I need to know Raft and Paxos for a system design interview? Be able to explain why consensus is needed, how majority-based agreement limits availability, and why leader election alone does not guarantee safe writes. For storage or infrastructure roles, prepare for deeper questions about terms, log commitment, recovery, and membership changes. Avoid claiming implementation details you cannot defend. ### What is the difference between distributed systems and system design interview questions? System design questions typically cover an entire service, including APIs, data models, capacity, and operations. Distributed systems questions focus more closely on coordination, consistency, concurrency, and partial failures. The two overlap whenever a design uses multiple nodes or crosses a transactional boundary. ## Related concepts - [CAP theorem](https://sysround.com/concepts/cap-theorem) - [Database replication](https://sysround.com/concepts/replication) - [Consensus algorithms](https://sysround.com/concepts/consensus) - [Idempotency and idempotency keys](https://sysround.com/concepts/idempotency) - [Eventual consistency](https://sysround.com/concepts/eventual-consistency) --- SysRound (https://sysround.com) is an AI practice tool for system design interviews: a voice or typed AI interviewer, a shared whiteboard, then feedback on every skill and a model answer. Its feedback is AI-generated and can be wrong.