# etcd for system design interviews > etcd is a distributed key-value store for small, critical coordination data such as configuration, service registrations, and leader ownership. It provides strongly consistent operations and change watches, making it useful for a system's control plane rather than its high-volume application data. Source: https://sysround.com/tech/etcd Updated: 2026-11-05 ## How it works An etcd cluster runs one Raft consensus group. A leader orders writes into a replicated log, and a write must reach a majority of voting members before it can be committed. Every member maintains the same logical keyspace; adding members does not partition the data. Members persist Raft state in a write-ahead log and apply committed operations to an MVCC key-value store backed by bbolt, an embedded B+ tree database. Changes receive increasing revisions, and a transaction that modifies multiple keys gives those modifications one shared revision. Revisions support historical reads, optimistic concurrency, and watch recovery. Range reads are linearizable by default. The serving member coordinates with Raft to establish a safe read point and waits until its local state has caught up. The optional serializable read mode serves local state without that coordination, so it can return stale data despite its name. Transactions atomically evaluate comparisons, such as a key's value or modification revision, and execute one of two operation lists. Leases attach expiration to keys, while watches stream changes from a requested revision. Together, these primitives support registrations, elections, and locks, but applications still need to handle expired ownership and uncertain request outcomes. ## What it gives you in a design - **Linearizable reads and atomic transactions.** Use comparisons and conditional writes to update coordination state without a read-then-write race. Transaction boundaries cover etcd keys, not external databases, queues, or services. - **Revision-based watches.** Clients can read a snapshot and then watch from the next revision to maintain a local view without constant polling. If the requested history has been compacted, the client must fetch a new snapshot and restart its watch. - **Leases and expiring keys.** A client can attach several keys to a lease and renew it with keepalives. Expiration removes the attached keys, which is useful for temporary registrations and ownership, but it does not stop a paused process from continuing external work. - **Lock and election primitives.** The v3 API includes lock and election services built on coordination state and leases. Protecting external resources still requires fencing or another check that rejects work from an obsolete owner. - **MVCC and optimistic concurrency.** A client can compare a key's modification revision before updating it, detecting changes since its last read. Retained revisions also support consistent historical reads until compaction removes that history. - **Transport and access security.** etcd supports TLS for client and peer traffic, along with authentication and role-based key-range permissions. Treat it as privileged infrastructure because changing coordination state can redirect traffic or alter ownership. ## Scaling, replication and failure etcd scales coordination better than bulk storage. All writes pass through one Raft leader, and every member stores the full keyspace. Use small values, keep write volume bounded, and distribute cached configuration through watches rather than making every application request read etcd. Three or five voting members are common deployments: three tolerate one unavailable voter, and five tolerate two. More voters increase failure tolerance but also add replication work. Place members across failure domains while keeping quorum network latency and durable disk latency within the control plane's requirements. During a network partition, only a component with a voting majority can commit writes. A minority cannot provide current linearizable reads, though explicitly requested local serializable reads may remain available and stale. Spanning distant regions increases coordination latency, so many designs use regional clusters and a separate mechanism for cross-region policy distribution. Leader failure causes an election and a temporary interruption. A client timeout does not prove a write failed, so recovery should inspect state or use a retry-safe conditional operation. Maintain tested snapshots and a restore procedure; replication does not protect against accidental deletion or a faulty administrative write. ## Limits and defaults worth knowing - **Default maximum request size:** The default max-request-bytes setting is 1.5 MiB. It is configurable, but etcd is designed for small metadata rather than large values. - **Default backend quota:** The default backend storage quota is 2 GiB and is configurable. Exhausting it raises a NOSPACE alarm and restricts writes until space is reclaimed and the alarm is resolved. - **Default transaction operation limit:** The max-txn-ops setting defaults to 128 and is configurable. Large batches should account for both this limit and the request-size limit. - **Quorum requirement:** A cluster needs floor(N/2) + 1 voting members to make progress. Three voters tolerate one unavailable voter; five tolerate two. - **Watch and historical-read retention:** History is available only until compaction removes it. Automatic compaction is configurable and is disabled by default; a watch starting at a compacted revision must resynchronize. ## Use it when - You need a small, authoritative store for cluster membership, configuration, shard assignments, or controller state. - Several controllers need to elect an owner and make atomic, conditional changes to coordination metadata. - Services need to cache configuration locally and receive revision-based updates. - The design can stop ownership changes when quorum is unavailable rather than risk conflicting authoritative decisions. ## Avoid it when - You need a primary store for large objects, user content, telemetry, or high-volume application events. - You need horizontal write scaling by adding storage nodes to one cluster. - Every region must independently accept writes to the same coordination state during a partition. - You need a durable event stream with independent consumer retention, acknowledgments, and replay beyond metadata compaction. ## What interviewers ask about it ### What happens when a lease holder pauses, loses its lease, and then resumes? Another process may already own the resource, while the old process still believes it is the owner. Give each owner a monotonically increasing fencing token, often derived from the ownership key's creation revision, and require the protected resource to reject older tokens. ### How do you initialize a local cache without missing changes? Read the relevant key range at a consistent revision R, then watch from R+1. For a paginated snapshot, keep every page pinned to R; if R is compacted before recovery completes, discard the partial view and resynchronize. ### Does adding members increase write capacity? Not in the way adding shards would: one Raft group still orders all writes, and every member stores the data. Add voters for failure tolerance, not as a write-throughput strategy; separate independent coordination domains if isolation is needed. ### What consistency does a watch provide? A watch delivers ordered changes from retained revision history, but receiving events is not a linearizable read or proof that the client is currently caught up. Use watches for maintaining local views and conditional transactions for decisions that require current authoritative state. ### How should a client handle a timed-out transaction? The transaction may have committed even though the response was lost. Record an operation identifier in the same transaction, or use a comparison whose outcome lets the client distinguish a completed operation from one it should retry. ### How do compaction and defragmentation differ? Compaction removes access to old MVCC history and makes its storage reusable internally. Defragmentation reclaims unused backend space for the filesystem; plan it member by member because it can block operations on the affected member. ## Common mistakes - Putting etcd on the hot path for every request instead of using it to distribute small amounts of control-plane state. - Assuming a lease-backed lock prevents a stale owner from modifying an external resource without fencing. - Treating watches as an indefinitely retained message log or assuming reconnects never require a full resynchronization. - Assuming a timeout means a write failed, or that adding replicas shards storage and increases write capacity. ## Alternatives - **Apache ZooKeeper:** It fits an existing coordination environment built around znodes, sessions, and ZooKeeper recipes. Its data model and watch semantics differ from etcd's revision-based MVCC model. - **Consul:** It is a stronger fit when service discovery, health checks, and service networking are the main requirements, with coordination KV as one part of the platform. - **PostgreSQL:** It often wins when ownership or scheduling state belongs beside relational business data and must participate in the same transaction. Conditional updates and constraints may remove the need for a separate coordination store. - **Redis:** It fits caching, counters, and ephemeral state where its configured persistence and failover guarantees are acceptable. Do not treat a Redis lock as a drop-in replacement for consensus-backed ownership with fencing. ## Interview questions where it comes up - [Design a control plane for a distributed database](https://sysround.com/questions/design-database-control-plane): etcd can hold desired state, controller ownership, and failover metadata, while fencing prevents an obsolete controller from changing a database. - [Design a distributed cache](https://sysround.com/questions/design-distributed-cache): etcd can coordinate versioned shard assignments and membership without storing cached application values. - [Design a distributed job scheduler](https://sysround.com/questions/design-job-scheduler): etcd can coordinate scheduler leadership and bounded ownership metadata, while durable job state and retry-safe execution remain separate concerns. - [Design a distributed message log like Kafka](https://sysround.com/questions/design-kafka): etcd can store broker registrations and partition assignments, while the replicated message data remains in dedicated log storage. ## FAQ ### What is etcd used for? etcd stores small, important pieces of distributed-system state, such as configuration, registrations, and controller ownership. Kubernetes uses it as the backing store for cluster data. In an interview, position it as coordination infrastructure rather than a general application database. ### Is etcd strongly consistent? Its default key-value reads are linearizable, and its transactions apply atomically. Clients can explicitly request serializable reads that may return stale local data. Watches are asynchronous change streams, so they should not be treated as current linearizable reads. ### Why does etcd usually have three or five nodes? Raft requires a majority of voters, so three members tolerate one failure and five tolerate two. A four-voter cluster still tolerates only one failure, despite the extra member. Choose membership based on failure domains and recovery needs, not a desire to shard data. ## Related concepts - [Raft](https://sysround.com/concepts/raft) - [Linearizability](https://sysround.com/concepts/linearizability) - [Distributed locks and fencing tokens](https://sysround.com/concepts/distributed-locks) - [Service discovery](https://sysround.com/concepts/service-discovery) - [MVCC](https://sysround.com/concepts/mvcc) --- SysRound (https://sysround.com) is an AI practice tool for system design interviews: a voice or typed AI interviewer, a shared whiteboard, then feedback on every skill and a model answer. Its feedback is AI-generated and can be wrong.