# Apache ZooKeeper for system design interviews > Apache ZooKeeper is a replicated coordination service for small amounts of shared metadata, such as leader ownership, service membership, and configuration. Use it to coordinate distributed components, not to store application records or carry high-volume events. Source: https://sysround.com/tech/zookeeper Updated: 2026-11-05 ## How it works ZooKeeper exposes a hierarchical namespace of znodes, addressed by paths such as /services/payments/instance-1. Each znode holds a small byte payload, access controls, and metadata including a data version. Servers keep the data tree in memory and persist changes through transaction logs and snapshots. An ensemble contains voting servers, with one elected leader and the others acting as followers. Clients connect to any server; write requests are forwarded to the leader. ZooKeeper uses Zab, its atomic broadcast protocol, to order transactions and replicate them through a quorum before committing them. Committed writes have a single global order, identified by ZooKeeper transaction IDs called zxids. Ordinary reads are served locally by the connected server and can lag committed writes elsewhere. A client's view does not move backward across normal reconnections within a session, but that does not make every read globally fresh. A client session survives temporary connection loss if it reconnects before the negotiated session timeout expires. Ephemeral znodes belong to the session and are removed when that session expires or closes. A missed heartbeat does not immediately remove them, so membership changes are delayed failure signals rather than proof that a process has stopped. Clients register watches to learn that a znode or its children changed, then reread authoritative state. Traditional watches are one-shot notifications, not a durable stream of every update. ZooKeeper also supports non-voting observers, which serve clients and receive committed changes without increasing the voting quorum. ## What it gives you in a design - **Leader election.** Ephemeral sequential znodes provide ordered candidates for an application leader. Each candidate can watch its predecessor instead of having every candidate watch the current leader. - **Distributed lock recipes.** Ordered ephemeral contenders support exclusive ownership recipes. Protect external resources with fencing tokens as well, because an expired lock holder may still be running. - **Session-bound membership.** Services can publish ephemeral registration nodes that disappear after session expiration. Consumers must tolerate delayed removal and still handle connection failures to registered endpoints. - **Versioned conditional updates.** Updates and deletes can require an expected znode version, providing compare-and-set behavior. This prevents one controller from silently overwriting another controller's change. - **Atomic multi-operation transactions.** The multi API can atomically combine checks and mutations across znodes within one ensemble. It does not extend atomicity to an application database, a message broker, or an external API. - **Change notifications.** Watches let clients maintain local copies of configuration or membership without continuously polling. Clients must reconcile state after notifications and connection changes rather than treating notifications as a complete history. ## Scaling, replication and failure ZooKeeper replicates the full namespace across the ensemble rather than sharding it among voting servers. It is designed for small coordination state; namespace size, connection count, watch count, and write volume all affect capacity. Writes pass through one leader and require a voting quorum. Adding voters increases failure tolerance at certain ensemble sizes, but does not partition write work and adds replication overhead. Observers can offload client connections and reads without enlarging the voting quorum. With the usual majority configuration, three voters tolerate one unavailable voter and five tolerate two. A partition without a voting majority cannot commit writes. Normal service requires a functioning quorum; optional read-only mode can expose stale local data and must not be used to authorize exclusive ownership. Place voters across failure domains with reliable, low-latency links. Cross-region voters increase coordination latency and can make regional connectivity failures disruptive. During elections or quorum loss, pause ownership-sensitive work; after session expiration, rebuild session-owned state rather than assuming previous locks still belong to the client. ## Limits and defaults worth knowing - **Payload size:** Znode data should be small. The default jute.maxbuffer serialization limit is approximately 1 MiB; request overhead also matters, so do not treat that as a safe application payload budget. - **Session timeout bounds:** Unless overridden, the server's minimum session timeout is 2 × tickTime and its maximum is 20 × tickTime. The client requests a timeout, and the server negotiates it within configured bounds. - **Majority quorum:** Under the usual majority configuration, tolerating f unavailable voting servers requires at least 2f + 1 voters. Observers do not count toward that quorum. - **Ephemeral node children:** Ordinary ephemeral znodes cannot have children. Their lifetime follows the owning session, not a separately configured per-node TTL. - **Watch behavior:** Traditional watches fire once and must be registered again. Persistent and persistent recursive watches are available starting with ZooKeeper 3.6.0. ## Use it when - Several controllers need to agree on which instance currently owns a shard, partition, or background task. - A design needs small, shared coordination metadata with ordered writes and conditional updates. - Service membership should follow client sessions, and delayed failure detection is acceptable. - Applications can cache coordination state locally instead of contacting ZooKeeper for every user request. ## Avoid it when - The workload is application records, large objects, telemetry, or a high-throughput event stream. - Every request requires a globally fresh read from any replica without additional coordination. - Independent regions must continue changing the same ownership state during a network partition. - An existing database transaction or platform-native lease already provides the required coordination with less operational complexity. ## What interviewers ask about it ### How does ZooKeeper prevent two application leaders from writing at once? ZooKeeper can select one current owner, but it cannot stop a paused or partitioned former owner from continuing external work. Issue a monotonically increasing fencing token for each ownership grant and require the protected storage or service to reject older tokens. ### Are ZooKeeper reads strongly consistent? Writes are globally ordered, but ordinary reads come from the connected replica and may be stale. Explain the freshness the operation actually needs; use conditional writes or an appropriate synchronization protocol rather than treating a local read as proof of current ownership. ### What happens when the client loses its connection? Connection loss alone does not expire the session or immediately delete its ephemeral znodes. Suspend ownership-sensitive actions while disconnected, and if the session expires, obtain a new session and reacquire ownership before resuming. ### How would you implement a lock without waking every waiter? Create an ephemeral sequential contender and acquire the lock when it has the lowest sequence number. Otherwise watch only its immediate predecessor, handling the race where that predecessor disappears before the watch is installed. ### What if a write times out and the client does not know whether it succeeded? A connection error can leave the result unknown because the write may have committed before the response was lost. Reconcile using an operation identifier, known path, or expected version before retrying; blindly retrying a sequential create can produce duplicate contenders. ### Can watches replace an event log? No, watches indicate that state may have changed rather than preserving every intermediate update. Reread and reconcile current state after notifications, reestablish traditional watches, and use a durable log when replay or complete event history is required. ## Common mistakes - Putting ZooKeeper on the hot path for every request instead of keeping coordination off the data-serving path. - Treating a session timeout as proof that the former owner has stopped, and omitting fencing at the protected resource. - Assuming every replica read is globally fresh or that a watch reports every state transition. - Adding voting servers to scale writes, even though the ensemble still has one ordered write stream. ## Alternatives - **etcd:** Choose it for a coordination key-value API with explicit linearizable reads, transactions, leases, and revision-based watches, especially when the surrounding platform already operates it. - **Consul:** Choose it when service discovery, health checks, and service networking are the main requirements rather than only a low-level coordination namespace. - **Relational database:** Use conditional updates, unique constraints, advisory locks, or lease rows when coordination is closely tied to data already stored in that database and its capacity is sufficient. - **Kubernetes Lease objects:** Use them for controller leader election inside Kubernetes when its API and existing leader-election libraries meet the requirements. External side effects still need protection against stale leaders. ## Interview questions where it comes up - [Design a control plane for a distributed database](https://sysround.com/questions/design-database-control-plane): ZooKeeper can coordinate controller leadership and ownership metadata, while fencing protects databases from stale failover controllers. - [Design a distributed message log like Kafka](https://sysround.com/questions/design-kafka): It can hold broker membership and partition leadership metadata while message payloads remain in the replicated log. - [Design a distributed job scheduler](https://sysround.com/questions/design-job-scheduler): It can elect a scheduler leader, but dispatch and job side effects still require durable state and retry-safe execution. - [Design a distributed cache](https://sysround.com/questions/design-distributed-cache): It can coordinate cache membership and ownership epochs without participating in every cache read or write. - [Design a distributed message queue like RabbitMQ](https://sysround.com/questions/design-message-queue): It can coordinate queue-node ownership and leader election, but should not become the storage path for high-volume queue messages. ## FAQ ### What is ZooKeeper architecture? ZooKeeper runs as a replicated ensemble with a leader, voting followers, and optional non-voting observers. The leader orders writes through Zab and commits them through a quorum, while connected servers serve reads from their local copies of the znode tree. ### Why does ZooKeeper usually use an odd number of servers? With majority voting, an odd number gives better failure tolerance per server. Three voters tolerate one failure, while four still tolerate only one; five tolerate two. Observers are separate from this calculation. ### Is ZooKeeper a database or a service discovery tool? It is a coordination service with a small hierarchical data store. Service discovery is one use: instances publish ephemeral registrations and clients watch membership. It is not a general-purpose database or a complete health-checking and traffic-routing system. ## Related concepts - [Consensus algorithms](https://sysround.com/concepts/consensus) - [Quorums](https://sysround.com/concepts/quorums) - [Leader election](https://sysround.com/concepts/leader-election) - [Distributed locks and fencing tokens](https://sysround.com/concepts/distributed-locks) - [Service discovery](https://sysround.com/concepts/service-discovery) --- SysRound (https://sysround.com) is an AI practice tool for system design interviews: a voice or typed AI interviewer, a shared whiteboard, then feedback on every skill and a model answer. Its feedback is AI-generated and can be wrong.