How the interviewer opens it
“Design a distributed key-value store like DynamoDB or Cassandra. It should support putting and getting values by key, scale across many machines, and keep working when machines fail.”
The prompt is vague on purpose. The first few minutes are yours: narrow it to the core features and agree on the numbers before you draw anything.
What to clarify first
- operations
- scale and read:write mix
- record sizes
- traffic skew
- latency
- consistency versus availability
- durability
- deployment and failures
- deletion semantics
- out of scope
The answers stay with the interviewer. In a SysRound round, Alex answers them as an interviewer would.
Scope of a strong answer
Core requirements
- Put and get an opaque value by key
- Delete a key without allowing old replicas to resurrect it
- Replicate data and continue serving through machine failures
- Add and remove storage machines without losing data
Usually out of scope
- Transactions across keys and conditional writes
- Range scans, secondary indexes, and queries by value
- Multi-region replication
- TTL expiration in v1
- Building authentication, billing, or a tenant-management system
What interviewers probe
- Placement changes without losing ownership of data
- Hashing determines placement, but failures and rebalancing create periods when old and new machines both have relevant data. Moving ownership too early loses reads or writes; deleting old copies too early can destroy the only current version.
- Available writes and conflict resolution during partitions
- The store must durably acknowledge writes even when the preferred replicas are split. Fallback replicas improve availability but break simple quorum-intersection reasoning, and concurrent successful writes still need to converge.
- Durability, repair, and preventing deleted data from returning
- Replication is not complete when a write is acknowledged. Delayed delivery, disk failures, missed tombstones, and compaction can cause permanent loss or resurrection unless local persistence and cluster repair agree.
- Hot keys and recovery competing with foreground traffic
- A single key remains tied to a small replica group even in a large cluster. Meanwhile, compaction and replica rebuilds consume the same disk and network resources needed to meet tail-latency targets.
Common mistakes
- Listing consistent hashing, replication, and gossip without walking through one put, one get, and one failed-node recovery.
- Claiming R+W>N proves strong consistency, especially when using fallback replicas.
- Acknowledging buffered writes without specifying durable storage on two machines.
- Relying only on hinted handoff or read repair, leaving lost hints and cold keys unrepaired.
- Removing tombstones by age alone and allowing stale machines to resurrect deleted data.
- Assuming more shards solve a hot key, or omitting the capacity needed for compaction and rebuilding replicas.
The bar by level
- Senior (L5/E5)
- Defines the acknowledgment and consistency contract precisely, including fallback replicas and insufficient durable copies. Covers version resolution, WAL recovery, handoff plus anti-entropy, safe ownership transfer, tombstone retention, and hot-read mitigation. Uses the workload estimates to justify capacity and recovery headroom.
- Staff (L6+/E6+)
- Tests the design against partitions, stale placement, long-offline nodes, and correlated failures without quietly changing its guarantees. Separates metadata agreement from data-plane availability, makes rejoin and garbage-collection safety explicit, and quantifies trade-offs between foreground latency, repair time, and durability exposure. Can explain exactly why adding linearizability or multi-region writes requires a different protocol or weaker availability.