How the interviewer opens it
“Design the control plane for a distributed database service. It should provision customer clusters, track their health, and handle failover, scaling, and upgrades across thousands of databases.”
The prompt is vague on purpose. The first few minutes are yours: narrow it to the core features and agree on the numbers before you draw anything.
What to clarify first
- Fleet size and topology
- Management traffic and health reporting
- Lifecycle and scaling scope
- Latency targets
- Failover contract and engine capabilities
- Consistency and durability
- Control-plane availability
- Infrastructure interfaces
- Upgrade safety
- Boundaries
The answers stay with the interviewer. In a SysRound round, Alex answers them as an interviewer would.
Scope of a strong answer
Core requirements
- Create and delete clusters asynchronously, with operation status and retry-safe requests.
- Track desired configuration, observed state, and health for each customer cluster.
- Replace failed replicas and coordinate failover without split brain or loss of acknowledged writes.
- Resize compute, grow storage, and perform supported rolling engine upgrades.
Usually out of scope
- Implementing the database engine, replication protocol, or query execution
- Automatic cross-region database failover
- Resharding, storage shrinking, and arbitrary engine-version migrations
- Database backup and point-in-time restore
- Billing and a complete identity platform
What interviewers probe
- Making long-running operations survive ambiguity
- Provisioning and replacement cross a transactional metadata store and nontransactional infrastructure APIs. A timeout can mean success, and a restarted worker can repeat or conflict with work that is still running.
- Failover authority and split-brain prevention
- The controller's view of health can disagree with the database network. Promoting based on missing heartbeats can create competing writers unless the engine's existing quorum rules remain authoritative.
- Scaling and upgrades without consuming quorum
- A replacement can exist without being caught up, a process can be running without being ready, and a safe binary rollback may become unsafe after a format change. Independent maintenance actions can combine into an outage.
- Containing fleet-wide control-plane failures
- The dangerous load is often correlated repair work, not customer API traffic. A broken telemetry path or controller release can turn a healthy fleet into thousands of simultaneous replacements.
Common mistakes
- Spending the interview designing the database replication algorithm instead of the control plane.
- Treating provisioning, resizing, or upgrades as synchronous HTTP requests.
- Confusing desired state, reported health, and authoritative engine membership.
- Assuming heartbeat loss proves a node is dead, or that changing DNS prevents split brain.
- Using locks and retries without explaining late workers, unknown external outcomes, or duplicate resource creation.
- Ignoring correlated failures and allowing repair, scaling, and upgrade work to consume the same cluster's quorum simultaneously.
The bar by level
- Senior (L5/E5)
- Designs durable reconciliation with recoverable external side effects, explicit operation ordering, and a credible fencing story. Separates failure suspicion from promotion authority, preserves quorum through replacements and upgrades, and keeps the data plane running during control-plane outages. Handles provider timeouts and a maintenance interruption concretely.
- Staff (L6+/E6+)
- Defines safety and availability boundaries across the API, metadata store, workers, agents, infrastructure provider, and engine. Identifies where fencing is actually enforceable and where unknown outcomes require serialization. Handles correlated failures, regional isolation, repair prioritization, and fleet rollout limits; distinguishes reversible actions from irreversible migrations and justifies a simple architecture at the stated scale.