How the interviewer opens it
“Design a system that lets an operator run commands or scripts on hundreds of thousands of machines and see each machine's result. Some machines may be offline or take a long time to respond.”
The prompt is vague on purpose. The first few minutes are yours: narrow it to the core features and agree on the numbers before you draw anything.
What to clarify first
- fleet and job scale
- traffic mix and heartbeats
- targeting and concurrency
- latency
- offline machines and deadlines
- consistency, durability, and retries
- results and retention
- cancellation
- security and connectivity
- deployment and scope
The answers stay with the interviewer. In a SysRound round, Alex answers them as an interviewer would.
Scope of a strong answer
Core requirements
- Create an authorized command job against a fixed set of machines
- Dispatch and execute work with deadlines and concurrency limits, including machines that reconnect later
- Report per-machine status, exit code, and bounded command output
- Cancel outstanding dispatch and request termination of running work
Usually out of scope
- Machine provisioning and agent installation
- Interactive terminals
- Sandboxing hostile scripts
- Automatic rollback of command side effects
- Active-active multi-region control plane
What interviewers probe
- Durable fan-out and offline delivery
- One small submission can create 300,000 obligations, many of which must survive days without a connected recipient. A broadcast or an in-memory queue cannot represent that responsibility.
- Duplicate delivery versus duplicate side effects
- The server can retry a message safely only if it distinguishes delivery from execution. There is no general atomic transaction between a local journal, spawning a process, and arbitrary external side effects.
- Deadlines, cancellation, and uncertain machine state
- The control plane cannot reliably distinguish a dead machine from a partitioned machine whose script is still running. Incorrect state labels can cause duplicate actions or give operators false confidence.
- Fleet-wide safety and backpressure
- A valid command can still cause a fleet outage, and a successful fleet-wide launch can overwhelm result ingestion. Limiting job submission QPS does not control either risk.
Common mistakes
- Designing around job-submission QPS instead of per-target fan-out, heartbeats, and result traffic.
- Using direct SSH connections or assuming every machine is reachable when a job is submitted.
- Claiming exactly-once arbitrary script execution through queue acknowledgements or leases.
- Treating offline, failed, expired, cancelled, and unknown as interchangeable states.
- Automatically retrying non-idempotent work after a network timeout.
- Ignoring fleet safety limits, output caps, and the hot aggregate row created by a fleet-wide job.
The bar by level
- Senior (L5/E5)
- Makes fan-out resumable, separates delivery retries from execution retries, uses durable agent deduplication, and explains the crash gap around process launch. Defines deadline, timeout, cancellation, and unknown-result semantics; includes concurrency limits and a scalable result path.
- Staff (L6+/E6+)
- Defines precise guarantees across control-plane, agent, and external side-effect boundaries. Handles stale attempts, agent identity loss, authorization expiry, and cancellation races without claiming impossible guarantees. Connects uncertain work to rollout safety and permit accounting, and gives a capacity and recovery plan for fleet-wide bursts without unnecessary infrastructure.