Design a distributed job scheduler: system design interview question

How the interviewer opens it

“Design a distributed job scheduler. Users submit one-off and recurring jobs, and the system runs them at the right time on a pool of workers, with retries and status tracking.”

The prompt is vague on purpose. The first few minutes are yours: narrow it to the core features and agree on the numbers before you draw anything.

What to clarify first

scale
read:write mix
job execution
latency and timing
delivery and consistency
durability and deployment
cron semantics
retry policy
submission semantics
scope boundaries

The answers stay with the interviewer. In a SysRound round, Alex answers them as an interviewer would.

Scope of a strong answer

Core requirements

Create one-off jobs and recurring cron schedules
Execute due occurrences on a worker pool
Retry failed or abandoned executions with bounded backoff
Read schedule details, execution status, and recent history

Usually out of scope

DAG workflows and dependencies between jobs
Building a sandbox for arbitrary user code
Subsecond scheduling
Active-active scheduling across regions

What interviewers probe

Durable occurrence creation and cron recovery: A recurring schedule is not a run. Scheduler replicas must create each intended occurrence once, advance the schedule without gaps, and recover from downtime without creating an unwanted catch-up storm.
The database-to-queue handoff: Creating a durable run and publishing a queue message are separate writes. Either ordering leaves a crash window unless the design makes the handoff recoverable.
Worker leases, retries, and external side effects: A missing heartbeat cannot distinguish a dead worker from a slow or partitioned one. Retrying can overlap with an old attempt that still performs work.
Meeting start-time targets under cron bursts: A queue protects the database but cannot make a job start on time without worker capacity. Minute-boundary bursts and long jobs can consume hundreds of thousands of execution slots.

Common mistakes

Treating a cron schedule, a scheduled occurrence, and a retry attempt as the same record or identifier.
Keeping future work only in process memory or assuming an ordinary FIFO queue supports arbitrary durable future scheduling.
Leaving the run-creation-to-queue-publication crash window unexplained.
Equating lease expiry with worker termination or promising exactly-once side effects.
Ignoring worker concurrency and cron-boundary bursts when claiming a start-time guarantee.
Scanning all schedules repeatedly or introducing a large coordination system before identifying the actual write and timing bottlenecks.

The bar by level

Senior (L5/E5): Closes the occurrence-creation and queue-handoff crash windows, defines cron recovery behavior, and uses renewable leases with attempt ownership checks. Gives a credible retry and side-effect contract, estimates worker concurrency, and handles backpressure and tenant contention.
Staff (L6+/E6+): Defines the timing and durability guarantees precisely, including what changes during overload and failover. Identifies lifecycle writes and execution slots as distinct scaling constraints, keeps transactions local when partitioning, and explains recovery from stale schedulers, ambiguous side effects, and large backlogs. Proposes admission and capacity policies that make the SLO defensible without adding unnecessary global coordination.

How the interviewer opens it

“Design a distributed job scheduler. Users submit one-off and recurring jobs, and the system runs them at the right time on a pool of workers, with retries and status tracking.”

The prompt is vague on purpose. The first few minutes are yours: narrow it to the core features and agree on the numbers before you draw anything.

Scope of a strong answer

Core requirements

Create one-off jobs and recurring cron schedules
Execute due occurrences on a worker pool
Retry failed or abandoned executions with bounded backoff
Read schedule details, execution status, and recent history

Usually out of scope

DAG workflows and dependencies between jobs
Building a sandbox for arbitrary user code
Subsecond scheduling
Active-active scheduling across regions

What interviewers probe

Durable occurrence creation and cron recovery

A recurring schedule is not a run. Scheduler replicas must create each intended occurrence once, advance the schedule without gaps, and recover from downtime without creating an unwanted catch-up storm.

The database-to-queue handoff

Creating a durable run and publishing a queue message are separate writes. Either ordering leaves a crash window unless the design makes the handoff recoverable.

Worker leases, retries, and external side effects

A missing heartbeat cannot distinguish a dead worker from a slow or partitioned one. Retrying can overlap with an old attempt that still performs work.

Meeting start-time targets under cron bursts

A queue protects the database but cannot make a job start on time without worker capacity. Minute-boundary bursts and long jobs can consume hundreds of thousands of execution slots.

Common mistakes

Treating a cron schedule, a scheduled occurrence, and a retry attempt as the same record or identifier.

Keeping future work only in process memory or assuming an ordinary FIFO queue supports arbitrary durable future scheduling.

Leaving the run-creation-to-queue-publication crash window unexplained.

Equating lease expiry with worker termination or promising exactly-once side effects.

Ignoring worker concurrency and cron-boundary bursts when claiming a start-time guarantee.

Scanning all schedules repeatedly or introducing a large coordination system before identifying the actual write and timing bottlenecks.

The bar by level

Senior (L5/E5)

Closes the occurrence-creation and queue-handoff crash windows, defines cron recovery behavior, and uses renewable leases with attempt ownership checks. Gives a credible retry and side-effect contract, estimates worker concurrency, and handles backpressure and tenant contention.

Staff (L6+/E6+)

Defines the timing and durability guarantees precisely, including what changes during overload and failover. Identifies lifecycle writes and execution slots as distinct scaling constraints, keeps transactions local when partitioning, and explains recovery from stale schedulers, ambiguous side effects, and large backlogs. Proposes admission and capacity policies that make the SLO defensible without adding unnecessary global coordination.