# Distributed cron and job scheduling > Distributed cron and job scheduling runs work at specified times across multiple machines, even when schedulers or workers fail. The core challenge is not parsing a cron expression: it is durably creating each scheduled occurrence, dispatching it without loss, and handling retries without repeating harmful side effects. Source: https://sysround.com/concepts/distributed-cron Updated: 2026-11-05 ## How distributed cron and job scheduling works Separate the schedule from its executions. A schedule stores a cron expression or interval, time zone, next due time, and overlap policy. Each execution is a durable occurrence with an identity such as (schedule_id, schedule_version, scheduled_time), plus its status and attempt history. Define time semantics before choosing infrastructure. “Every day at 9 a.m. in New York” differs from “every 24 hours.” Specify what happens during daylight saving time changes, whether missed occurrences are skipped or replayed after an outage, and whether a new occurrence can run while the previous one is still active. Schedulers query an index on next_due_at rather than scan every schedule. In a transaction, a scheduler claims due schedules, inserts occurrence records under a uniqueness constraint, and advances their next due times. Multiple schedulers can coordinate through row locks or shard leases; durable progress and uniqueness make crash recovery safe. Dispatch occurrences through a durable handoff. For example, write an outbox record in the same transaction as the occurrence, then let a relay publish it to a worker queue. If the relay crashes after publishing but before recording success, it can publish twice, so consumers must tolerate duplicate delivery. A worker atomically claims an occurrence with an expiring lease and renews it while running. Completion updates must match the current lease token so an expired worker cannot overwrite a newer attempt. Lease expiry allows recovery, but does not stop the old process or undo its external effects; use downstream idempotency keys or fencing where supported. Retry transient failures with backoff, cap attempts, and retain terminal failures for inspection or replay. Partition schedule ownership and scale workers separately as demand grows. Measure scheduled-to-start delay, oldest overdue occurrence, queue age, and lease expirations so a scheduler that is alive but falling behind is visible. ## Worked example: Handling a top-of-hour report burst Suppose 60,000 reports are due at 09:00, each taking two worker-seconds, and all must start within one minute. With 2,000 execution slots, the idealized processing rate is 1,000 reports per second: the final batch starts around 59 seconds after the first, leaving almost no margin for dispatch or slow reports. Schedulers create uniquely keyed occurrences and outbox records transactionally. Workers claim occurrences with renewable leases and use the occurrence ID as the report-publication idempotency key. Add capacity or negotiate a wider start window; average hourly throughput hides this burst. After an outage, enforce a bounded catch-up policy rather than releasing every missed report at once. ## Trade-offs - **At-least-once execution versus duplicate effects.** Retrying ambiguous failures avoids silently losing work, but safe effects require idempotency or transactional coordination beyond the scheduler. - **Polling cost versus start-time precision.** Shorter polling intervals reduce discovery delay but increase database traffic, and queueing or worker startup can still delay execution. - **Simple leadership versus partitioned scheduling.** A single active scheduler simplifies ownership, while partitioned schedulers increase capacity at the cost of lease management and reassignment. - **Fast recovery versus premature retries.** Short worker leases detect failures sooner but increase overlapping attempts when healthy workers pause or cannot renew. - **Catch-up completeness versus overload.** Replaying every missed occurrence preserves work but can overwhelm workers and downstream services after an outage. - **Strict timing versus capacity cost.** Meeting tight start-time targets during synchronized cron bursts requires spare capacity, admission limits, or permission to spread executions over time. ## When to use it in a system design interview - The design must run recurring jobs or one-time tasks at future deadlines and survive process restarts. - Multiple scheduler instances need to coordinate without losing occurrences or treating every replica as an independent cron runner. - Calendar reminders or escalation deadlines must remain correct after edits, cancellations, retries, and outages. - The interviewer asks how scheduled work behaves when execution takes longer than its interval or a worker disappears. ## How to bring it up in the interview - Start with the contract: time zone, acceptable start delay, missed-run policy, overlap policy, and cancellation semantics. Interviewers often probe these before throughput. - Draw schedule storage, occurrence storage, durable dispatch, and workers separately. Explain which transaction protects each state transition. - Walk through crashes before dispatch, after queue publication, and after an external effect but before completion is recorded. State explicitly where duplicates remain possible. - Distinguish claiming work from controlling its effects: a lease coordinates attempts, while idempotency or downstream fencing protects against stale workers. - Estimate the busiest scheduled instant rather than only average jobs per second, then defend worker capacity, tenant limits, backpressure, and the lateness metrics you would alert on. ## Common mistakes - Running the same cron configuration on every application replica and assuming only one instance will execute it. - Updating next_due_at separately from occurrence creation, leaving a crash window that can skip work. - Claiming that a distributed lock or queue acknowledgment guarantees exactly-once external side effects. - Treating lease expiration as proof that the previous worker has stopped. - Ignoring daylight saving time, cancellation races, overlap rules, and the backlog created by outage recovery. ## Interview questions where it comes up - [Design a distributed job scheduler](https://sysround.com/questions/design-job-scheduler): Durable occurrence creation, reliable dispatch, worker leases, and cron-burst capacity are the core design problems. - [Design Google Calendar](https://sysround.com/questions/design-google-calendar): Recurring reminders require durable scheduling with local-time semantics and protection against stale executions after event edits. - [Design an on-call escalation system like PagerDuty](https://sysround.com/questions/design-pagerduty): Durable escalation deadlines must survive failures and coordinate correctly with acknowledgments and changing responder schedules. ## FAQ ### What is the difference between distributed cron and a message queue? Distributed cron decides which occurrences become due and preserves recurring schedule progress across failures. A message queue distributes ready work and usually provides acknowledgment and redelivery. A durable queue with delayed delivery can support one-time timers, but recurring schedules still need recurrence and missed-run semantics. ### How do you prevent duplicate jobs in a distributed scheduler? Use a unique occurrence key and atomic claims to prevent duplicate occurrence records and competing valid owners. Retries can still repeat execution after a lease expires or an acknowledgment is lost. Protect the actual side effect with a stable idempotency key, a transaction, or a downstream fencing mechanism where available. ### How should distributed cron handle missed jobs and daylight saving time? Store the intended time zone and define behavior for local times that occur twice or do not occur during daylight saving transitions. After downtime, explicitly choose to skip missed occurrences, coalesce them into one run, or replay them with a limit. A bounded replay policy prevents recovery traffic from overwhelming the system. ## Related concepts - [Delay queues and scheduled delivery](https://sysround.com/concepts/delay-queue) - [Distributed locks and fencing tokens](https://sysround.com/concepts/distributed-locks) - [Idempotency and idempotency keys](https://sysround.com/concepts/idempotency) - [Transactional outbox pattern](https://sysround.com/concepts/outbox-pattern) --- SysRound (https://sysround.com) is an AI practice tool for system design interviews: a voice or typed AI interviewer, a shared whiteboard, then feedback on every skill and a model answer. Its feedback is AI-generated and can be wrong.