How it works
Each machine maintains its own clock. Skew is the difference between clocks; drift is how their rates diverge over time. If server A is 200 milliseconds fast and server B is 200 milliseconds slow, a request received by B can appear to arrive before A sent it.
Network Time Protocol (NTP) estimates clock offset using timestamped exchanges with time servers and adjusts the local clock. Variable network delay, asymmetric paths, and synchronization outages leave uncertainty. Synchronization reduces skew; it does not establish a universal maximum error unless the system separately enforces that assumption.
Use wall-clock time for calendar dates and externally meaningful timestamps, but a monotonic clock for local elapsed durations such as request timeouts. Wall clocks may jump or change rate during correction. Monotonic readings are not comparable across machines or restarts, and whether they include machine suspension depends on the clock API.
Choose an ordering mechanism that matches the requirement. A partition leader can assign sequence numbers; Lamport clocks preserve causal precedence but cannot identify every concurrent pair; vector clocks can distinguish concurrency. Hybrid logical clocks combine physical time with logical increments, but neither NTP nor logical timestamps alone provide linearizability.
Make expiration and ownership decisions at a defined authority. For example, a database can atomically check a seat hold's expiration against its own time rather than trusting an application server's clock. Lease-based safety still needs explicit timing assumptions, and protected resources should reject stale owners using fencing tokens or another authoritative ownership check.
For event processing, distinguish event time from processing time. A click timestamped 12:00:02 might arrive at 12:00:10 because of buffering, and the producer's clock may also be wrong. Define acceptable lateness, validate implausible timestamps, and decide whether late events update old windows or go through reconciliation.
A worked example
A paused worker outlives its lease
A scheduler grants worker A a 30-second lease and fencing token 41. After 10 seconds, A pauses for 40 seconds. The scheduler's authoritative lease state expires, and worker B receives token 42. When A resumes, its belief about ownership is irrelevant: its next protected write carries token 41. Once the resource has accepted token 42, it rejects token 41. The scheduler uses authoritative time for expiration, while workers use monotonic timers for local renewal attempts. This handles stale writes without assuming perfectly synchronized worker clocks. For an external email or payment API that cannot enforce fencing, use idempotency keys or another deduplication mechanism; the lease alone cannot prevent duplicate effects.
Trade-offs to say out loud
- Synchronization improves usefulness, not certainty
- NTP makes timestamps more comparable, but correctness still needs an explicit policy for excessive offset, lost synchronization, and clock corrections.
- Centralized time decisions simplify semantics
- An authoritative store can decide expiration consistently, but adds a dependency and requires safe behavior when that authority fails over.
- Logical ordering is not physical time
- Logical clocks and sequence numbers help order events, but cannot tell you how many real seconds elapsed or when a calendar deadline occurred.
- Conservative timing costs latency or availability
- Waiting through a defensible uncertainty bound or refusing work when clock health is unknown can preserve safety at the cost of slower or rejected operations.
- Late-event tolerance delays finality
- Allowing more late data improves aggregation completeness but retains more state and postpones final results.
- Fencing requires resource cooperation
- A monotonically increasing ownership token protects against stale workers only if the destination checks it and rejects older tokens.
When to reach for it
- Your design depends on expiring leases, seat holds, sessions, visibility timeouts, or leader authority.
- You are about to resolve concurrent writes or sort messages using timestamps from different machines.
- You need event-time windows, freshness guarantees, or meaningful latency measurements across services.
- A scheduler must execute durable deadlines despite restarts, outages, and wall-clock corrections.
- An interviewer asks what happens during a VM pause, clock rollback, synchronization failure, or regional failover.
How to bring it up in the interview
- Start with the required meaning of time: elapsed duration, calendar deadline, event timestamp, or causal order. Assign a clock or ordering mechanism to each rather than proposing one clock for everything.
- State the authority explicitly: 'The queue decides lease expiration; workers do not compare their local wall clocks to decide ownership.' Expect questions about authority failover and stale workers.
- Separate synchronization from correctness: 'We use NTP for usable timestamps, but sequence numbers and conditional writes enforce ordering.' If safety needs a clock-error bound, explain how exceeding it changes system behavior.
- Walk through a paused worker resuming after its lease expires. Explain where fencing is enforced and how retries remain idempotent; lease expiration alone cannot stop an old process.
- Name the operational checks: estimated clock offset, synchronization health, clock jumps, and implausible event timestamps. Expect probes about whether your response should alert, degrade, or stop accepting safety-sensitive work.
Common mistakes
- Saying NTP makes all clocks identical or guarantees a fixed skew bound under arbitrary network conditions.
- Using wall-clock subtraction for local timeouts, or comparing monotonic readings from different hosts.
- Treating the largest wall-clock timestamp as the latest causal update; a fast writer can overwrite a genuinely later write.
- Assuming an expired lease prevents its previous owner from continuing to write, without fencing or an equivalent authority check.
- Treating a watermark as proof that all earlier events have arrived, rather than a progress estimate under the processing system's assumptions.
Where it comes up
Interview questions where clock skew and time in distributed systems is part of a strong answer, and why.
- Design a distributed message queue like RabbitMQVisibility leases need authoritative expiration and safe handling of consumers that continue processing after ownership changes.
- Design a distributed job schedulerDurable deadlines and worker leases must survive clock corrections, restarts, and delayed execution without assuming exactly-once side effects.
- Design Netflix's concurrent screen limitPlayback lease expiration and slot reuse require explicit timing authority and protection against stale sessions.
- Design a distributed key-value storeTimestamp-based conflict resolution can discard causally later writes when replicas have skewed clocks.
- Design an ad click aggregatorEvent-time aggregation and final billing must account for late arrivals and inaccurate producer timestamps.
- Design a metrics monitoring system like DatadogSample timestamps, rate calculations, freshness checks, and missing-data alerts depend on clear clock and lateness semantics.
- Design a distributed tracing systemCross-host clock skew can produce misleading span timelines, so causal links and local duration measurements must guide interpretation.
- Design a control plane for a distributed databaseSafe failover must handle expired leader leases and paused controllers without allowing stale owners to mutate database resources.
Questions people ask
What is the difference between clock skew and clock drift?
Clock skew is the difference between two clock readings at the same instant. Clock drift describes how a clock's rate diverges from a reference over time, causing skew to change unless corrected.
Does NTP solve clock skew in distributed systems?
NTP estimates and corrects clock offset, making wall-clock timestamps more useful. It cannot eliminate uncertainty caused by network asymmetry, outages, or local clock behavior. Do not derive safety guarantees from a typical observed offset; define and enforce the timing assumptions your protocol actually requires.
When should I use a monotonic clock instead of a wall clock?
Use a monotonic clock to measure local durations, such as request latency, retry delays, and timeout budgets. Use wall-clock time for externally meaningful dates and durable calendar deadlines. Do not persist a monotonic reading as a portable deadline across machines or restarts.