How it works
A participant periodically sends a heartbeat to a monitor, or a monitor sends probes and expects replies. For example, a worker sends its worker ID, process incarnation, and current task IDs to a scheduler every two seconds.
The receiver records when it last heard from each participant. It measures elapsed time with its own monotonic clock rather than comparing wall-clock timestamps from different machines. If no fresh heartbeat arrives within a configured timeout, it marks the participant as suspected or unavailable.
A timeout cannot distinguish a crashed process from a network partition, CPU starvation, or a long garbage-collection pause. Different observers can disagree about the same node, so a local suspicion should not automatically become a cluster-wide ownership change.
Choose a response based on the risk. A presence service can expire a viewer session; a router can stop sending new requests to an unreachable server. Reassigning a job may cause duplicate execution, while promoting a database replica requires an authoritative ownership decision and a way to prevent the old primary from continuing to write.
At scale, avoid having every node probe every other node. Use a coordinator, hierarchical monitoring, or bounded peer probes with gossip to spread suspicions. Add scheduling jitter to avoid synchronized heartbeat bursts, and use process incarnation IDs or sequence numbers to reject stale messages.
Recovery also needs a policy. A returning node may need several successful checks, replica catch-up, or a new ownership epoch before serving traffic. A heartbeat proves that its reporting path is responsive; it does not prove the application can perform useful work.
A worked example
Reassigning a stalled worker safely
A scheduler manages 1,000 workers, each sending a heartbeat every two seconds: about 500 heartbeat messages per second. It suspects a worker after six seconds without a heartbeat, with a one-second scan adding up to another second of detection delay. A worker pauses for eight seconds, so the scheduler suspects it even though it has not crashed. Before reassigning its exclusive task, the scheduler atomically advances the task's ownership epoch from 41 to 42. The storage service rejects writes carrying epoch 41. When the old worker resumes, its writes fail instead of conflicting with the replacement. External side effects still need idempotency or equivalent protection.
Trade-offs to say out loud
- Fast detection versus false suspicions
- Short timeouts reduce detection delay but make ordinary network jitter and process pauses more likely to trigger unnecessary recovery.
- Heartbeat frequency versus overhead
- With N participants reporting every H seconds, a central monitor receives roughly N/H heartbeat messages per second before retries and replication.
- Simple monitoring versus monitor failure
- A central monitor simplifies decisions but needs its own availability plan, while decentralized monitoring introduces delayed and conflicting views.
- Reachability versus application health
- A lightweight heartbeat can remain healthy while requests fail, but deep dependency checks can be expensive and cause cascading service removal.
- Recovery speed versus exclusive ownership
- Replacing a suspected owner quickly improves availability, but exclusive writes require fencing or another enforceable ownership mechanism.
- Stable membership versus rapid reinstatement
- Requiring a recovery window reduces repeated removal and reinstatement, but keeps a recovered participant out of service longer.
When to reach for it
- A scheduler or queue must reclaim work when a worker stops responding.
- A replicated store needs to detect an unreachable leader before attempting safe failover.
- A chat or viewer-count service must expire presence without relying on reliable disconnect events.
- A service registry or routing layer must stop directing new traffic to unreachable instances.
- A fleet controller must distinguish recently reachable agents from machines whose execution state is unknown.
How to bring it up in the interview
- Start with the failure being detected and the action it enables: expiring presence is different from transferring write authority.
- Give an initial heartbeat interval and timeout as design assumptions, then explain how observed network delays, process pauses, and recovery objectives would guide tuning.
- Say explicitly that missing heartbeats create suspicion, not proof of death; expect the interviewer to ask what happens during a partition or a long process pause.
- For exclusive ownership, explain who authorizes the replacement and how stale owners are rejected, such as fencing tokens validated by the storage layer.
- Account for the full recovery path: detection, decision, fencing, reassignment, and warm-up all contribute to downtime; also explain what happens when the monitor fails.
Common mistakes
- Treating a timeout as proof that a process has stopped and cannot produce more side effects.
- Promoting a replica or reassigning exclusive work based only on one observer's missing heartbeats.
- Comparing sender and receiver wall-clock timestamps without accounting for clock skew.
- Assuming a responsive heartbeat endpoint means dependencies, request handling, and data freshness are healthy.
- Ignoring late heartbeats, restarted processes, and repeated failure-and-recovery cycles when updating membership.
Where it comes up
Interview questions where heartbeats and failure detection is part of a strong answer, and why.
- Design a distributed cacheFailure detection drives replica failover and membership changes without allowing conflicting ownership decisions.
- Design a chat app like WhatsAppHeartbeats detect stale client connections and expire presence when disconnect notifications never arrive.
- Design a distributed message log like KafkaBroker failure detection initiates leader failover, while consumer heartbeats determine when partition assignments can be transferred.
- Design a distributed message queue like RabbitMQWorker liveness and lease renewal determine when abandoned messages become eligible for redelivery.
- Design a distributed job schedulerWorker heartbeats and task leases enable recovery from stalled execution while exposing duplicate-execution risks.
- Design a control plane for a distributed databaseMissing database heartbeats must lead to an authoritative, fenced failover rather than an unsafe second primary.
- Design a live viewer countHeartbeat expiration defines when abandoned viewing sessions stop contributing to the live count.
- Design a system to run commands across a fleetAgent heartbeats establish recent reachability without falsely proving whether an interrupted command completed.
Questions people ask
What is a heartbeat mechanism in a distributed system?
It is a periodic exchange used to track whether a participant is reachable. A monitor records recent contact and suspects failure when messages stop arriving within a timeout. Heartbeats can also carry progress or lease-renewal information, but those meanings require explicit rules.
How do you choose a heartbeat interval and failure timeout?
Start with the desired detection delay, expected network variation, process pauses, and monitoring overhead. Allow enough time for multiple heartbeat opportunities rather than treating one missed message as failure. An adaptive detector, such as a phi-accrual detector, can estimate suspicion from observed arrival patterns, but it still cannot prove that a node has crashed.
What is the difference between a heartbeat and a lease?
A heartbeat reports recent contact; a lease grants authority for a limited period under defined renewal and expiration rules. A heartbeat may renew a lease, but receiving it does not inherently grant ownership. Safe lease-based ownership requires explicit timing assumptions and often fencing to reject operations from a stale holder.