How it works
Start with a resource and an invariant: for example, only the current worker may update a job's result. Workers acquire a lock scoped to that resource before doing work. Use a database transaction or conditional update instead when it can enforce the invariant directly.
A lease expires unless its owner renews it, allowing another worker to take over after a crash. But a long process pause or network partition can outlast the lease: worker A pauses, its lease expires, worker B acquires ownership, and A later resumes. Both workers can now attempt writes.
For a basic Redis distributed lock, acquire with SET lock:job:7 <random-owner-id> NX PX 10000. Release with an atomic compare-and-delete operation, commonly a Lua script, so an expired owner cannot delete its successor's lock. Renewal must also atomically check ownership; a client-side GET followed by DEL or PEXPIRE is unsafe.
For fencing, the ownership authority assigns a strictly increasing token to each successful acquisition, such as 41 for A and 42 for B. Token assignment must be tied to the ownership grant, with ordering that survives failover. A random owner ID is useful for safe release, but it is not a fencing token.
Every protected write carries the token. The destination atomically checks it against a durable per-resource high-water mark, rejects lower tokens, and records higher ones with the write. Once it accepts token 42, it rejects A's token 41. Equal tokens may allow multiple operations by the same owner; operation ordering and retry deduplication are separate concerns.
Fencing only works where it is enforced. A highest-seen-token check does not immediately revoke A when B acquires a lease: the destination must first learn the newer epoch, or validate current authority through a stronger protocol. For strict handoff, establish the new fence before enabling the new owner. Redis asynchronous replica promotion can lose an acknowledged lock, so basic Redis locking alone is not a correctness proof.
A worked example
Rejecting a late scheduler result
A scheduler grants worker A a 10-second lease for job 7 with token 41. A pauses for 15 seconds. After the lease expires, worker B acquires token 42. Before B starts, the scheduler establishes epoch 42 in the job's result row through an atomic update that cannot lower the epoch. B publishes its result with a conditional write requiring epoch 42. A resumes and tries to publish with epoch 41; the database rejects it. The row update is the enforcement point, not the worker's belief that it holds a lock. If the job also charges a payment provider, that separate side effect needs provider-supported idempotency or another explicit safety mechanism.
Trade-offs to say out loud
- Recovery speed versus lease churn
- Short leases speed up takeover but increase renewal traffic and ownership changes during pauses; long leases delay recovery.
- Safety versus availability
- If ownership cannot be established safely during a partition, a correctness-critical design must delay acquisition rather than let both sides proceed.
- Destination enforcement
- Fencing requires every protected mutation path to validate authority, which may be impossible for an external API that does not support it.
- Coordination cost
- Lock acquisition, renewal, and handoff add latency and contention, especially when a coarse lock serializes unrelated work.
- Additional durable state
- Token issuance and destination high-water marks must survive failover without moving backward, adding storage and recovery requirements.
- Limited scope of protection
- Fencing blocks stale owners but does not make several writes atomic or prevent duplicate side effects from retries by the current owner.
When to reach for it
- A scheduler or queue worker can lose its lease while still running, and its late result must not overwrite a newer attempt.
- A leader controls a database, shard, or other resource whose old leader may remain reachable after failover.
- An expiring slot or assignment can be reused, but the previous holder may continue acting after expiration.
- A migration cutover needs to revoke the old writer before allowing the new system to accept writes.
- Several services need exclusive access to a resource that cannot be protected by one local database transaction.
How to bring it up in the interview
- State the invariant and enforcement point first: 'Only the current ownership epoch may publish this result, and the result store checks that epoch atomically.'
- Walk through the paused-worker timeline explicitly; interviewers often ask what happens when the process resumes after its lease expires.
- Separate the random lock-owner ID, the increasing fencing token, and the operation's idempotency key; each solves a different problem.
- Explain who issues tokens, how acquisition and token ordering are coupled, and what prevents token rollback during failover.
- Describe the handoff barrier and every side-effect destination; expect follow-up questions about external APIs, renewal failures, and whether a conditional database update would be simpler.
Common mistakes
- Treating TTL expiration as proof that the previous worker has stopped executing.
- Releasing or renewing a Redis lock without an atomic ownership check.
- Calling a random UUID a fencing token, or issuing increasing tokens independently of successful ownership grants.
- Assuming a fencing token protects anything when the destination does not validate it atomically with the mutation.
- Using a lock as a substitute for transactions, idempotency, or a safe failover protocol.
Where it comes up
Interview questions where distributed locks and fencing tokens is part of a strong answer, and why.
- Design a distributed job schedulerWorker leases can expire during execution, so stale attempts must be prevented from publishing results after reassignment.
- Design a distributed message queue like RabbitMQVisibility leases allow redelivery while an old consumer is still running, requiring explicit ownership checks and duplicate-processing semantics.
- Design Netflix's concurrent screen limitReusing an expired playback slot is safe only if the old session's authority is also revoked or bounded at playback enforcement points.
- Design a control plane for a distributed databaseSafe failover requires fencing the old primary rather than assuming a lost lease or missed heartbeat has stopped it.
- Design a large data migration to the cloudA safe cutover must fence old writers before directing authoritative writes to the destination.
Questions people ask
What is a distributed lock?
A distributed lock coordinates exclusive access to a named resource across processes or machines. Most practical implementations use leases so a crashed owner does not block progress forever. Lease expiration permits takeover, but does not physically stop the old owner.
What is a fencing token, and how is it different from a lock ID?
A fencing token is an increasing ownership epoch attached to operations so the protected resource can reject older owners. A random lock ID only identifies the holder, typically to make release and renewal safe. Fencing requires destination-side checks and token ordering that remains valid through failures.
Is a Redis distributed lock safe?
A Redis lease can be useful for reducing duplicate cache refreshes or other work where occasional overlap is acceptable. Atomic acquisition and ownership-checked release are necessary, but lease expiry and asynchronous failover still create hazards. For correctness-critical writes, enforce the invariant with transactions, conditional writes, or a reliable ownership authority plus destination-side fencing; neither a basic Redis lock nor Redlock alone stops stale side effects.