Cache stampede and thundering herd in system design

How it works

Start with a hot cache entry. If it receives 10,000 requests per second and takes 200 milliseconds to rebuild, roughly 2,000 requests can arrive during one rebuild. Without coordination, each miss may start the same expensive operation; backend slowdown then extends the miss window.

Use per-key request coalescing, often called singleflight: one request performs the fetch while others share its result. An in-process implementation only coordinates requests within one application instance. Coordinating across instances requires a shared mechanism, such as a cache-fill lease.

For a shared lease, a miss handler tries to become the refresher, then rechecks the cache before fetching. Other callers wait within a deadline, serve an allowed stale value, or receive a controlled failure. The lease needs an expiry for crashed refreshers and ownership-checked release; an expired lease can still allow overlapping refreshes.

When stale data is acceptable, separate freshness from retention. For example, mark a value stale after 60 seconds but retain it for another 30 seconds. During that grace period, serve the old value while one worker refreshes it. After the maximum allowed age, stop serving it.

Add randomized TTLs to spread expiration across different keys, and consider refreshing popular entries before they expire. TTL jitter does not stop simultaneous misses for a single hot key, so it complements rather than replaces request coalescing.

Protect the origin even when coordination or the cache fails. Bound concurrent refreshes, cap waiting callers, and use admission control or load shedding rather than sending every miss to the database. The broader thundering herd can also follow reconnects, retries, or worker wakeups; randomized backoff and controlled admission address those triggers.

A worked example

Refreshing a popular weather forecast

Suppose a city's forecast receives 12,000 requests per second across 20 application instances. An upstream fetch takes 250 milliseconds, so about 3,000 requests arrive during one refill. Independent miss handling could turn those requests into thousands of duplicate upstream calls. Keep the forecast fresh for 60 seconds and retain it for a 30-second stale grace period. After freshness expires, one worker acquires a shared refresh lease; other requests receive the retained forecast with its observation time. A successful refresh replaces the cached value. If refreshes keep failing, stop serving it after 90 seconds and return a controlled unavailable response. A global refresh limit also protects the upstream when many cities expire together.

Trade-offs to say out loud

Less duplicate work, more coordination: Shared leases reduce cross-instance fetches but introduce another dependency, coordination latency, and lease-expiry failure cases.
Backend protection versus request latency: Waiting for one refresh saves backend capacity but can increase tail latency unless callers have bounded deadlines.
Availability versus freshness: Serving stale data keeps reads available during refresh failures, but requires an explicit age limit and is unsuitable for some authorization or inventory decisions.
Expiration smoothing versus single-key protection: TTL jitter spreads refreshes across keys but does not prevent a stampede when one popular entry expires or is invalidated.
Prewarming versus wasted work: Refreshing before expiry can avoid cold misses but consumes capacity for entries that may never be read again.
Per-key protection versus aggregate overload: Coalescing each key still permits too many distinct keys to refresh together, so the backend also needs a global concurrency budget.

When to reach for it

A read-heavy design depends on cached values that are expensive to fetch or compute.
A celebrity post, popular short URL, or widely requested forecast concentrates traffic on one cache key.
Bulk invalidation, synchronized TTLs, deployment, or cache-node loss can turn many hits into misses at once.
The interviewer asks what happens when the cache is cold, unavailable, or much less effective than expected.
Many clients reconnect or retry after the same outage, creating a broader thundering herd.

How to bring it up in the interview

Quantify the exposure: multiply the hot key's request rate by its rebuild time to estimate requests arriving during a miss window, then compare the resulting work with backend capacity.
State the coordination scope. Say whether singleflight operates per process, per cache node, or across the fleet; interviewers will probe whether each application instance still starts its own fetch.
Define the freshness contract before proposing stale serving: how old may the value be, and which changes must bypass or invalidate that grace period?
Walk through refresher failure, lease expiry, and cache outage. Explain bounded waits, ownership-checked lease release, and whether a late refresh could overwrite newer data.
Name the operational signals: origin fetches per key, refresh concurrency, coalesced waiters, stale-response age, and backend saturation. Test synchronized expiry and cache loss rather than only warm-cache traffic.

Common mistakes

Treating TTL jitter as a complete solution for a single hot key.
Claiming in-process singleflight produces only one backend fetch across the entire fleet.
Adding a distributed lock without specifying lease expiry, safe release, waiter deadlines, or behavior when the lock service is unavailable.
Letting every request bypass the cache during an outage, shifting the full traffic load onto a smaller backend.
Serving stale values indefinitely or using cached inventory or permissions as the final authority for correctness-sensitive actions.

Where it comes up

Interview questions where cache stampede and thundering herd is part of a strong answer, and why.

Questions people ask

What is the difference between a cache stampede and a thundering herd?

A cache stampede is duplicate backend work caused by concurrent cache misses, often after expiration or invalidation. A thundering herd is broader: many clients or workers act simultaneously after any shared trigger, such as an outage ending or a lock becoming available.

Does adding random TTLs prevent cache stampedes?

Random TTLs reduce synchronized expiration across different keys. They do not prevent many requests from missing the same hot key together. Use per-key request coalescing or coordinated refreshes for that case.

How do you prevent a cache stampede across multiple servers?

Combine local request coalescing with a shared per-key refresh lease or a cache layer that coordinates fills. Callers that do not refresh should serve permitted stale data or wait within a deadline. Bound total refresh concurrency as well, because per-key coordination does not protect against a large number of distinct misses.