How it works
Start with the routing key. If requests are partitioned by post ID, every request for one viral post may reach the same shard. A partition can also become hot because it contains many active keys, a large tenant, or the newest entries in a timestamp-based range.
Measure load rather than just data size. Compare per-key request rates, per-partition bytes and operations, queue lag, and tail latency. A shard holding little data can still saturate its CPU, network, disk, or write lock while the rest of the cluster is idle.
For hot reads, distribute copies of the value through local caches, a CDN, or read replicas where freshness requirements allow. A distributed cache partitioned by key can itself develop a hot key. Request coalescing limits duplicate cache fills, while expiration jitter helps prevent many entries from expiring together.
For hot writes that can be combined, split one logical key into physical subkeys. For example, route a post's counter updates across post:42:0 through post:42:15, then sum partial counts. This distributes writes but makes reads and retry-safe aggregation more complicated; it does not preserve one global update order.
The celebrity problem is also a fan-out problem. One author posting to millions of followers can generate millions of feed writes. A hybrid feed can materialize ordinary authors' posts on write but merge celebrity posts on read; live broadcasts can distribute delivery through multiple relay servers rather than one per-stream worker.
Match the remedy to the bottleneck. Repartitioning can spread a hot range or tenant, but ordinary hashing and virtual nodes do not split one indivisible hot key. If operations require serialized decisions, such as claiming the last seat, preserve that authority and use admission control, bounded queues, and backpressure to protect it.
A worked example
A viral post overwhelms one counter partition
Assume a viral post receives 120,000 like events per second, while one aggregation partition can process 10,000 per second. Partitioning only by post ID overloads one worker even if the cluster has spare capacity. Route events by (post ID, hash(user ID) mod 32), giving an expected 3,750 events per second per bucket if user IDs spread evenly. Keep authoritative like/unlike state per user and post, and use versioned, retry-safe transitions to update bucket totals. A read service sums the 32 partial counts and caches the displayed result briefly. This trades immediate count freshness and cheap reads for write capacity. Monitor bucket skew and aggregation lag; correctness still depends on handling retries and out-of-order transitions, not just adding buckets.
Trade-offs to say out loud
- Read replication versus freshness
- More cached copies spread reads but add invalidation work and can serve stale values.
- Write sharding versus simple reads
- Splitting a counter or aggregate spreads updates but requires merging partial results and handling duplicates correctly.
- Parallelism versus ordering
- Splitting one key across partitions removes their shared ordering boundary, so strict global order needs coordination that can become the next bottleneck.
- Fan-out on write versus fan-out on read
- Write-time fan-out makes feed reads cheap, while read-time merging avoids huge celebrity write bursts at the cost of more read work.
- Adaptive routing versus operational complexity
- Special routing for hot keys needs detection, routing metadata, safe migration, and protection against repeatedly moving load.
- Overload protection versus immediate service
- Rate limits and bounded queues preserve system health but may reject requests or delay work rather than eliminate excess demand.
When to reach for it
- The prompt includes viral links, celebrity accounts, popular posts, live events, or heavily watched stock symbols.
- A large tenant, geographic region, or time range can dominate traffic despite otherwise balanced storage.
- A counter, inventory row, or stream partition must absorb many concurrent operations for one logical entity.
- Adding consumers or storage nodes leaves one partition saturated while others remain underused.
- One accepted event triggers delivery or materialized updates for millions of recipients.
How to bring it up in the interview
- Name the partition key and trace one popular entity through storage, cache, queues, and delivery; interviewers often probe whether you merely moved the hotspot.
- Estimate the hottest entity's operations or bytes per second against one node's assumed capacity, rather than relying only on cluster-wide averages.
- Separate read skew, write contention, and fan-out amplification before proposing a fix; each has a different scaling limit.
- State the semantics you can relax: cached-value freshness, count freshness, per-key ordering, or delivery latency; expect follow-ups about what must remain exact.
- Explain detection and recovery, including per-partition metrics, bounded queues, cache refill behavior, and how routing changes avoid missing or duplicating work.
Common mistakes
- Claiming consistent hashing or more partitions will automatically spread requests for one unchanged key.
- Adding a shared cache without checking whether its single owner for the hot key becomes overloaded.
- Salting inventory or financial state like an additive counter without preserving the original correctness invariant.
- Adding consumers to a partitioned log and expecting them to process one ordered partition concurrently within the same consumer group.
- Treating celebrity fan-out as only a database read problem, while ignoring generated writes, network delivery, and slow recipients.
Where it comes up
Interview questions where hot keys and hot partitions is part of a strong answer, and why.
- Design a URL shortener like BitlyA viral short URL can overload one cache owner or storage shard on the redirect read path.
- Design a news feed like FacebookCelebrity authors force a choice between massive write-time fan-out and read-time feed merging.
- Design a distributed cacheHot-key ownership and cache-miss amplification determine whether the cache can survive skewed demand.
- Design an ad click aggregatorPopular ads require splitting ingestion and aggregation without losing retry-safe counting.
- Design a distributed message log like KafkaA hot routing key exposes the limit of scaling throughput while preserving partition ordering.
- Design Facebook likes at scaleViral posts require sharded aggregation, correct like/unlike transitions, and scalable count delivery.
- Design a top K trending systemA viral item's update stream must be distributed without breaking its aggregate score or ranking.
- Design Facebook live commentsA hot stream needs distributed fan-out and bounded delivery rather than one worker serving every viewer.
Questions people ask
What is the difference between a hot key and a hot partition?
A hot key is one logical key receiving disproportionate traffic or expensive operations. A hot partition is an overloaded shard or log partition, which may contain one hot key or many moderately busy keys. Splitting a range can help the latter without solving the former.
Does consistent hashing solve the hot key problem?
No. Consistent hashing distributes key ownership and limits reassignment when nodes change, but requests for one key still normally reach its owner. A hot key needs read replication, application-level splitting where valid, reduced work, or overload control.
How do you solve the celebrity problem in system design?
First identify whether the bottleneck is feed materialization, content reads, or live delivery. A hybrid feed can avoid writing a celebrity's post into every follower's feed, while caches and CDNs spread reads. Live delivery needs partitioned recipient groups, relay servers, and backpressure so one publisher does not directly serve every viewer.