How it works
Start with access patterns and the unit you need to keep together. Partitioning messages by conversation ID keeps one conversation's history and ordering local; partitioning by sender ID spreads a group conversation across partitions. A logical partition is not necessarily a server: one server can own many partitions.
Range partitioning assigns contiguous key intervals to partitions, such as customer IDs 1–100,000 and 100,001–200,000. It supports efficient range scans and ordered processing because a query can target the relevant intervals. Uneven key distributions or increasing keys, such as timestamps, can concentrate writes in one range.
Hash partitioning computes a hash of the partition key and uses it to select a partition. For example, hash(user_id) modulo 64 maps users into 64 logical partitions. This usually spreads many distinct keys well, but a query across user IDs may need every partition, and one heavily used user still maps to one partition.
Composite strategies combine locality and distribution. A metrics store might first group data by day, then hash series IDs into buckets within each day. A six-hour query touches the relevant day buckets, while writes to different series can spread across owners; deleting old data can remove entire expired day partitions.
A routing layer maps logical partitions to their current owners. A partition directory, range map, or consistent-hashing scheme can support ownership changes without changing every key's placement. Directly changing hash(key) modulo N to use a new N can remap most keys, so distinguish logical partition count from physical server count.
Growing the cluster requires moving data or splitting partitions, not just starting servers. A migration typically copies existing records, catches up concurrent changes, and then switches ownership with protection against stale routing. Replication is separate: partitioning decides which group owns a record, while replication maintains copies of that group.
A worked example
Partitioning a metrics ingestion workload
Suppose a metrics service ingests 800,000 samples per second and retains raw samples for seven days. Partitioning only by day sends all current writes to today's partition. Instead, use (day, hash(series_id) modulo 64) as the logical partition key and sort locally by series ID and timestamp. Assuming balanced traffic, today's 64 buckets average 12,500 samples per second each; benchmarks must determine their physical placement. A two-hour query for one known series targets one bucket per intersected day, while a fleet-wide query fans out across buckets. One unusually busy series still stays in one bucket. If it exceeds capacity, split its writes into sub-buckets and accept the extra merge work on reads.
Trade-offs to say out loud
- Range locality versus uneven load
- Range partitioning makes interval queries efficient, but skewed values or append-only keys can overload a single range.
- Hash distribution versus query fan-out
- Hashing spreads distinct keys, but queries without the partition key often require scatter-gather across many partitions.
- Colocation versus hot partitions
- Keeping a tenant or conversation together simplifies local operations, but a large tenant or busy conversation can exceed one partition's capacity.
- Salting versus ordering and aggregation
- Adding a bucket suffix can spread one hot logical key, but reads must merge buckets and strict per-key ordering becomes harder.
- Small partitions versus operational overhead
- More logical partitions give finer migration and scheduling control, but increase routing metadata, maintenance work, and potentially query fan-out.
- Local operations versus cross-partition coordination
- Transactions, uniqueness checks, joins, and secondary indexes spanning partitions need coordination or a different data model.
When to reach for it
- The dataset, write rate, or processing demand exceeds what one machine can handle with enough headroom.
- The workload has a natural ownership key, such as account ID, conversation ID, metric series ID, or queue ID.
- Range queries, retention windows, or globally ordered batch output make key locality important.
- A large tenant, popular item, or busy geographic area creates load skew that adding servers alone will not fix.
- A distributed log or processing pipeline needs parallelism while preserving order within a defined key.
How to bring it up in the interview
- Name the dominant reads and writes before choosing a key: 'Most requests fetch one conversation, so I would partition by conversation ID and order messages within that partition.'
- Separate the partition key, local sort key, logical partition count, and physical placement; interviewers often probe whether your range queries actually stay local.
- Estimate average and peak load per partition, then challenge the average with a large tenant or hot key; hashing balances distinct keys, not necessarily traffic.
- Explain what happens when capacity grows: which data moves, how concurrent writes are captured, and how stale routers avoid writing to an old owner.
- State the guarantee boundary explicitly: ordering or atomic updates may be local to one partition, and splitting that partition may require changing the guarantee or adding coordination.
Common mistakes
- Choosing a partition key solely for even row counts while ignoring request volume, row sizes, or query patterns.
- Claiming that hashing or consistent hashing automatically fixes a single hot key.
- Partitioning only by timestamp in a write-heavy system without addressing the active range becoming a bottleneck.
- Treating an index as a way to avoid cross-partition queries without explaining how that index is partitioned and maintained.
- Saying 'add more shards' without covering data movement, routing updates, and correctness during ownership changes.
Where it comes up
Interview questions where partitioning strategies is part of a strong answer, and why.
- Design a distributed key-value storePartition ownership, request routing, hot keys, and safe data movement are core parts of distributing the store.
- Design a system to sort huge datasetsRange boundaries determine global output order, while skew and duplicate keys determine whether workers receive balanced workloads.
- Design a distributed message log like KafkaThe partition key defines ordering scope and consumer parallelism, making hot-key scaling a direct design trade-off.
- Design a distributed cacheKey placement and ownership changes determine routing, cache redistribution, and behavior as cluster membership changes.
- Design a metrics monitoring system like DatadogTime buckets and series-based distribution control ingestion hotspots, retention operations, and query fan-out.
- Design an ad click aggregatorPopular ads require deliberate ingestion and aggregation partitioning so one ad does not become a single processing bottleneck.
- Design a distributed message queue like RabbitMQPartitioning determines how a hot queue scales and which delivery or ordering guarantees remain local.
- Design Google AnalyticsTenant skew and session-oriented processing require partition keys that balance ingestion without making analysis prohibitively expensive.
Questions people ask
What is the difference between range partitioning and hash partitioning?
Range partitioning groups neighboring key values, which helps interval scans and ordered processing. Hash partitioning maps keys through a hash function to spread distinct keys, but loses that cross-partition locality. Choose based on query patterns and expected skew rather than assuming either strategy is always faster.
What is the difference between partitioning and sharding?
Partitioning is the general act of dividing data into groups, even within one database server. Sharding usually means distributing horizontal partitions across independent database nodes. Terminology varies by system, so define whether you mean logical data groups or physical placement.
How do you choose a good partition key?
Choose a key that common queries can supply, distributes storage and traffic acceptably, and keeps operations needing local ordering or transactions together. Test it against large tenants, hot keys, and future growth. If those goals conflict, use composite partitioning, selective sub-bucketing, or additional query-oriented indexes and state their costs.