How it works
Choose the unit of reuse. Whole-file deduplication hashes an entire file and reuses an existing object when the content matches. Chunk-level deduplication splits files first, allowing two different files or versions to share their unchanged pieces.
Fixed-size chunking cuts at regular offsets, such as every 4 MiB. It is simple and supports predictable parallel transfers, but inserting one byte near the start shifts subsequent boundaries and can make otherwise unchanged content produce different chunk hashes.
Content-defined chunking chooses boundaries from the bytes themselves. A rolling fingerprint scans a window of bytes, and a boundary rule selects cuts, subject to minimum and maximum chunk sizes. After an insertion, boundaries usually realign farther into the file, preserving reuse of much of the unchanged suffix. The configured target size is an average, not an exact size.
Compute a cryptographic digest, such as SHA-256, for each completed chunk. The rolling fingerprint chooses boundaries; the cryptographic digest identifies content. Look up each digest in a chunk index, upload missing chunks, and verify received bytes rather than trusting a client-supplied digest.
Represent each file version with an ordered manifest containing chunk identifiers, lengths, and the information needed to reconstruct the file. Make chunks durable before atomically publishing the manifest. Concurrent uploads of identical chunks should converge through conditional creation or an equivalent idempotent operation.
On reads, fetch the manifest and assemble its chunks in order, optionally fetching ahead. On deletion, remove the file's references, not necessarily its chunks: other versions may still need them. Reclaim unused chunks through coordinated reference tracking or reachability-based garbage collection that protects uploads still in progress.
A worked example
Syncing a small insertion into a large file
Suppose a 1 GiB file uses fixed 4 MiB chunks: its manifest contains 256 entries. A user inserts 1 KiB near the beginning. Fixed boundaries shift relative to the original content, so nearly every later chunk can receive a new hash. With content-defined chunking targeting a 4 MiB average, boundaries can realign after the insertion, allowing most later chunks to be reused. The client computes the new manifest, asks which chunks are missing within its authorized deduplication scope, and uploads only those. The server verifies and persists them before publishing the new version. The old version keeps its references until retention expires; exact savings depend on where boundaries fall.
Trade-offs to say out loud
- Smaller chunks improve reuse but expand metadata
- Smaller chunks can isolate edits more precisely, but create more hashes, index entries, manifest entries, and storage operations.
- Content-defined chunking handles shifted content at a CPU cost
- Scanning for content-based boundaries adds work compared with cutting at fixed offsets, so its value depends on how files change.
- Savings depend on the actual data
- Repeated versions can share substantial content, while unique media, recompressed files, and randomized ciphertext may offer little chunk reuse.
- Shared chunks complicate correctness and reclamation
- Manifest publication, concurrent uploads, and garbage collection must coordinate so a live file never references a missing chunk.
- Deduplication can increase read overhead
- Reconstructing a file from scattered chunks adds lookups and requests, which may require caching, prefetching, or packing chunks into larger storage objects.
- Broader deduplication creates privacy risks
- Cross-tenant existence checks can reveal whether content is stored, while tenant-scoped deduplication sacrifices some savings to simplify isolation.
When to reach for it
- A file-sync or backup design must transfer small edits without uploading every file in full.
- Many retained file versions contain mostly unchanged bytes, making storage reuse worth evaluating.
- Large transfers need bounded retry units, resumability, parallelism, and per-piece integrity checks, even if deduplication itself offers little benefit.
- The interviewer asks whether inserting bytes near the beginning of a file invalidates every later chunk.
- A storage or bandwidth estimate shows that repeated content is a material cost, rather than assuming deduplication always pays off.
How to bring it up in the interview
- Separate the requirements: chunking supports resumability and parallel transfer; deduplication avoids storing or transferring identical content. Either can be useful without the other.
- Start with fixed-size chunks unless the edit pattern justifies content-defined chunking. Explain the one-byte insertion problem, then discuss average, minimum, and maximum chunk sizes.
- Describe the write protocol, not just the hash function: verify chunks, make them durable, publish the manifest atomically, and clean up abandoned uploads safely.
- Estimate chunk count and metadata size alongside byte savings. Interviewers may probe index lookup throughput, small-object costs, and read amplification.
- State the deduplication scope and threat model. Expect follow-ups about unauthorized chunk references, hash collisions, deletion races, and whether end-to-end encryption prevents server-side reuse.
Common mistakes
- Using a rolling fingerprint or a Bloom filter as proof that two chunks contain identical bytes.
- Claiming content-defined chunking preserves every boundary after an edit; it usually resynchronizes, but reuse depends on the data and boundary rules.
- Publishing a file manifest before all referenced chunks are durable, or deleting a shared chunk when only one file reference disappears.
- Treating possession of a chunk hash as authorization to read it or attach it to a file.
- Confusing byte-level storage deduplication with event deduplication: identical event payloads can represent separate legitimate actions.
Where it comes up
Interview questions where data deduplication and chunking is part of a strong answer, and why.
- Design Dropbox or Google DriveChunk manifests, missing-chunk uploads, and safe version publication support resumable transfers and efficient synchronization of changed files.
- Design BitTorrentPiece boundaries and cryptographic hashes are central to parallel downloads, verification of untrusted peers, and resuming after crashes.
- Design a large data migration to the cloudChunked transfer with durable progress and integrity checks enables bounded retries and resumable copying across a bandwidth-limited link.
Questions people ask
What is the difference between fixed-size and content-defined chunking?
Fixed-size chunking cuts at regular byte offsets; content-defined chunking chooses boundaries using the content. Fixed-size chunks are simpler, but insertions can disrupt reuse across the rest of a file. Content-defined chunking usually restores alignment farther along the unchanged content, at the cost of additional scanning and variable-size chunks.
How does data deduplication differ from compression?
Deduplication replaces repeated files or chunks with references to one stored copy. Compression encodes data more compactly within its compression scope. They can be combined: identify reusable chunks from plaintext, then compress each unique chunk before storage.
Can encrypted data be deduplicated?
A service that can access plaintext can deduplicate before applying storage encryption. With client-side randomized encryption, identical plaintext generally produces different ciphertext, preventing ordinary server-side content deduplication. Deterministic or message-locked encryption can enable some reuse, but introduces equality leakage and possible guessing attacks; it is not a free substitute for standard end-to-end encryption.