# MapReduce for system design interviews > MapReduce is a distributed batch-processing model that transforms input records into key-value pairs, then groups and processes values by key. It fits large offline jobs such as aggregations, index construction, and historical recomputation, where throughput matters more than low latency. Source: https://sysround.com/tech/mapreduce Updated: 2026-11-05 ## How it works An input format divides a dataset into logical splits and turns each split into records for a map task. Splits are units of processing, not necessarily files or storage blocks. With distributed filesystems, a scheduler can place computation near the data; with object storage, workers generally read over the network. Each mapper emits intermediate key-value pairs. A partitioner chooses a reducer partition for each key, commonly using a hash. Map output is buffered, sorted, and spilled to local disk as needed, so processing does not require the entire dataset to fit in memory. During the shuffle, reducers fetch their partitions from map tasks and merge the sorted data. The framework groups values by key before invoking the reduce function. A combiner can perform safe local aggregation to reduce shuffle traffic, but correctness cannot depend on whether it runs. Reducers process each key and its values, then write output through an output format and commit mechanism. Reducers can start fetching completed map output while other maps still run, but final reduction requires the relevant map outputs. Multi-stage computations usually require multiple jobs with intermediate data written between them. A coordinator tracks task attempts and schedules retries when workers fail. In classic Hadoop MapReduce, intermediate map output is usually on worker-local disk, so losing a worker can require rerunning its completed maps. Hadoop MapReduce on YARN uses an ApplicationMaster to coordinate the job and YARN to allocate containers. ## What it gives you in a design - **Distributed aggregation.** Grouping by key makes counts, sums, grouped statistics, and similar batch computations straightforward. Define the intermediate key carefully because it determines both correctness and work distribution. - **Disk-backed processing.** Sorting, spilling, and merging let jobs process data larger than worker memory. The trade-off is substantial disk and network I/O, especially across multiple stages. - **Combiners.** Local partial aggregation can shrink shuffle volume when the operation supports safe repeated combination. For an average, combine sum-and-count pairs rather than averaging partial averages. - **Custom partitioning and sorting.** Partitioners control which reducer receives a key, while sorting and grouping rules control how records reach reduce calls. Range partitioning is useful for global sorting; composite keys support secondary-sort patterns. - **Task retries and speculative execution.** Failed attempts can be rerun, and speculative execution can launch duplicate attempts for slow tasks. This helps with slow workers, but it does not solve a reducer that owns disproportionately large input. - **Replayable batch computation.** Retained input can be recomputed after a logic change or used to rebuild derived datasets. Reproducibility still depends on versioned inputs, deterministic logic, and controlled external dependencies. ## Scaling, replication and failure Map parallelism largely follows the number of input splits, while reducer parallelism is configured separately. More reducers can distribute work, but they also increase shuffle connections, scheduling overhead, and output file counts. Estimate input bytes, intermediate expansion, shuffle bytes, and the largest partition before choosing capacity. Partitioning balances keys, not necessarily bytes or processing cost. A hot key still goes to one reducer under ordinary hash partitioning. For decomposable aggregates, salt the key, compute partial results, and run a second aggregation; for sorting, sample the key distribution and choose range boundaries carefully. Durability usually comes from the underlying filesystem or object store, not from replicating every task's intermediate state. Lost intermediate output can be recomputed from durable input. Storage replication, object-store guarantees, and coordinator recovery behavior depend on the deployment. Task retries do not provide general exactly-once external side effects. Use the framework's supported output committer for file output, and idempotent writes or deduplication for external sinks. Publish a completed dataset version through a manifest or metadata pointer so readers do not observe a partially produced batch. ## Limits and defaults worth knowing - **Reducer count default:** Hadoop MapReduce's documented default for mapreduce.job.reduces is 1. Configure it for the workload; map-only jobs can use 0 reducers. - **Task attempt defaults:** Hadoop MapReduce defaults mapreduce.map.maxattempts and mapreduce.reduce.maxattempts to 4 attempts each. Both are configurable. - **Combiner execution guarantee:** A combiner may run zero, one, or multiple times. The result must remain correct regardless of its execution. - **Ordering guarantee:** Reducer input is sorted by key within its partition. Multiple reducer outputs are not globally ordered with ordinary hash partitioning, and reducer-emitted records are not automatically re-sorted. - **Single-key parallelism:** Under normal grouping semantics, all values for one key are processed by one reduce call. Adding reducers alone does not split that key's work. ## Use it when - The design needs large historical aggregations or periodic reports, and batch completion time is acceptable. - You need to rebuild indexes or derived datasets from durable raw input. - The computation is naturally expressed as independent record transformations followed by grouping by key. - You need distributed external sorting with bounded worker memory. - An existing Hadoop environment makes operational reuse more valuable than adopting a newer processing engine. ## Avoid it when - The result must update continuously with low latency, such as live alerts or real-time counters. - The workload repeatedly reuses the same working set across iterative steps, where disk materialization becomes expensive. - You need interactive queries or synchronous request-path computation. - The job primarily performs non-idempotent external side effects that cannot safely tolerate retries or speculative attempts. ## What interviewers ask about it ### What is likely to dominate the cost of a MapReduce job? Look at input scanning, intermediate data expansion, shuffle traffic, disk spills, and the slowest reducer. Filter early and use combiners where valid, but estimate intermediate bytes rather than assuming input size alone predicts cost. ### How would you handle one extremely popular key? Adding reducers does not help because the key still belongs to one reduce group. If the computation is decomposable, distribute it across salted keys and merge partial results in a second stage; otherwise, discuss a specialized algorithm or the unavoidable bottleneck. ### What happens when a worker fails? The framework retries affected tasks, subject to configured attempt limits. If a worker held local map outputs that reducers still need, those maps may need to run again, while successfully committed durable output is handled by the configured recovery and commit mechanisms. ### Does retrying tasks mean the output is exactly once? No: a task's code can execute more than once. Supported file committers can select successful task output, but arbitrary database writes or API calls need their own idempotency or deduplication strategy. ### How do you produce a globally sorted dataset? Sample the input to select ordered range boundaries, partition by those ranges, and sort within each partition. Ensure the reducer preserves key order and readers consume files in range order; hash partitioning provides only partition-local ordering. ### When is a combiner correct? It must preserve the final answer even if applied repeatedly or not at all, usually through an associative and commutative aggregation with compatible intermediate types. Sum and count work directly, while averages require carrying sum and count rather than just a partial average. ## Common mistakes - Placing MapReduce on a user-facing request path without accounting for scheduling, shuffle, and batch completion latency. - Assuming more reducers solve hot keys or guarantee evenly sized partitions. - Treating combiners as guaranteed steps or using mathematically invalid partial aggregation. - Claiming retries make arbitrary side effects exactly once, or assuming output files become visible as one atomic dataset. ## Alternatives - **Apache Spark:** Choose it for multi-stage batch pipelines, iterative algorithms, and workloads that benefit from reusing data in memory. It still requires explicit handling of skew, shuffle cost, and sink semantics. - **Apache Flink:** Choose it for continuous, stateful processing with event-time windows and low-latency updates rather than periodic full-dataset jobs. - **Distributed SQL engines:** Choose engines such as Trino or a cloud data warehouse for SQL-oriented analysis where an optimizer and declarative queries reduce custom processing code. ## Interview questions where it comes up - [Design a system to sort huge datasets](https://sysround.com/questions/design-distributed-sort): MapReduce directly illustrates range partitioning, external sorting, shuffle-and-merge processing, and recovery after worker failure. - [Design an ad click aggregator](https://sysround.com/questions/design-ad-click-aggregator): Batch jobs can recompute historical click totals and reconcile live aggregates against durable, deduplicated input. - [Design Google Analytics](https://sysround.com/questions/design-google-analytics): Offline aggregation over retained events supports historical reports and backfills without putting batch processing on the collection path. - [Design Facebook post search](https://sysround.com/questions/design-facebook-post-search): MapReduce can build a baseline inverted index from a large corpus before a separate incremental pipeline maintains freshness. - [Design an A/B testing platform](https://sysround.com/questions/design-ab-testing-platform): Batch processing can join exposure and outcome data and recompute experiment results after attribution or data-quality corrections. ## FAQ ### What is MapReduce? MapReduce is a model for processing large datasets across workers. Map functions emit key-value pairs, the framework groups intermediate values by key, and reduce functions produce results from those groups. ### What is the difference between MapReduce and Hadoop? MapReduce is a programming and execution model, while Hadoop is an ecosystem that includes an implementation of that model. Hadoop MapReduce commonly runs on YARN and uses HDFS or another supported storage system. ### What is the difference between MapReduce and Spark? Classic Hadoop MapReduce organizes processing into map and reduce jobs, often materializing data between jobs. Spark supports more general execution graphs and can cache reused datasets in memory, making it a better fit for many iterative and multi-stage workloads. Both can spill to disk and both can suffer from expensive shuffles and skew. ## Related concepts - [Partitioning strategies](https://sysround.com/concepts/data-partitioning) - [Hot keys and hot partitions](https://sysround.com/concepts/hot-keys) - [Data lakes and warehouses](https://sysround.com/concepts/data-lake) - [Idempotency and idempotency keys](https://sysround.com/concepts/idempotency) - [Inverted index](https://sysround.com/concepts/inverted-index) --- SysRound (https://sysround.com) is an AI practice tool for system design interviews: a voice or typed AI interviewer, a shared whiteboard, then feedback on every skill and a model answer. Its feedback is AI-generated and can be wrong.