# Design a batch processing system like MapReduce: system design interview question

> Large dataset execution: task placement, data movement, uneven partitions, and recovery after worker or coordinator failures.

Source: https://sysround.com/questions/design-batch-processing-system-like-mapreduce
Updated: 2026-11-05


Difficulty: hard. Levels: Senior, Staff+.

## How the interviewer opens it

"Design a distributed engine that processes datasets larger than one machine can handle. It must distribute work across workers and recover from failures without restarting entire jobs."

## What to clarify first

- Job interface
- Job volume and size
- Worker capacity
- Read and write mix
- Latency targets
- Shared capacity
- Correctness and availability
- Storage and durability
- Retention
- Failure scope and exclusions

## Scope of a strong answer

Core requirements:

- Accept jobs that describe a graph of data operations.
- Distribute tasks across workers with finite resources.
- Recover unfinished work after worker or coordinator failures.
- Publish complete job results without partial output.

Usually out of scope:

- A SQL parser or query optimizer
- Stream processing
- Tasks with external side effects
- A new storage service or multi-region recovery

## What interviewers probe

### Data between stages

A job can move as much data between stages as it reads. Uneven keys and excessive file counts can dominate completion time.

### Worker failures and visible results

Workers can finish after the coordinator declares them lost. Duplicate attempts must not expose duplicate records or partial results.

### Job control after a failure

A coordinator must recover accepted work without trusting stale workers. Two coordinators must not make conflicting decisions during a network partition.

### Shared capacity and slow tasks

Twenty jobs share finite CPU, memory, disk, and network capacity. One large job or slow task can delay many otherwise healthy jobs.

## Common mistakes

- Treating the engine as a task queue without explaining data dependencies or intermediate outputs.
- Estimating CPU capacity but ignoring shuffle traffic, disk limits, and memory use.
- Restarting the entire job after any worker failure.
- Promising exactly-once execution instead of defining which result becomes authoritative.
- Using heartbeat expiry as proof that a worker cannot report a late result.
- Publishing output files individually without a complete-result visibility rule.

## The bar by level

- **Senior:** Quantifies data movement and task counts. Handles disk spill, skew, durable intermediate data, coordinator recovery, stale attempts, and atomic result publication.
- **Staff+:** Defines ownership and publication invariants across failures. Connects resource shares and spare capacity to recovery targets. Explains storage costs and control limits.

Practice this question with an AI interviewer: https://sysround.com/practice/new?problem=distributed-batch-processing


---
SysRound (https://sysround.com) is an AI practice tool for system design interviews. It has a voice or typed AI interviewer and a shared whiteboard. At the end, you get feedback on each skill and a model answer. The AI feedback can be wrong.