Side by side
| Dimension | Batch processing | Stream processing |
|---|---|---|
| Input model | Bounded dataset or snapshot | Continuously arriving events |
| Result freshness | Schedule delay plus job runtime | Processing delay plus any windowing or finalization delay |
| Computation | Scans, joins, and aggregates over finite input | Incremental updates, often with keyed state |
| Late data | Include in a later run or recompute | Use lateness policies and result corrections |
| Failure recovery | Retry tasks or rerun bounded input | Restore state and replay from saved positions |
| Capacity | Provision to finish within a deadline | Provision to keep lag within a freshness target |
| Cost drivers | Data scanned, shuffle, frequency, and runtime | Continuous capacity, state, checkpoints, and retention |
| Typical uses | Historical reports, bulk transforms, backfills | Live metrics, alerts, continuously updated rankings |
The differences that matter
Freshness is the first design constraint. An hourly batch job does not guarantee one-hour freshness: an event can wait almost an hour before the job starts, then wait for execution and publication. A streaming pipeline reduces scheduling delay, but queue lag, window finalization, and slow sinks still affect when users see results.
Batch has a natural input boundary, which makes full-dataset joins and recomputation easier to reason about. Streaming joins and aggregations often retain state across events. You must define how that state is partitioned, how long it lives, and what happens when a popular key overloads one worker.
Neither model automatically solves duplicate or late events. A batch result is complete only relative to its chosen input snapshot or cutoff. Streaming needs an explicit event-time policy, often using watermarks and allowed lateness, to decide when to emit provisional results, finalize windows, or apply corrections.
Recovery and capacity planning differ. Batch jobs need enough resources to finish before a deadline, and reruns must publish safely. Streaming must preserve state and input progress across failures while preventing an ever-growing backlog. Streaming is not inherently more expensive: incremental computation can avoid repeated scans, but continuous capacity and state management add operational cost.
Choose Batch processing when
- Reports are consumed daily or hourly, and the schedule plus worst-case runtime meets the freshness target.
- You need large historical joins, global sorting, or a full rebuild over a bounded dataset.
- Work arrives in periodic files or snapshots, and there is no product requirement to process each record immediately.
- You need backfills or reconciliation against an authoritative snapshot, with repeatable inputs and versioned output.
Choose Stream processing when
- Alerts, rankings, or dashboards need updates within seconds rather than at the next scheduled run.
- The application must react to events promptly, such as evaluating a price threshold or detecting a service failure.
- Results can be updated incrementally, avoiding repeated scans of a large historical dataset.
- Continuous ingestion would create too much delay or too large a workload if deferred until a scheduled batch.
A worked example
Live ad counts versus billing totals
Suppose an ad platform receives 10,000 click events per second. Advertisers need dashboard counts within five seconds, while billing requires reconciled daily totals. An hourly batch cannot meet the dashboard target, so a streaming pipeline deduplicates clicks by event ID, updates per-ad windowed counts, and publishes provisional results. Define how long deduplication state is retained and how clicks arriving two hours late update old windows. Store raw events durably so a daily batch can recompute totals using a declared cutoff and the same counting rules. Publish reconciled totals as a new version rather than adding them to the live counts. If daily reporting were the only requirement, the streaming pipeline would be unnecessary.
In the interview
- State the freshness target before naming technology. Distinguish event-to-visible latency from API response latency, and say whether results may be provisional.
- Estimate average and peak event rates, bytes per event, key cardinality, and retained state. For batch, estimate the completion deadline; for streaming, explain how much lag is acceptable and how recovery catches up.
- Define correctness using a concrete duplicate and a late event. Expect probes about event time, window boundaries, deduplication retention, and whether a late correction changes a previously published result.
- Walk through a worker failure and a sink timeout. Explain input replay, checkpoint recovery, and idempotent output; do not claim end-to-end exactly-once behavior without covering the destination.
- If proposing both models, explain how they share transformation rules and how reconciled results replace provisional ones without double counting. Two pipelines introduce extra testing and operational work.
Common mistakes
- Calling streaming instantaneous or batch inherently slow. Actual freshness depends on scheduling, runtime, queue lag, windowing, and publication.
- Assuming batch produces correct totals automatically or streaming provides exactly-once effects automatically. Both need explicit input boundaries, retry behavior, and safe output writes.
- Keeping streaming state forever without estimating its size or defining expiration and late-event behavior.
- Adding separate batch and streaming pipelines by default, without showing why one execution model cannot meet the requirements.
Where it comes up
Interview questions where choosing between Batch processing and Stream processing comes up.
- Design an ad click aggregatorFresh click counts and reconciled billing totals expose the trade-offs around incremental aggregation, late events, and bounded recomputation.
- Design a metrics monitoring system like DatadogAlerts require timely stream processing, while historical rollups and recomputation can use batch jobs.
- Design Google AnalyticsLive dashboards and historical reports need different freshness guarantees while sharing retry-safe event counting and late-data policies.
- Design a top K trending systemContinuously changing rankings require incremental window maintenance, while periodic batch rankings are sufficient only when stale results are acceptable.
- Design a system to sort huge datasetsGlobal sorting over a bounded dataset illustrates why some workloads naturally favor batch execution rather than continuous updates.
Questions people ask
What is the main difference between batch and stream processing?
Batch processing operates on a bounded dataset and finishes that computation. Stream processing continuously consumes events and updates results. The practical choice depends on required freshness, computation shape, and the complexity of maintaining incremental state.
Is micro-batching batch processing or stream processing?
Micro-batching processes small bounded groups of events repeatedly and can implement a streaming system. Its freshness includes the trigger interval, processing time, and output publication delay. Evaluate whether it meets the latency and correctness requirements rather than treating the label as decisive.
Can batch and stream processing be used together?
Yes: streaming can serve fresh provisional results while batch handles historical backfills or reconciliation. Use shared counting rules, explicit cutoffs, and versioned or replacement outputs so the two paths do not double count. Only accept the extra complexity when the requirements justify it.