# Stream processing > Stream processing continuously transforms incoming events into useful outputs, such as live counts, alerts, or updated recommendations. In system design, it lets you keep derived data fresh without repeatedly scanning an entire dataset, but introduces challenges around ordering, late events, state, and recovery. Source: https://sysround.com/concepts/stream-processing Updated: 2026-11-05 ## How stream processing works Start with a continuous source of events: clicks, metric samples, price changes, or location updates. Producers typically write to a durable partitioned log, and processors read from it independently. For example, a click event might contain an event ID, ad ID, event timestamp, and ingestion timestamp. Partition events by the key whose state must be updated together. Counting clicks per ad usually means grouping by ad ID so one logical owner updates each ad's count. Ordering is generally limited to a partition; it does not guarantee timestamp order when producers send events late or out of order. Apply transformations such as filtering, enrichment, aggregation, or joins. Stateless filters can handle each event independently, while a five-minute click count needs state indexed by ad and time window. Stream joins also need rules for how long to retain unmatched events, or their state can grow without bound. Choose whether calculations use event time, when the event happened, or processing time, when a worker handled it. For event-time windows, a watermark estimates how far event time has progressed; it is not proof that all earlier events arrived. Define whether late events update prior results, go to a correction path, or are excluded under an explicit policy. Recover state and source progress together. A processor can checkpoint its aggregation state and consumed offsets, then restore and replay after a crash. Replayed output can still duplicate external writes, so use idempotent upserts, deduplication, or a compatible transactional sink rather than assuming checkpoints make every side effect exactly once. Publish results to a serving database, another stream, or an alerting service. Track consumer lag, event-to-result latency, state size, and sink failures. If processing falls behind, backpressure limits in-flight work while the log absorbs a bounded backlog; catching up requires spare processing capacity and enough retention. ## Worked example: Live click counts with late-event corrections Suppose an ad analytics service receives 60,000 click events per second and needs counts by ad for one-minute event-time windows, visible within five seconds. Write events to a durable log and partition processing by ad ID. Processors deduplicate click IDs within a defined retention horizon, update window state, and upsert provisional counts using (ad_id, window_start) as the output key. Emit revisions every five seconds and retain windows until the watermark passes their end by two minutes. Later events go to reconciliation rather than silently disappearing. Checkpoint state with source offsets, and version output updates so replay cannot overwrite newer counts. Billing uses reconciled totals, not the provisional dashboard. ## Trade-offs - **Freshness versus efficiency.** Processing each event immediately reduces delay, while buffering small batches can improve throughput at the cost of freshness. - **Timely results versus complete results.** Publishing windows early gives fast answers, but late arrivals require revisions or an explicit distinction between provisional and finalized results. - **Stateful computation versus operational cost.** Windows, deduplication, and joins need bounded retained state, durable recovery, and careful handling when workers or partition ownership change. - **Parallelism versus per-key coordination.** More partitions spread independent keys across workers, but a hot key may require partial aggregation and a second merge stage where the computation permits it. - **Stronger guarantees versus coordination.** Atomic state, offset, and output commits reduce duplicate effects within supported boundaries but add coordination and do not automatically cover external APIs. - **Replayability versus retention cost.** Retaining source events enables recovery and recomputation, but consumes storage and requires a plan to keep replayed outputs from corrupting live results. ## When to use it in a system design interview - The problem asks for continuously updated aggregates, such as clicks per minute, trending items, or metrics rollups. - Alerts or decisions must react to incoming events within seconds rather than waiting for a scheduled batch. - You need to join or correlate events over time, such as associating page views with sessions or outcomes with exposures. - High-volume updates must become queryable derived state without repeatedly scanning raw data. - Late arrivals, duplicate delivery, and replay are central correctness requirements rather than rare edge cases. ## How to bring it up in the interview - Start with the output contract: what result is being maintained, how stale it may be, and whether published values can change. Interviewers will probe what 'real time' and 'correct' actually mean. - Draw the source log, partition key, stateful operators, and serving sink. Explain which operations require repartitioning and where one popular key could become a bottleneck. - State the time model explicitly: event time or processing time, window type, watermark behavior, and late-event policy. Expect questions about idle partitions, incorrect timestamps, and events arriving after window cleanup. - Walk through a crash between updating state and writing output. Explain how checkpoints, replay, and sink idempotency work together, including the limits of any exactly-once claim. - Estimate input rate, per-event work, retained state, and catch-up capacity. Discuss lag, backpressure, and what happens when an outage lasts longer than source retention. ## Common mistakes - Treating any asynchronous queue worker as a complete stream-processing design without explaining continuous computation, state, or time semantics. - Using processing-time windows for business events while assuming late or replayed events will land in their original windows. - Claiming a watermark proves completeness, or retaining window and deduplication state forever to avoid choosing a late-data policy. - Claiming exactly-once behavior because the broker or processor offers it, without checking the output database or external side effect. - Assuming more workers solve every throughput problem while ignoring hot keys, state movement, and a slow output sink. ## Interview questions where it comes up - [Design an ad click aggregator](https://sysround.com/questions/design-ad-click-aggregator): The core pipeline maintains replay-safe windowed click counts while handling late events, hot ads, and billing reconciliation. - [Design a metrics monitoring system like Datadog](https://sysround.com/questions/design-metrics-monitoring-system): Continuous ingestion, rollups, and alert evaluation depend on stateful processing with explicit handling of late samples and backlog. - [Design a top K trending system](https://sysround.com/questions/design-top-k): Maintaining a changing ranking requires continuous aggregation, window expiration, and correct distributed updates under replay. - [Design Google Analytics](https://sysround.com/questions/design-google-analytics): Sessionization, funnel calculations, and distinct-user metrics require processing event streams with duplicates and late arrivals. - [Design Uber surge pricing](https://sysround.com/questions/design-surge-pricing): Fresh regional demand and supply estimates require continuous aggregation with event-time handling and controlled publication of price updates. - [Design a price alert system](https://sysround.com/questions/design-price-alert-system): Incoming price events must be evaluated against registered thresholds with correct ordering and durable, replay-safe triggers. - [Design Facebook likes at scale](https://sysround.com/questions/design-facebook-likes): High-volume like and unlike changes need replay-safe incremental aggregation before publishing fresh totals. - [Design an ETA and live location sharing service](https://sysround.com/questions/design-eta-service): Continuous location updates must be filtered, ordered, and combined with routing information to maintain trustworthy trip progress and ETAs. ## FAQ ### What is the difference between stream processing and batch processing? Batch processing works on a bounded dataset, such as yesterday's events, and finishes a run. Stream processing maintains results as an ongoing, potentially unbounded input arrives. A stream processor may use small internal batches, so the main distinction is continuous computation and incremental results, not necessarily one-event-at-a-time execution. ### Is Kafka a stream-processing system? Kafka's brokers primarily provide a durable, partitioned event log; storing and delivering events is not the same as computing windowed results. Kafka Streams is a processing library, while engines such as Apache Flink can also consume Kafka topics to maintain state and compute results. In an interview, distinguish the transport layer from the processing layer. ### How does stream processing handle duplicate and late events? Duplicates can be handled with stable event IDs and bounded deduplication state, or with operations whose repeated application has the same effect. Late events require event-time windows and an explicit policy for revising or reconciling earlier outputs. Checkpointing supports recovery, but output correctness still depends on coordinating writes or making them replay-safe. ## Related concepts - [Windowing and watermarks](https://sysround.com/concepts/windowing) - [Backpressure](https://sysround.com/concepts/backpressure) - [Exactly-once processing](https://sysround.com/concepts/exactly-once) - [Message queues](https://sysround.com/concepts/message-queues) --- SysRound (https://sysround.com) is an AI practice tool for system design interviews: a voice or typed AI interviewer, a shared whiteboard, then feedback on every skill and a model answer. Its feedback is AI-generated and can be wrong.