How it works
Choose which clock defines the result. Event time is when an action happened; processing time is when a worker handles it. A click timestamped 12:00:45 that arrives at 12:03 belongs to the 12:00 minute under event-time processing, but the 12:03 minute under processing-time processing. Event time preserves historical meaning across delays and replay, assuming timestamps are trustworthy.
Choose the window shape. Tumbling windows are fixed and nonoverlapping: [12:00, 12:01), then [12:01, 12:02). Sliding windows overlap: a five-minute window evaluated every minute includes an event in five windows. The start is inclusive and the end exclusive, so an event exactly at 12:01 belongs to the next tumbling window.
Use session windows when activity, rather than fixed boundaries, defines the group. With a 30-minute inactivity gap, nearby events form one session. A late event can bridge two previously separate sessions, requiring their aggregates to merge and earlier outputs to be corrected.
Track a watermark as an estimate of event-time progress, not proof that earlier events cannot arrive. One strategy advances each source partition's watermark to its maximum observed event timestamp minus an out-of-order allowance. A downstream operator generally uses the minimum watermark of its active inputs, so a slow partition can hold back every window it contributes to.
Separate producing a result from discarding its state. A common policy emits a window result when the watermark reaches its end, then retains state for an additional allowed-lateness period. Accepted late events can update the result; events arriving after cleanup must be dropped, routed for separate handling, or incorporated through a backfill. Exact trigger and cleanup semantics depend on the stream processor.
Maintain aggregates by key and window, such as (ad_id, minute). Checkpoint window state and source progress consistently, and publish updates using stable keys and versions or another retry-safe scheme. Monitor watermark lag, late-event rates, and state size. Explicitly handle idle inputs so an inactive partition does not block progress forever.
A worked example
Minute-level ad clicks with late corrections
An ad dashboard groups clicks into event-time tumbling windows of one minute. For ad A, [12:00, 12:01) contains 1,000 deduplicated clicks when the operator watermark reaches 12:01, so it publishes a provisional total. The system retains that window until the watermark reaches 12:03. A click timestamped 12:00:40 arrives while the watermark is 12:01:20; the processor accepts it and upserts the total to 1,001 using the same ad-and-minute key. After cleanup, another click for that minute goes to a late-event stream rather than silently changing the dashboard. A reconciliation job later incorporates such events into billing totals.
Trade-offs to say out loud
- Freshness versus completeness
- A more conservative watermark accommodates more out-of-order data before emission but delays watermark-driven results.
- Late corrections versus state cost
- Longer allowed lateness supports more corrections but retains more window state and increases recovery work.
- Sliding-window precision versus computation
- Overlapping windows can multiply updates, although mergeable aggregates such as counts and sums can reuse smaller time buckets.
- Event-time meaning versus timestamp quality
- Event-time results reflect when actions occurred, but clock skew and malformed future timestamps can corrupt assignments or advance heuristic watermarks prematurely.
- Progress versus idle-input completeness
- Excluding idle partitions lets watermarks advance, but records from a resumed partition may then be classified as late.
- Early output versus downstream complexity
- Provisional results reduce visible latency but require consumers to handle updates or retractions instead of treating every output as a new fact.
When to reach for it
- The requirement asks for per-minute totals, rolling-hour rankings, time-bounded joins, or sessions over a continuous stream.
- Events arrive out of order because of mobile buffering, network delays, retries, or recovery after an outage.
- A live dashboard needs fast provisional numbers while reporting or billing needs later reconciliation.
- The design must bound aggregation state without silently assuming that all events arrive on time.
- Replaying the same event history should preserve which time intervals events belong to.
How to bring it up in the interview
- Start with the product definition: “Does clicks per minute mean when the click happened or when we received it?” Then specify the key, window width, slide, and boundary convention.
- State the lateness contract explicitly: when results first appear, how long they can change, and what happens after state cleanup. Interviewers often ask whether a watermark means a count is final.
- Explain how watermarks combine across partitions and how idle inputs are detected. Be ready to discuss a stalled source, a resumed source, and a timestamp far in the future.
- Estimate state from active keys, retained windows, and aggregate size. For sliding windows, explain whether you store overlapping aggregates or reuse smaller buckets.
- Distinguish time handling from delivery correctness. Watermarks do not remove duplicates; interviewers may probe event IDs, checkpoint recovery, and retry-safe writes of corrected results.
Common mistakes
- Using processing time without acknowledging that ingestion delays and replay can move events into different reporting windows.
- Calling a watermark a guarantee that no earlier event will arrive, rather than a progress estimate governed by a source policy.
- Treating the watermark delay and the allowed-lateness retention period as the same setting.
- Emitting corrected totals as additive increments, causing downstream consumers to double-count previous results.
- Ignoring idle partitions, clock skew, or the memory cost of many keys and overlapping windows.
Where it comes up
Interview questions where windowing and watermarks is part of a strong answer, and why.
- Design an ad click aggregatorEvent-time windows and explicit late-arrival policies separate fast live counts from reconciled billing totals.
- Design a metrics monitoring system like DatadogWindow boundaries, late samples, and watermark progress determine rollup accuracy and whether alerts reflect unhealthy services or delayed data.
- Design a top K trending systemSliding-window expiration and late-event corrections change item scores and therefore the correctness of continuously updated rankings.
- Design Google AnalyticsSession windows and event-time handling are central to reconstructing user activity when events arrive late or out of order.
- Design Uber surge pricingRecent demand and supply measurements need explicit time windows and freshness policies so delayed events do not distort current prices.
- Design an A/B testing platformTime-bounded exposure-to-outcome attribution must account for delayed events and define when results remain open to correction.
Questions people ask
What is the difference between tumbling and sliding windows?
Tumbling windows divide time into fixed, nonoverlapping intervals, so each event belongs to one window. Sliding windows have a width and an evaluation interval; when the interval is smaller than the width, windows overlap and an event contributes to multiple results. Use tumbling windows for minute-by-minute reports and sliding windows for questions such as “what is trending over the last hour, refreshed every minute?”
What is a watermark in stream processing?
A watermark represents estimated progress through event time. It allows a processor to trigger window results and, under its retention policy, clean up old state. It is not a wall-clock timer or a guarantee that older events cannot still arrive.
What happens to events that arrive after the watermark?
An event with a timestamp behind the current watermark is late, but it is not necessarily discarded. If its window still has retained state and the policy permits updates, it can produce a correction. After cleanup, the system needs an explicit policy such as dropping with metrics, routing to a late-event stream, or reconciling through batch processing.