How it works
Separate delivery, execution, and committed effects. A queue can deliver an event twice, and a worker can execute its code twice, while the database records the intended change only once. Exactly-once semantics usually describe those committed effects, not a promise that code runs only once.
Start with the crash window. If a worker updates a database and then crashes before acknowledging the message, the queue redelivers it. Acknowledging first creates the opposite risk: a crash can lose the update. Moving the acknowledgment alone cannot solve both problems.
For a database-backed consumer, give each logical event a stable ID. In one database transaction, insert a processed-event record protected by a unique constraint and apply the business update. Only apply the update if the insert succeeds, and acknowledge after commit. A retry finds the existing record and skips the update; concurrent duplicates are resolved by the database constraint.
For stream processing, coordinate input progress with state and output. A transactional consume-transform-produce pipeline can commit output records and consumed offsets together, with downstream readers ignoring aborted transactions. Stateful processors can instead coordinate checkpoints with transactional or idempotent sinks. Recovery may repeat computation without publishing its effects twice.
External side effects need their own protection. A stream transaction does not make a payment API call or email send exactly once. Use a stable provider idempotency key when supported, persist the operation state, and reconcile ambiguous outcomes. Without cooperation from the destination, you generally cannot guarantee both no lost effects and no duplicate effects after an uncertain response.
A worked example
Counting an ad click once after a worker crash
An ad has 12,000 billable clicks. Event click-847 should increase the count to 12,001, but its queue delivery can repeat. The consumer begins a database transaction, inserts (billing-consumer, click-847) into a table with a unique constraint, increments the ad's counter, and commits. It then crashes before acknowledging the message. On redelivery, the insert conflicts, so the consumer skips the increment and acknowledges; the count remains 12,001. If the first transaction had rolled back, neither the marker nor the increment would exist, and the retry would apply both. This guarantee covers that database update, not a separate invoice API call. Deduplication records must remain available for the supported replay horizon.
Trade-offs to say out loud
- The guarantee has a boundary
- Exactly-once behavior inside a log or database does not automatically extend to downstream databases, APIs, or physical actions.
- Atomic commits add coordination
- Transactions and coordinated checkpoints consume resources and can increase latency or delay progress during failures.
- Deduplication state needs retention
- Deleting processed-event records makes old duplicates eligible for processing again unless another durable business constraint prevents it.
- Identity defines correctness
- Retries must reuse the same logical operation ID, while genuinely separate operations must receive different IDs.
- Safety is not completion
- Preventing duplicate effects does not ensure every accepted operation finishes, so durable retries, failure handling, and reconciliation still matter.
- Replay requires an explicit policy
- Rebuilding a derived view may require a new processing namespace or output version, while replaying payments must not create new charges.
When to reach for it
- A retried payment, transfer, or job could create an expensive or irreversible duplicate side effect.
- Event aggregates must remain correct across redelivery, worker crashes, checkpoint recovery, and historical replay.
- A consumer updates durable state and the interviewer asks what happens if it crashes before acknowledging its message.
- A design claims exactly-once delivery, and you need to distinguish broker guarantees from application-level correctness.
How to bring it up in the interview
- Define the invariant and scope first: for example, one ledger transfer per customer operation ID, rather than saying the entire system is exactly once.
- Walk through crashes before commit, after commit but before acknowledgment, and during concurrent redelivery; explain what recovery reads in each case.
- Name the mechanism that closes the gap: a unique constraint plus a database transaction, transactional output and offsets, or coordinated checkpoints with a compatible sink.
- Expect probes about deduplication retention, reused IDs, changed request payloads, rebalances, and replay; explain how identity and ownership remain stable.
- At an external boundary, state what the provider guarantees and how you resolve unknown outcomes instead of treating a timeout as proof of failure.
Common mistakes
- Equating exactly-once delivery with exactly-once business effects, or claiming a broker can prevent every duplicate execution.
- Checking whether an event was processed and then updating state in separate operations, leaving a race or crash window.
- Recording an event as processed before its effect commits, which can permanently skip unfinished work.
- Assuming a transactional outbox eliminates duplicates; it closes the database-to-message handoff gap, but publishing and consumption can still repeat.
- Using short-lived deduplication records while promising correctness for arbitrary historical replay.
Where it comes up
Interview questions where exactly-once processing is part of a strong answer, and why.
- Design an ad click aggregatorBillable click counts must survive duplicate delivery, crashes, and replay without counting the same logical click twice.
- Design a payment systemAmbiguous charge outcomes require scoped idempotency and reconciliation rather than blind retries.
- Design a wire transfer systemOne committed transfer per operation ID must be distinguished from uncertain execution across another bank's boundary.
- Design a distributed message queue like RabbitMQConsumer crashes and expiring visibility leases expose why queue delivery alone cannot guarantee exactly-once side effects.
- Design a distributed message log like KafkaConsumer progress must be coordinated with processing results to avoid losing work or duplicating effects after recovery.
- Design Google AnalyticsRetry-safe collection and processing require stable event identity and replay-safe aggregation.
- Design a distributed job schedulerWorker retries and lease expiration require a clear distinction between repeated execution and duplicate business effects.
- Design a notification systemProvider timeouts and duplicate queue deliveries force an explicit boundary between durable notification intent and external delivery.
Questions people ask
What is the difference between exactly-once delivery and exactly-once processing?
Delivery describes messages reaching consumers; processing semantics describe their committed effects. Messages may be delivered and executed repeatedly while transactional updates or idempotency ensure one visible effect per logical event.
Does Kafka guarantee exactly-once processing?
Kafka transactions can atomically commit consumed offsets and output records within Kafka, and read-committed consumers ignore aborted output. Kafka Streams also supports exactly-once processing for its managed state and Kafka outputs. These guarantees do not automatically cover writes to an external database or calls to a payment provider.
Is exactly-once processing the same as idempotency?
No. Idempotency means repeating an operation has the same effect as performing it once, making retries safe. Combined with durable at-least-once processing and appropriate atomicity, it can provide exactly-once business effects within a specified scope.