How it works
Start with two required effects: save a notification request in a database and publish a job to a message queue. A normal database transaction can protect the database write, but it does not automatically include the queue publication.
Writing to the database first creates a failure window: the commit succeeds, then the process crashes before publishing. Publishing first reverses the problem: a worker may receive a job for a database record that never commits. Running both writes in parallel does not make them atomic.
Timeouts add uncertainty. A broker may accept a message even though its acknowledgment never reaches the producer. Retrying can create a duplicate, while not retrying can lose a message if the original attempt actually failed. An in-memory retry loop does not survive a process crash.
The transactional outbox pattern replaces the independent writes with one local transaction: save the business record and an outbox event together. A relay publishes committed outbox events and records successful publication afterward. If it crashes between publishing and recording success, it publishes again, so consumers must tolerate duplicates.
Change data capture can derive downstream updates from committed database changes or relay an outbox through the database log. This avoids a separate application-level publish step, but requires durable capture progress, sufficient log retention, and a recovery plan for outages. Derived stores still lag behind the source.
If both resources support a shared distributed transaction protocol, two-phase commit can provide atomic commit across them, with coordination and availability costs. Otherwise, choose an authoritative record and a durable propagation or reconciliation mechanism. A saga can coordinate compensating business actions, but does not make two independent writes atomic.
A worked example
A notification request survives a queue outage
A notification API receives request N42. In one database transaction, it inserts notification N42 with status pending and outbox event E42. The transaction commits, so the API can acknowledge durable acceptance even if the queue is unavailable. After a 30-second outage, the relay publishes E42. The broker accepts it, but the relay crashes before marking it published. On restart, the relay publishes E42 again. The consumer uses E42 to deduplicate creation of its durable delivery job in the same transaction. Sending through an external email provider remains a separate uncertainty: a timeout may require a provider idempotency key or reconciliation, not just local event deduplication.
Trade-offs to say out loud
- Atomic intent, delayed delivery
- An outbox atomically records the state change and its publication intent, but downstream systems can remain stale while the relay is delayed.
- Duplicates instead of silent loss
- Retrying an uncertain publication favors eventual delivery, but requires stable event IDs and retry-safe consumer effects.
- Additional operational work
- Outbox relays and CDC pipelines need monitoring, backlog capacity, retention policies, cleanup, and recovery procedures.
- Ordering needs explicit design
- Concurrent relays and retries can reorder updates, so use appropriate partitioning, sequencing, or version checks when order matters.
- Stronger atomicity has a price
- Two-phase commit requires compatible participants and coordination, and failures can leave participants waiting for a transaction decision.
When to reach for it
- A request must commit database state and enqueue work that must not disappear.
- Database changes must reliably update a search index, feed, cache, or other derived view.
- A worker must update local state and publish another event before acknowledging its input.
- A migration proposes application writes to both the old and new databases.
- A workflow combines local state changes with an external side effect, such as a payment or notification.
How to bring it up in the interview
- Name the invariant first: for example, every committed notification request must eventually become eligible for delivery. Identify which system owns the authoritative state.
- Walk through a crash between the two writes and then reverse their order. Interviewers want to see why reordering calls or adding retries does not close the gap.
- Draw the local transaction boundary around the business row and outbox row. Explain when the API acknowledges success and what remains asynchronous afterward.
- Expect probes about relay crashes, duplicate events, and ordering. Explain how consumer deduplication and its business update share a transaction when possible, and how external effects need separate handling.
- Describe recovery, not just the happy path: monitor the oldest unpublished event, preserve replay data, and reconcile downstream state against the source when necessary.
Common mistakes
- Assuming two successful-looking API calls are equivalent to an atomic transaction.
- Claiming that database-first or queue-first ordering eliminates partial failures.
- Treating an in-memory retry or a best-effort rollback as durable recovery.
- Claiming that an outbox guarantees exactly-once external effects; publication and processing can still repeat.
- Assuming CDC solves every downstream issue without addressing retention gaps, duplicates, schema changes, or stale updates.
Where it comes up
Interview questions where the dual-write problem is part of a strong answer, and why.
- Design a notification systemPersisting a notification request and reliably enqueueing delivery work requires a crash-safe database-to-queue handoff.
- Design a news feed like FacebookA committed post must reliably trigger asynchronous feed fan-out without losing updates between the database and queue.
- Design Facebook post searchCommitted post changes must reach the search index without missing updates or resurrecting deleted content.
- Design a distributed job schedulerDurable job occurrences must reach the execution queue even when dispatchers crash between database and broker operations.
- Design Facebook MarketplaceListing changes must propagate reliably from authoritative database state to searchable views.
- Design Facebook likes at scaleAuthoritative like and unlike changes must reliably feed asynchronous aggregation without lost or duplicate count effects.
- Design a wire transfer systemCommitting transfer state and recording durable outbound work must be atomic locally while external bank outcomes remain separately reconciled.
Questions people ask
What is the dual-write problem?
It is the risk that one logical change updates one system but fails to update another because the writes are not atomic together. A common example is committing a database record and then failing to publish its corresponding event.
How does the outbox pattern solve dual writes?
It saves the business change and an event record in the same database transaction, then publishes that event asynchronously. This removes the gap between committing the change and durably recording the need to publish. The relay can still publish duplicates, so consumers need retry-safe processing.
Do retries or Kafka transactions solve the dual-write problem?
Retries alone do not preserve unfinished work across crashes and can repeat effects after ambiguous timeouts. Kafka transactions can atomically commit supported operations within Kafka, such as output records and consumed offsets, but do not automatically include an external database write. That boundary still needs an outbox, CDC, or another suitable coordination mechanism.