How it works
A producer publishes a message describing work, such as sending an email or transcoding an uploaded video. For reliable background work, use a durable queue and wait for the broker's publish acknowledgment before treating the enqueue as successful. Explain what that acknowledgment guarantees: accepting a message into memory is different from persisting it with replication.
Workers consume messages and acknowledge them after completing the required work. Many work queues use a visibility timeout or lease: a delivered message temporarily becomes unavailable to other workers. If the worker crashes without acknowledging, the message becomes eligible for delivery again; long-running tasks may need lease renewal.
Redelivery creates an important failure case: a worker can complete a side effect, then crash before acknowledging the message. Another worker may perform the same action again. Use a stable operation ID with an idempotent downstream API, or atomically record both the operation ID and the local database change; a separate 'already processed' check is not enough.
Retry transient failures with exponential backoff and jitter, and cap attempts or total retry time. Move repeatedly failing messages to a dead-letter queue for investigation and controlled replay. Preserve the message and failure context, make the transfer reliable, and alert on it rather than treating the dead-letter queue as successful completion.
Add workers to increase throughput until another limit, such as database capacity or a provider quota, becomes the bottleneck. Partition work when necessary, using an entity key if related operations need ordering. Parallel workers, retries, and lease expiration can change completion order even when messages were initially delivered in order.
When a database update must cause queued work, use a transactional outbox: commit the business change and an outbox record in one database transaction, then relay that record to the broker. Distinguish distribution models too: competing workers share work from one queue, while independent subscribers need separate queues or consumer groups to each receive the event.
A worked example
Sizing a notification backlog
Suppose a notification service receives 5,000 jobs per second for 60 seconds, while its workers can safely send 2,000 per second within the provider quota. Starting empty, the queue accumulates roughly 180,000 jobs. If arrivals then fall to 500 per second, spare capacity is 1,500 per second, so the backlog drains in about 120 seconds, ignoring retries. Adding workers will not help if the provider quota is already the bottleneck. Put urgent security notifications in a separate queue with reserved sending capacity so a marketing burst cannot delay them. Use stable notification IDs, provider idempotency where available, bounded retries, and oldest-message-age alerts.
Trade-offs to say out loud
- Faster requests, delayed results
- Enqueueing can shorten the request path, but the API must distinguish work accepted from work completed and expose status when users need it.
- Burst absorption, not unlimited capacity
- A queue buffers temporary excess traffic, but sustained arrivals above processing capacity require admission control, more capacity, or an explicit expiration or dropping policy.
- Reliable recovery, duplicate processing
- At-least-once delivery supports recovery from worker failures but requires idempotent side effects because delivery can repeat.
- Ordering limits parallelism
- Strict ordering requires serializing work within the required scope, so a busy entity can bottleneck even when other workers are idle.
- Failure isolation needs deliberate boundaries
- Separate queues and worker budgets can protect critical jobs from slow or bulk workloads, at the cost of more configuration and capacity management.
- Durability and operations cost resources
- Replication, retention, retry traffic, monitoring, and dead-letter replay consume storage, network bandwidth, and engineering effort.
When to reach for it
- A request triggers slow work, such as sending notifications or processing media, that does not need to finish before responding.
- Traffic arrives in bursts, and the system can tolerate bounded waiting while workers process the backlog.
- Work must survive worker crashes or temporary downstream outages.
- Several worker instances should share background jobs without producers choosing a particular worker.
- Bulk and critical workloads need separate concurrency limits, retry policies, or failure boundaries.
How to bring it up in the interview
- Start with the user-visible contract: 'The API returns accepted after durable enqueue; completion happens asynchronously.' Explain how clients learn whether work succeeded.
- Name the failure window: 'The side effect may succeed before the acknowledgment is recorded, so redelivery must be safe.' Expect questions about external APIs that lack idempotency support and how you reconcile uncertain outcomes.
- Estimate arrival rate, processing capacity, and acceptable queue age. Interviewers often ask how long a burst takes to drain and what happens when autoscaling hits a downstream quota.
- Define ordering narrowly, such as per order or per account, rather than requesting global FIFO by default. Be ready to explain concurrent processing, retries, and hot partitions.
- Describe the operating policy: visibility timeout, bounded retries, dead-letter handling, oldest-message age alerts, and admission control. If enqueueing follows a database write, explain the outbox rather than leaving an unsafe dual write.
Common mistakes
- Claiming the queue guarantees exactly-once business effects; broker guarantees alone do not make an external email, payment, or database update happen exactly once.
- Acknowledging before durable completion, which can lose work, or assuming acknowledging afterward eliminates duplicate side effects.
- Treating a queue as an unlimited buffer without storage limits, expiration rules, backlog-age targets, or overload behavior.
- Assuming FIFO delivery guarantees ordered completion across parallel workers and retries.
- Publishing to the broker and updating a database as independent writes without addressing the crash between them.
Where it comes up
Interview questions where message queues is part of a strong answer, and why.
- Design a notification systemDurable queues, channel isolation, retries, and duplicate-safe delivery form the core notification processing path.
- Design a distributed message queue like RabbitMQThe design must implement durable enqueueing, visibility leases, acknowledgments, redelivery, and bounded resource use.
- Design a distributed job schedulerScheduled occurrences need a reliable database-to-queue handoff and workers that tolerate retries and lease expiration.
- Design a video streaming service like YouTubeQueued processing stages coordinate asynchronous transcoding work and recover failed tasks without blocking uploads.
- Design a web crawlerThe URL frontier distributes durable crawl work while enforcing host politeness, retry limits, and bounded concurrency.
- Design a news feed like FacebookAsynchronous fan-out uses queued work to build feeds while handling bursts, retries, and delayed updates.
- Design Google PhotosAsynchronous image processing requires durable jobs, retry-safe workers, and checks that prevent processing from reviving deleted photos.
Questions people ask
What is a message queue in system design?
A message queue is an intermediary that holds messages until consumers process them. It separates when work is requested from when it executes, which helps with background jobs, burst handling, and recovery from temporary failures. Durability and delivery guarantees depend on the queue's configuration and acknowledgment protocol.
What is the difference between a message queue and pub/sub?
In a work queue, competing consumers share jobs, with each job intended for one successful processing outcome even though delivery may repeat. In pub/sub, each independent subscription receives the published event. A subscription can itself use a queue to distribute its work among multiple workers.
Is Kafka a message queue?
Kafka is a partitioned, retained event log that can support queue-like work distribution through consumer groups. Unlike a typical acknowledgment-and-delete work queue, it retains records according to policy and tracks consumer positions separately, enabling replay and independent readers. Choose based on replay, routing, retry, ordering, and retention needs rather than treating the products as interchangeable.