# Priority queues in distributed systems > A priority queue selects eligible work by importance rather than arrival time. In distributed systems, the hard part is preserving useful priority behavior across durable storage, multiple workers, retries, and overloaded partitions, not implementing a heap. Source: https://sysround.com/concepts/priority-queue Updated: 2026-11-05 ## How priority queues in distributed systems works Define what priority means before choosing storage. A notification service might put password resets above marketing emails, with FIFO ordering within each class. Specify whether priority applies globally, per tenant, or per queue, and whether it controls dispatch order or a latency target. Separate eligibility from priority. Store fields such as job_id, priority, available_at, enqueue_sequence, and attempt_count. A high-priority retry scheduled for five minutes from now must not block lower-priority work that is ready now; move due jobs into ready queues or maintain an index that supports the eligibility filter. Choose a ready-queue implementation. A heap supports insertion and removal in O(log n) on one node, but supplies neither durability nor distributed ownership. For a few priority classes, separate durable FIFO queues and a scheduler are often simpler; another option is a database-backed queue with indexed selection and atomic claims. Claim work atomically and lease it to a worker. The job remains recoverable until the worker acknowledges completion; an expired lease makes it eligible for another attempt. This usually means at-least-once execution, so workers need idempotent effects or downstream deduplication. Choose how workers share capacity. Always selecting the highest class can starve background work, while weighted scheduling gives each class some service. Reserve worker capacity for urgent traffic when needed, and limit prefetch: an urgent job cannot jump ahead of hundreds of low-priority jobs already buffered by workers. Make the scope of ordering explicit when partitioning. Independent shard schedulers provide local priority, not a globally highest-priority claim. Strict global selection requires coordinated selection and claims across partitions; many designs instead accept approximate ordering and enforce per-class latency targets. ## Worked example: Keeping password resets ahead of marketing email Suppose an email service has 100 workers, each sending 10 messages per second. A campaign adds 500,000 marketing messages while password resets arrive at 100 per second. One FIFO queue could leave resets waiting behind roughly 500 seconds of campaign work. Split ready work into transactional and marketing queues. Reserve 20 workers for transactional messages, giving that class a nominal capacity of 200 sends per second, and let them borrow marketing work only when idle with minimal prefetch. The remaining 80 workers drain the campaign. Both queues use leases and idempotent send tracking. This protects reset dispatch under these assumptions, but provider throttling or higher send latency can still violate the target; monitor queue age and provider limits. ## Trade-offs - **Strict priority versus starvation.** Continuous urgent traffic can indefinitely delay ordinary jobs, so use weighted scheduling, aging, or reserved capacity when lower classes need progress. - **Global ordering versus scale.** Selecting the highest-priority eligible job across all shards adds coordination and can bottleneck dispatch. - **Priority versus preemption.** Queue priority normally affects only work not yet claimed, so long-running jobs may require separate worker pools or cooperative cancellation. - **Batching versus responsiveness.** Large dispatch batches and deep worker prefetch improve throughput but delay newly arrived urgent work. - **Durability versus implementation simplicity.** An in-memory heap is simple, but reliable recovery requires durable queue state, safe ownership transitions, and replay or reconstruction. - **Urgency versus tenant fairness.** A tenant that labels everything urgent can monopolize workers unless priority assignment, quotas, and admission control are enforced. ## When to use it in a system design interview - A notification system must keep transactional messages responsive during large marketing campaigns. - A job scheduler has interactive and background workloads competing for the same worker fleet. - A crawler needs to rank ready URLs while respecting independent per-host crawl delays. - An interviewer introduces overload, multiple service tiers, or latency targets that a single FIFO backlog cannot satisfy. ## How to bring it up in the interview - Start with the requirement: which work should wait less, under what load, and within what scope? Distinguish strict dispatch priority from a best-effort latency target. - Propose a small number of priority classes before a custom distributed heap, and explain the scheduling rule between classes. Expect questions about starvation and tenant abuse. - Walk through a worker crash after an external side effect but before acknowledgment. Explain lease expiry, duplicate execution, and the idempotency boundary. - State that priority does not create capacity or interrupt running work. Discuss reserved workers, prefetch limits, provider rate limits, and admission control. - Explain how sharding weakens global ordering, then name the signals you would monitor: oldest ready-job age, enqueue-to-start latency by class, lease expirations, and retry volume. ## Common mistakes - Treating a heap or sorted set as a complete durable distributed queue without explaining atomic claims, leases, acknowledgments, and recovery. - Promising strict global priority while workers independently poll different shards or hold prefetched batches. - Sorting by priority without accounting for available_at, allowing future retries to interfere with ready work. - Using strict priority without a starvation policy, or assuming weighted job counts guarantee fair CPU time when job durations differ. - Assuming high-priority jobs will meet deadlines even when workers or downstream services are saturated. ## Interview questions where it comes up - [Design a notification system](https://sysround.com/questions/design-notification-system): Priority-aware dispatch and capacity isolation keep transactional notifications from waiting behind bulk campaigns or retries. - [Design a web crawler](https://sysround.com/questions/design-web-crawler): The URL frontier must select valuable ready URLs without violating per-host politeness constraints. - [Design a distributed job scheduler](https://sysround.com/questions/design-job-scheduler): Ready-work scheduling must separate execution times from priority and allocate limited worker capacity during bursts. - [Design a distributed message queue like RabbitMQ](https://sysround.com/questions/design-message-queue): Priority-aware work distribution requires explicit claim semantics, fairness, and ordering guarantees across queue partitions. ## FAQ ### How do you implement a distributed priority queue? For a few classes, use durable queues per class and a scheduler that applies strict or weighted priority. Alternatively, keep jobs in a durable store and atomically claim eligible rows using an appropriate index and locking strategy. Either design still needs leases, acknowledgment, retries, and a clear ordering scope. ### How do you prevent starvation in a priority queue? Give lower classes a minimum service share through weighted scheduling or reserved workers. Aging can raise a job's effective priority as it waits, but the implementation must actually update or recompute its scheduling position. Fairness across tenants may require another scheduling layer within each class. ### What is the difference between a priority queue and a delay queue? A delay queue controls when a job becomes eligible; a priority queue chooses which eligible job goes next. A distributed scheduler often needs both. Keep future jobs out of ready selection so a high-priority job with a later execution time does not block work that can run now. ## Related concepts - [Message queues](https://sysround.com/concepts/message-queues) - [Delay queues and scheduled delivery](https://sysround.com/concepts/delay-queue) - [Backpressure](https://sysround.com/concepts/backpressure) - [Delivery guarantees](https://sysround.com/concepts/delivery-guarantees) --- SysRound (https://sysround.com) is an AI practice tool for system design interviews: a voice or typed AI interviewer, a shared whiteboard, then feedback on every skill and a model answer. Its feedback is AI-generated and can be wrong.