Side by side
| Dimension | Latency | Throughput |
|---|---|---|
| Meaning | Latency: time for one operation | Throughput: completed work per unit of time |
| Typical units | Milliseconds or seconds | Requests/s, events/s, or bytes/s |
| Useful measurements | p50, p95, p99, and mean | Sustained successful completion rate |
| Primary concern | How long a caller waits | Whether processing keeps up with demand |
| Common improvements | Caching, fewer round trips, shorter queues | Batching, parallelism, partitioning |
| Near saturation | Queueing can sharply increase latency | Completion rate approaches a capacity limit |
| Batching trade-off | Waiting to fill a batch can add delay | Amortizes fixed processing costs |
| Typical objective | Keep p99 below a response-time budget | Sustain peak demand with headroom |
The differences that matter
They measure different things, not competing architectures. A service can complete many requests per second while each request takes a long time because many requests run concurrently. Conversely, a single-worker service can answer individual requests quickly but support little total traffic.
Define the measurement boundary before comparing numbers. API acknowledgment latency is not the same as end-to-end job completion latency. A queue can make an ingestion endpoint respond quickly while downstream processing falls minutes behind; measure both accepted events and completed events.
Queueing connects the two near saturation. When arrivals exceed processing capacity, backlog grows unless work is rejected or dropped. Adding concurrency helps only while the bottleneck has spare capacity; beyond that, extra requests can increase contention and tail latency without increasing useful throughput.
Batching trades waiting time for efficiency when it amortizes fixed costs such as network calls or disk flushes. Parallelism and caching can improve both metrics, so the trade-off is not inevitable. Check whether an optimization moves the bottleneck to the database, network, or another dependency.
For a stable system, Little’s Law relates averages: average in-flight work equals average completion rate multiplied by average time in the system. At 2,000 requests/s and 100 ms mean latency, average concurrency is 200 requests. Do not substitute p99 latency into this average-based equation.
Choose Latency when
- Prioritize latency for interactive search, redirects, checkout, or other operations where the user is waiting.
- Prioritize latency for synchronous dependency calls that must fit within a larger end-to-end deadline.
- Prioritize latency for live updates where delayed delivery makes otherwise correct results less useful.
- Prioritize tail latency when a request fans out to many dependencies and slow responses can delay the whole operation.
Choose Throughput when
- Prioritize throughput for bulk imports, migrations, transcoding, and other jobs with completion windows rather than per-item response deadlines.
- Prioritize throughput for event pipelines that must keep up with sustained input and drain backlogs after outages.
- Prioritize throughput when batching can reduce per-item cost and the added waiting time stays within the freshness budget.
- Prioritize throughput for large transfers where total bytes delivered within a deadline matters more than individual operation latency.
A worked example
A checkout service meets capacity but misses its latency goal
Suppose a checkout service completes 2,000 requests/s with 100 ms mean latency and a 250 ms p99 target. Little’s Law gives 200 requests in flight on average. During a sale, arrivals rise to 4,000 requests/s, but the database limits completions to 2,500 requests/s. If every request is queued and that limit holds, backlog grows by 1,500 requests each second. After 10 seconds, roughly 15,000 requests are waiting, representing about six seconds of processing at the bottleneck. More API servers will not fix that database limit. Protect checkout with bounded admission and load shedding, then address database contention or capacity. Track successful checkout throughput alongside tail latency and rejection rate.
In the interview
- State both requirements, such as a hypothetical p99 response target and a peak successful requests-per-second target. Clarify payload size, request mix, and whether errors count toward throughput.
- Separate offered load, accepted load, and completed work. An interviewer may ask whether a fast acknowledgment hides an unbounded queue or whether retries inflate request counts.
- Identify the bottleneck before proposing more workers. Explain whether CPU, storage I/O, network capacity, locks, or a downstream service limits useful throughput.
- Use Little’s Law for average concurrency estimates, then discuss tail latency and capacity headroom separately. Expect questions about bursts, dependency slowdowns, and overload behavior.
- Describe how you would validate the design with a load test: increase offered load, track successful throughput and latency percentiles, and observe queue depth, errors, and resource saturation.
Common mistakes
- Treating throughput as the inverse of latency. That relationship only fits narrow serial-processing assumptions and ignores concurrency.
- Reporting average latency alone, or quoting throughput without the payload size, workload mix, error rate, and latency achieved at that load.
- Claiming that queues remove latency. They decouple stages but add waiting time and need bounded capacity or an explicit backlog policy.
- Assuming more threads or larger batches always help. Contention can reduce useful throughput, and batch-fill delays can violate latency targets.
Where it comes up
Interview questions where choosing between Latency and Throughput comes up.
- Design a URL shortener like BitlyRedirect latency and sustained redirect throughput require separate targets, especially when hot keys concentrate traffic.
- Design an ad click aggregatorBatching can improve event-processing throughput while increasing the delay before aggregates become queryable.
- Design a distributed message queue like RabbitMQProducer acknowledgments, consumer throughput, and end-to-end processing latency must be measured separately to expose backlog growth.
- Design a ticket booking system like TicketmasterFlash-sale admission control must protect booking latency when inventory contention limits successful transaction throughput.
- Design a large data migration to the cloudBulk-copy throughput determines migration duration, while rate limits protect the latency of ongoing production traffic.
Questions people ask
What is the relationship between latency and throughput?
Latency is time per operation, while throughput is completed work per unit of time. In a stable system, average concurrency equals throughput multiplied by average latency. Throughput is not generally the inverse of latency because operations can overlap.
Does increasing throughput reduce latency?
Increasing processing capacity can reduce queueing latency if it relieves the actual bottleneck. Increasing offered traffic is different: near saturation, latency can rise sharply while completed throughput barely changes. Batching may improve throughput while adding per-item waiting time.
What is the difference between throughput and bandwidth?
Bandwidth usually describes a link’s data-carrying capacity, while throughput describes the transfer rate actually achieved. Protocol overhead, packet loss, congestion, and application limits can keep throughput below available bandwidth. Application throughput can also be measured in requests or events rather than bytes.