Load shedding for system design interviews

How it works

Start with the bottleneck and its safe operating range. Load tests can establish how much concurrent work a service handles before latency rises sharply. During operation, watch in-flight requests, queue wait, memory, and dependency saturation; CPU alone misses bottlenecks such as exhausted database connections.

Make admission decisions before expensive work begins. An API gateway can reject excess traffic cheaply, while each service enforces local limits because a gateway cannot see every overloaded instance or dependency. For example, a booking service might allow only 200 concurrent inventory operations per instance.

Bound both active work and waiting work. Once concurrency slots and a small queue are full, reject new requests rather than grow an unbounded backlog. Expire queued requests whose deadlines have passed, and propagate cancellation where possible so abandoned requests stop consuming resources.

Choose which work to shed. Reserve capacity for critical operations, apply per-tenant limits, and reject lower-priority work first. During a ticket sale, existing reservation confirmations may deserve capacity ahead of new browsing requests; separate resource pools help enforce that priority.

Make the outcome explicit. HTTP APIs commonly use 503 for temporary service overload and 429 for a caller exceeding a quota or rate limit. Provide Retry-After when meaningful, and require bounded retries with backoff and jitter so rejected requests do not immediately return as a larger traffic wave.

Recover cautiously and measure the result. Fixed limits are simple; adaptive limits can reduce admission as queue delay or latency rises, but need damping to avoid oscillation. Track offered, admitted, rejected, and successfully completed work separately, and restore capacity gradually as the bottleneck recovers.

A worked example

Protecting ticket inventory during a flash sale

Suppose load tests show an inventory database can sustain 2,000 seat-hold attempts per second within the latency target, but launch traffic reaches 20,000 attempts per second. Configure admission below the measured ceiling, initially at 1,600 attempts per second, and cap concurrent inventory calls based on tested safe concurrency. A virtual waiting room holds users outside the inventory path; its tokens must be validated before database work. Reserve a tested share of capacity for confirming existing holds. Reject excess direct attempts with an explicit retry policy rather than queueing them indefinitely. If database latency rises, lower admission even if application CPU is idle. Track completed holds, rejection rate, queue wait, and confirmation latency, then increase admission cautiously after recovery.

Trade-offs to say out loud

Partial rejection versus widespread failure: Some callers receive immediate failures so admitted requests can keep completing instead of nearly every request timing out.
Early rejection versus precise decisions: Edge rejection saves the most resources, but service-local admission has better information about request cost and dependency health.
Utilization versus safety margin: Conservative limits leave spare capacity, while aggressive limits risk crossing the point where queues and latency grow rapidly.
Priority versus fairness: Protecting critical traffic improves business outcomes, but lower-priority tenants or operations need explicit starvation and fairness policies.
Adaptation versus stability: Adaptive limits respond to changing capacity, but noisy signals and delayed feedback can cause repeated rejection and recovery cycles.
Data loss versus contract preservation: Dropping sampled telemetry may be acceptable, but durably accepted payments or jobs must retain their promised delivery guarantees.

When to reach for it

A flash sale or giveaway can generate bursts far beyond the capacity of the inventory or claim path.
A slow dependency, failover, or cache-miss surge reduces effective capacity even though incoming traffic has not increased.
Queues, connection pools, or in-flight requests grow while completed throughput stops improving.
Shared infrastructure needs to protect critical requests and well-behaved tenants from expensive or excessive work.
Autoscaling cannot react quickly enough, or scaling application servers would only overload a fixed-capacity database.

How to bring it up in the interview

Name the resource you are protecting and place admission control before it: for example, reject before password hashing or before acquiring a database connection. Expect questions about how you choose and validate the limit.
Define the overload policy concretely: maximum concurrent work, bounded queue length or queue-wait deadline, rejection behavior, and which operations retain reserved capacity.
Separate overload rejection from durable acceptance. Explain that rejecting a new job is different from dropping one already acknowledged, and describe how clients resolve ambiguous outcomes using idempotency keys.
Explain how retries interact with shedding. Interviewers may probe retry amplification, so include backoff, jitter, retry budgets, and deadlines rather than assuming clients will behave well.
Describe an overload test and recovery criteria. Measure successful throughput and admitted-request latency alongside rejection rates, since fast failures alone can make latency dashboards look deceptively healthy.

Common mistakes

Treating autoscaling as immediate protection without accounting for startup time or a downstream bottleneck that cannot scale.
Adding an unbounded queue and calling it overload handling; this often converts excess traffic into memory pressure and expired work.
Using only a fixed requests-per-second limit when request costs or dependency latency vary substantially.
Rejecting at the gateway but leaving services without local concurrency limits, allowing skewed routing or internal fan-out to overload individual instances.
Silently dropping acknowledged durable work, or returning overload responses without a retry policy that prevents another surge.

Where it comes up

Interview questions where load shedding is part of a strong answer, and why.

Questions people ask

What is the difference between load shedding and rate limiting?

Rate limiting controls how much traffic a caller or operation may submit under a policy, such as 100 requests per minute. Load shedding protects available capacity during overload, often using concurrency, queue delay, or dependency health. A rate limiter can implement a shedding policy, but a fixed quota alone does not adapt to reduced capacity.

How is load shedding different from backpressure?

Backpressure tells upstream producers to slow down or stop sending because downstream capacity is limited. Load shedding rejects or discards selected work when the system cannot safely accept it. They work together: apply backpressure where producers can cooperate, and shed excess work when they cannot or when waiting would exceed a deadline.

Where should load shedding happen in a distributed system?

Reject as early as practical to avoid wasting resources, but also enforce limits near each bottleneck. Edge admission protects the fleet, while service-local controls protect individual instances and dependencies. Avoid making every admission decision depend on a new synchronous call to one central coordinator, which can itself become an overload bottleneck.