# Autoscaling > Autoscaling automatically adjusts compute capacity as demand changes. In system design interviews, the important part is explaining what triggers scaling, how quickly capacity becomes useful, and what protects the system while it catches up. Source: https://sysround.com/concepts/autoscaling Updated: 2026-11-05 ## How autoscaling works Choose the unit to scale. Horizontal scaling adds or removes instances, containers, or workers; vertical scaling changes their CPU or memory. Stateless request handlers and independent queue workers are usually easier to scale horizontally than databases or services that own long-lived connections. Measure demand against a capacity target. For a request-serving tier, useful signals include requests per instance, CPU utilization, and in-flight requests. For workers, use backlog, arrival rate, processing rate, and oldest-job age. CPU alone can miss an overloaded service whose workers are waiting on a database or external API. Run a feedback loop: observe metrics, calculate desired capacity, and reconcile actual capacity toward that target. If a load test establishes a target of 200 requests per second per instance, 2,000 requests per second calls for 10 instances before any additional failure reserve. Bound the result with minimum and maximum capacity. Account for the delay between requesting capacity and serving traffic. Scheduling, machine provisioning, application startup, and cache warmup all take time. Readiness checks keep unready instances out of load balancing, and the underlying compute pool must have room to run new instances. Stabilize the loop. Scale out promptly when demand rises, but require sustained low demand before scaling in. Stabilization windows and bounded scaling steps reduce repeated expansion and contraction. Before removing capacity, drain requests or connections and let workers finish or safely relinquish leased jobs. Combine reactive scaling with scheduled capacity for predictable peaks. Keep enough headroom to absorb startup delays, and use bounded queues, backpressure, or load shedding when demand exceeds available capacity. Adding application instances does not increase a shared database's throughput or a provider's quota. ## Worked example: Scaling notification workers through a burst Assume each notification worker can process 100 jobs per second, and the operating target is 80 to leave headroom. At 400 incoming jobs per second, five workers meet that target. Traffic jumps to 1,200 jobs per second, so the controller requests 15 workers. If new workers take 60 seconds to become ready, the original five can process at most 500 jobs per second, accumulating about 42,000 jobs. Once all 15 are ready, their maximum throughput is 1,500 jobs per second, leaving 300 per second to drain the backlog: roughly 140 seconds. These estimates assume uniform jobs and no provider bottleneck; a tighter waiting-time target requires pre-scaling or more temporary capacity. ## Trade-offs - **Cost versus spare capacity.** Low minimum capacity saves money, but leaves less room for sudden bursts, failures, and slow instance startup. - **Responsiveness versus stability.** Aggressive scaling reacts quickly but can oscillate when metrics are noisy or capacity takes time to become ready. - **Simple signals versus accurate signals.** CPU is easy to measure, while workload-specific signals such as oldest-job age better reflect user impact but require careful instrumentation. - **More workers versus downstream overload.** Additional workers can exhaust database connections or provider quotas unless concurrency and scaling limits respect those dependencies. - **Elasticity versus state movement.** Scaling stateful services may require data transfer, partition rebalancing, or connection migration, making changes slower and riskier. - **Scale-to-zero versus startup latency.** Removing all idle workers reduces compute cost but requires an external demand signal and makes the first work wait for startup. ## When to use it in a system design interview - Traffic changes substantially over the day, and maintaining peak capacity continuously would be wasteful. - A durable queue absorbs bursts, but processing capacity must grow to meet a job-start or completion deadline. - A service has a measurable per-instance capacity limit and can distribute work across additional instances. - Predictable events, such as scheduled jobs or ticket sales, require capacity before reactive scaling could respond. ## How to bring it up in the interview - Start with the service objective: request latency, oldest-job age, or job completion time. Then choose a scaling signal that predicts trouble before that objective is violated. - Estimate per-instance throughput from an explicit load-test assumption, calculate the required instance count, and explain the headroom reserved for bursts and failures. - Separate application scaling from infrastructure provisioning. Interviewers may ask what happens when more containers are requested but no machines are available. - Describe both directions: readiness and warmup during scale-out, then stabilization, draining, and safe job handoff during scale-in. - Name the next bottleneck and the overload policy. Expect questions about database capacity, provider quotas, partition limits, stale metrics, and behavior at maximum scale. ## Common mistakes - Saying 'add autoscaling' without specifying a signal, target, capacity bounds, or startup delay. - Assuming horizontal scaling fixes hot keys, lock contention, or a saturated shared database. - Scaling workers only on queue length without considering job duration, arrival rate, or waiting-time targets. - Terminating instances immediately during scale-in, dropping connections or interrupting work without a safe retry path. - Relying on reactive autoscaling alone for a flash crowd that arrives faster than new capacity can start. ## Interview questions where it comes up - [Design a notification system](https://sysround.com/questions/design-notification-system): Per-channel worker scaling must balance notification waiting times against provider quotas, retries, and competing traffic. - [Design a video streaming service like YouTube](https://sysround.com/questions/design-youtube): Transcoding worker capacity must adapt to upload volume and processing cost while keeping the publication backlog within a target delay. - [Design a distributed job scheduler](https://sysround.com/questions/design-job-scheduler): Meeting start-time targets under cron bursts requires worker capacity planning, pre-scaling, and explicit handling of provisioning delays. ## FAQ ### What is the difference between autoscaling and load balancing? Load balancing distributes work across available instances. Autoscaling changes how many instances or how much compute capacity is available. They work together: new instances must become ready and discoverable before the load balancer sends them traffic. ### Which metrics should trigger autoscaling? Choose metrics that reflect the workload's limiting resource and service objective. Request-serving tiers might use requests per instance, CPU, or concurrency; asynchronous workers often need arrival rate, backlog, and oldest-job age. Latency is useful for validating the outcome, but can be a late or misleading trigger when a downstream dependency is the bottleneck. ### How do you autoscale queue consumers? Estimate the workers needed to keep up with arrivals, then add capacity to drain existing work within the required time. Account for processing time, startup delay, provider quotas, and partition limits; in a consumer group with one active consumer per partition, extra consumers beyond the partition count cannot increase parallelism. During scale-in, finish or safely relinquish leased work and make retries idempotent. ## Related concepts - [Stateless services](https://sysround.com/concepts/stateless-services) - [Load balancing](https://sysround.com/concepts/load-balancing) - [Backpressure](https://sysround.com/concepts/backpressure) - [Load shedding](https://sysround.com/concepts/load-shedding) --- SysRound (https://sysround.com) is an AI practice tool for system design interviews: a voice or typed AI interviewer, a shared whiteboard, then feedback on every skill and a model answer. Its feedback is AI-generated and can be wrong.