How it works
Clients connect to a service endpoint rather than choosing an application server themselves. The load balancer selects a healthy backend and forwards traffic to it; for example, an API endpoint might distribute requests across six interchangeable application servers.
A Layer 4 (L4) load balancer routes TCP or UDP flows using network information such as IP addresses and ports. A Layer 7 (L7) load balancer understands an application protocol such as HTTP, allowing routing by hostname or path, such as sending /uploads to a separate server pool. HTTPS inspection requires TLS termination.
The selection algorithm determines how traffic spreads. Round robin cycles through servers, weighted routing accounts for different capacities, and least-connections routing favors servers with fewer active connections. Connection counts are only a rough measure of work, especially when connections carry different amounts of traffic.
Health checks and service discovery maintain the backend pool. Readiness checks determine whether an instance should receive new traffic, while draining stops new assignments and gives existing work time to finish during deployments. Removing a failed server does not rescue requests already running on it.
Stateless requests can usually reach any healthy replica if session state lives in shared storage or a verifiable client token. WebSocket connections remain attached to one backend for their lifetime; reconnects may reach another server, so message routing and recovery need a separate design.
The balancing layer also needs redundancy and enough connection and network capacity. Multiple load-balancer instances, often managed across availability zones, avoid a single appliance becoming the failure point. Monitor backend latency, errors, active connections, and traffic skew to verify that balancing actually works.
A worked example
Balancing a chat service during a server failure
Suppose a chat service has 120,000 WebSocket connections across six servers. At an even distribution, each holds 20,000 connections; assume load testing establishes a safe limit of 30,000 per server for the expected message traffic. A load balancer uses least connections for new handshakes, but existing sockets stay attached to their original server. If one server fails, its 20,000 clients reconnect with randomized backoff. The remaining five servers would average 24,000 connections each, within the assumed limit. A connection directory or subscription layer routes messages to each recipient's current server. Clients resume from an acknowledged sequence so reconnecting does not silently lose messages. Connection counts and outbound traffic both need monitoring.
Trade-offs to say out loud
- L4 simplicity versus L7 control
- L4 routing needs less protocol inspection, while L7 routing enables application-aware policies at the cost of more processing and configuration.
- Even distribution versus useful load signals
- Round robin is simple, but variable request costs may require concurrency, latency, or capacity-aware routing rather than equal request counts.
- Session affinity versus flexibility
- Sticky sessions preserve locality but can create uneven load and complicate failover when important state exists only on one backend.
- Fast failure detection versus false alarms
- Aggressive health checks remove bad instances sooner but can eject usable capacity during brief slowdowns and overload the remaining servers.
- Retries versus duplicate work
- Retrying another backend can hide transient failures but can amplify overload or repeat side effects unless retries are bounded and operations are safe to retry.
- Higher availability versus operational cost
- Redundant balancers, cross-zone traffic, TLS termination, and connection tracking add cost and capacity limits that must be monitored.
When to reach for it
- The service needs multiple interchangeable application replicas to meet throughput or availability requirements.
- Different HTTP paths or hostnames need separate backend pools, scaling policies, or deployment schedules.
- Long-lived connections require explicit placement, connection limits, draining, and reconnect behavior.
- Deployments or instance failures must stop receiving new traffic without taking down the entire service.
- Traffic is uneven across replicas, and you need to explain which routing signal reflects actual backend load.
How to bring it up in the interview
- Name the unit being balanced: TCP connections, HTTP requests, or long-lived sessions; interviewers often probe whether your chosen algorithm measures the right thing.
- Choose L4 or L7 based on a concrete requirement, such as TCP pass-through or HTTP path routing, rather than drawing an unexplained load-balancer box.
- Explain why any selected backend can handle the work; for stateful services, distinguish initial connection placement from routing messages to the server holding that connection.
- Estimate peak requests or connections per backend, then leave headroom for failures; expect questions about losing an availability zone or draining a busy instance.
- Describe overload behavior explicitly: bounded queues, admission control, timeouts, and limited retries, because redistributing traffic cannot fix insufficient total capacity.
Common mistakes
- Assuming a load balancer automatically scales application capacity or removes database bottlenecks.
- Treating equal connection counts as equal CPU use, request volume, or network traffic.
- Using sticky sessions as the only plan for preserving state across backend failures.
- Assuming health checks make failover instantaneous or transparently recover in-flight requests.
- Confusing connection placement with pub/sub delivery, partition ownership, or routing to the server that holds a user's active connection.
Where it comes up
Interview questions where load balancing is part of a strong answer, and why.
- Design a chat app like WhatsAppConnection placement, server draining, and reconnect routing are central to operating a fleet of stateful chat servers.
- Design a real-time stock price feedBalancing long-lived subscriber connections must account for uneven symbol traffic, outbound bandwidth, and reconnect behavior during failures.
- Design Facebook live commentsServing millions of live viewers requires distributing persistent connections across delivery servers without confusing connection balancing with comment fanout.
Questions people ask
What is a load balancer?
A load balancer distributes incoming traffic across a pool of backend servers. It commonly checks backend health and stops assigning new traffic to instances that are unhealthy or being drained.
What is the difference between L4 and L7 load balancing?
L4 load balancing selects a backend for a network flow without routing on HTTP content. L7 load balancing understands an application protocol and can route individual HTTP requests by host, path, or headers. The choice affects protocol support, TLS handling, routing flexibility, and resource use.
What is the difference between a load balancer and an API gateway?
A load balancer primarily distributes traffic across backends, while an API gateway commonly adds policies such as authentication, quotas, and API-level routing. One product can perform both roles, so explain the responsibilities in your design rather than assuming they require separate components.