Service discovery for system design interviews

How it works

A service registry stores instance records under a logical name. For example, inventory might map to 10.0.1.8:8080 and 10.0.2.4:8080, with metadata such as zone, version, and readiness.

Instances register themselves, or a platform controller registers them. Heartbeats, leases, and health checks help remove unavailable instances; readiness checks keep newly started instances out until they can serve traffic.

With client-side discovery, the caller obtains endpoints through a lookup or watch, caches them, and selects an instance using a load-balancing policy. The application or its client library must handle endpoint updates and connection failures.

With server-side discovery, the caller connects to a stable load balancer or proxy, which discovers backends and selects one. A service-mesh sidecar can do similar work on the caller's behalf without putting discovery logic in application code.

DNS can expose a stable virtual address or return instance addresses directly. DNS caches and long-lived connections affect how quickly changes take effect, so a short DNS TTL alone does not guarantee fast traffic migration.

Discovery updates propagate asynchronously. Callers may temporarily use a failed endpoint, so requests still need timeouts and bounded retries where safe. During planned shutdown, remove the instance from new-request routing and drain existing work before stopping it.

A worked example

Scaling and draining an inventory service

Suppose an inventory service has 12 instances and scales to 18. A controller publishes the six new endpoints only after their readiness checks pass. Client libraries watch the registry, keep a local endpoint set, and distribute requests across ready instances. Each instance renews a lease every 5 seconds, with a 20-second expiry. If one crashes, callers may still select it before failure detection and update propagation finish; request timeouts and bounded retries handle that gap. For a planned deployment, the controller first removes an old instance from new-request routing, allows time for propagation, and drains active requests. If the registry becomes unavailable, existing clients can temporarily use cached endpoints, but new clients without a cache may fail to initialize.

Trade-offs to say out loud

Client-side control versus client complexity: Direct endpoint selection supports locality-aware routing but requires compatible discovery, balancing, and connection-management behavior across clients.
Proxy simplicity versus another network component: A proxy centralizes routing behavior but adds resource cost and a component whose capacity and failure modes need attention.
Fast failure detection versus false removals: Aggressive health checks or short leases remove failed instances sooner but can eject healthy instances during pauses or network problems.
Fresh endpoints versus control-plane load: Frequent polling reduces some staleness but increases registry traffic, while watches reduce polling at the cost of reconnect and resynchronization logic.
Cached availability versus stale routing: Keeping a last-known endpoint set can preserve traffic during a registry outage, but those endpoints may fail or become invalid as the outage continues.
Dynamic registration versus security exposure: Registry writes and reads need appropriate authentication and authorization because a forged registration can redirect service traffic.

When to reach for it

Service instances scale or restart frequently, so callers cannot rely on fixed addresses.
Internal services communicate directly and need an explicit way to find available peers.
A partitioned system needs bootstrap endpoints and metadata describing current node addresses.
Rolling deployments or failover require routing changes without updating every caller's configuration.

How to bring it up in the interview

Start with the routing requirement: who needs to find whom, how often endpoints change, and whether a platform-provided service name already solves it.
Separate discovery from load balancing: discovery supplies eligible endpoints, while load balancing chooses where a request goes.
Name who registers instances, what makes them ready, and how callers learn about removals; expect questions about stale caches and long-lived connections.
Describe registry failure explicitly: existing callers may continue with cached endpoints, while cold-starting callers may have no usable endpoint set.
For stateful systems, separate reachability from authority: finding a node does not prove it is the current leader or owns a partition, so interviewers may probe epochs, fencing, and stale metadata.

Common mistakes

Adding a custom service registry when the platform's DNS and load-balancing facilities already satisfy the requirements.
Querying the registry synchronously for every request and making it a dependency of every data-plane operation.
Treating a heartbeat as proof that an instance can successfully handle application requests.
Assuming removing an endpoint immediately closes pooled connections or stops all in-flight traffic.
Using discovery records as the sole mechanism for leader election or split-brain prevention.

Where it comes up

Interview questions where service discovery is part of a strong answer, and why.

Questions people ask

What is the difference between service discovery and a service registry?

Service discovery is the process of finding service endpoints. A service registry is a directory used by that process, usually storing instance addresses and metadata. Discovery also includes registration, health filtering, update propagation, and client lookup behavior.

What is the difference between service discovery and load balancing?

Service discovery answers which endpoints are available for a service. Load balancing decides which eligible endpoint receives a request or connection. A proxy or client library often performs both, but they solve different problems.

Can DNS be used for service discovery?

Yes. DNS can return instance addresses or a stable address for a load balancer, and SRV records can also describe ports. DNS caching, resolver behavior, and connection reuse influence how quickly callers react to endpoint changes.