How the interviewer opens it
“Design a distributed tracing system like Jaeger or Zipkin. Engineers should be able to follow a request across hundreds of microservices and work out where time went or what failed.”
The prompt is vague on purpose. The first few minutes are yours: narrow it to the core features and agree on the numbers before you draw anything.
What to clarify first
- Deployment and instrumentation
- Traffic and span size
- Sampling
- Read:write mix
- Search semantics
- Latency and freshness
- Trace lifecycle
- Availability, consistency, and durability
- Retention
- Boundaries and data safety
The answers stay with the interviewer. In a SysRound round, Alex answers them as an interviewer would.
Scope of a strong answer
Core requirements
- Capture spans and propagate trace context across service and queue boundaries
- Collect and retain sampled spans without blocking application requests
- Retrieve a trace by ID and show timing, parent relationships, and errors
- Find traces using service, operation, time range, duration, error status, and selected tags
Usually out of scope
- Implementing instrumentation libraries for every language
- A general-purpose logs or metrics platform
- Frontend implementation
- Multi-region operation in v1
What interviewers probe
- Preserving causality and explaining where time went
- A trace is not simply a list sorted by timestamp. Parallel calls overlap, client and server spans describe the same interaction, clocks differ, and asynchronous work may outlive the root request.
- Durable ingestion without harming the application
- The application must keep serving when tracing is unhealthy, yet acknowledged telemetry must survive an entire zone failing. The durability guarantee must hold both while data is in the log and after it moves into the span store.
- Coherent sampling and the limits of error retention
- Sampling spans independently destroys request structure. Sampling whole traces at the root preserves structure but cannot know whether a later hop will fail; changing that policy changes the scale of the entire pipeline.
- Reconciling trace lookup with searchable span attributes
- Trace lookup groups data by trace ID, while search groups it by service, time, status, and tags. Arbitrary attributes can make indexes much larger than the telemetry itself.
Common mistakes
- Designing only a dashboard and database while leaving context propagation unspecified.
- Sampling spans independently and producing disconnected fragments.
- Treating a root span's completion as proof that the whole trace has arrived.
- Putting synchronous tracing writes on the application request path.
- Summing span durations despite parallelism, client/server overlap, and clock skew.
- Promising zone-failure durability without zone-aware replicas in both storage stages and safe log-retention rules.
The bar by level
- Senior (L5/E5)
- Makes sampling coherent across services, identifies the durable acknowledgment boundary, handles replay and duplicate spans, and designs both storage access paths. Specifies cross-zone replication for the log and store, retains the log until safe handoff, and explains late arrivals, backpressure, parallel timing, and clock skew.
- Staff (L6+/E6+)
- Traces the acknowledged-data guarantee through zone failures, partial writes, log deletion, and exhausted backlog capacity. Quantifies the move from head to tail sampling and defines bounded decision and recovery behavior. Defends a simple initial storage design, states search semantics precisely, and adapts it for cross-span queries, regional isolation, or cheaper retention without hiding consistency and operational costs.