# Distributed tracing > Distributed tracing follows a request or workflow across services by recording linked operations called spans. It helps explain where time went and which dependency failed when a single log file or service-level metric cannot show the whole path. Source: https://sysround.com/concepts/distributed-tracing Updated: 2026-11-05 ## How distributed tracing works Each recorded operation produces a span with a trace ID, span ID, parent or link information, start time, duration, and attributes such as service name and operation. For example, a checkout trace might contain spans for the API handler, inventory lookup, payment call, and database queries. Services propagate trace context across boundaries, commonly through W3C Trace Context headers on HTTP requests and metadata on messages. Each downstream operation creates its own span while preserving the relationship to the caller; sharing only a trace ID does not preserve the full operation structure. For asynchronous work, attach context to the message and instrument both production and consumption. A consumer can continue a trace or start a new trace with a span link, depending on the workflow; batching and long-running jobs often need links rather than one simple parent-child tree. Instrumentation exports spans asynchronously through bounded buffers to collectors. Collectors can batch, filter, and forward data into a durable ingestion pipeline without making application requests wait for trace storage. When capacity runs out, an explicit drop policy protects application availability at the cost of missing telemetry. Sampling controls volume. Head sampling decides near the start and propagates a consistent decision across services; tail sampling waits for spans and can favor slow or failed traces. Tail samplers commonly route spans by trace ID so one decision-maker can assemble them, but bounded waiting means late spans and incomplete traces still occur. The backend stores spans for trace-ID lookup and maintains selected indexes for searches such as service, operation, error, and duration. The UI reconstructs relationships and a timeline, but must show missing spans honestly: overlapping spans cannot simply be added, and clock skew can distort cross-host timing. ## Worked example: Sizing tracing for a checkout service Suppose checkout receives 10,000 requests per second and produces an average of 20 spans per request. Full recording creates 200,000 spans per second. With coherent 5% head sampling, about 10,000 spans per second reach the backend; at an assumed 1 KB per encoded span, that is roughly 10 MB/s before replication and indexes. A retained trace shows a 900 ms checkout with a 700 ms payment call and a concurrent 100 ms inventory lookup: those durations should not be added. To retain more failed checkouts, consider tail sampling, but budget for receiving and buffering the unsampled stream. It cannot recover failures already excluded by head sampling. ## Trade-offs - **Visibility versus application overhead.** Span creation, context propagation, and export consume CPU, memory, and bandwidth, so instrumentation and buffering need explicit budgets. - **Head sampling versus tail sampling.** Head sampling is cheaper and simpler, while tail sampling can select interesting outcomes but requires buffering and cannot recover spans that were never recorded or exported. - **Completeness versus application availability.** Bounded queues and dropping protect application traffic during tracing outages, but leave gaps that prevent complete diagnosis. - **Search flexibility versus index cost.** Indexing every span attribute increases write amplification and storage cost, so choose searchable fields deliberately. - **Diagnostic detail versus privacy.** Request bodies, credentials, and sensitive identifiers can leak through attributes or propagated baggage, requiring allowlists, redaction, and access controls. - **Retention versus cost.** Long retention helps investigate older incidents, but raw spans and their indexes need separate retention and storage-tier decisions. ## When to use it in a system design interview - A request crosses several services, and you need to distinguish local work, dependency calls, retries, and waiting time. - An asynchronous workflow needs producer-to-consumer correlation that ordinary request logs do not preserve. - Tail latency or intermittent failures appear in aggregate metrics, but the responsible execution path is unclear. - The interview explicitly asks you to design a tracing platform with high-volume ingestion, sampling, and searchable trace storage. ## How to bring it up in the interview - Start with the diagnostic question: “For a slow checkout, I want to identify the blocking dependency and any retries.” Then show the span relationships and propagation boundaries that answer it. - For a tracing-system design, estimate spans per second as requests per second × average spans per request × retained fraction, then account separately for collector ingress, storage, replication, and indexes. - State where sampling happens and whether it is trace-consistent. Expect questions about why independent per-service sampling creates fragmented traces and why tail sampling cannot guarantee retaining every error. - Explain the failure policy before discussing dashboards: bounded SDK buffers, asynchronous export, collector backpressure, and explicit drops must keep a tracing outage from becoming an application outage. - Distinguish causal relationships from timestamp ordering. Expect follow-ups on parallel calls, queue delays, clock skew, late spans, and how confidently the UI can explain the critical path. ## Common mistakes - Describing tracing as centralized logging without explaining spans, context propagation, or parent-child relationships. - Sampling each service independently and assuming the retained spans will form complete traces. - Promising to retain every failed trace with tail sampling even when spans were dropped upstream or arrive after the decision window. - Adding every span duration to calculate request latency, even though parent spans include child time and parallel work overlaps. - Treating traces as an audit log or proof of business correctness despite sampling, missing spans, and best-effort export. ## Interview questions where it comes up - [Design a distributed tracing system](https://sysround.com/questions/design-distributed-tracing): The entire design centers on propagating context, ingesting spans safely, making coherent sampling decisions, and supporting trace lookup alongside indexed search. ## FAQ ### What is the difference between distributed tracing, logging, and metrics? Metrics summarize behavior across many operations, logs record individual events, and traces connect operations across service boundaries. Use metrics to detect a latency increase, traces to investigate representative slow paths, and correlated logs for additional detail. ### How does distributed tracing work with message queues? The producer includes trace context in message metadata, and the consumer extracts it when processing begins. Consumer spans preserve the relationship through a parent or span link. Record production and processing timing explicitly so queue delay is not confused with execution time. ### What is head sampling versus tail sampling in distributed tracing? Head sampling makes an early decision, usually before the request outcome is known, and can reduce export traffic substantially. Tail sampling evaluates collected spans to favor errors, high latency, or other outcomes, but needs buffering and trace-aware routing. Its decisions are limited by missing spans, late arrivals, and the configured waiting window. ## Related concepts - [Observability](https://sysround.com/concepts/observability) - [Log aggregation](https://sysround.com/concepts/log-aggregation) - [Microservices architecture](https://sysround.com/concepts/microservices) - [Backpressure](https://sysround.com/concepts/backpressure) --- SysRound (https://sysround.com) is an AI practice tool for system design interviews: a voice or typed AI interviewer, a shared whiteboard, then feedback on every skill and a model answer. Its feedback is AI-generated and can be wrong.