How it works
Start with user-visible outcomes, not a dashboard inventory. For a checkout service, measure successful checkouts, request latency, and orders stuck waiting for payment; then add diagnostic signals such as database connection usage and queue age.
Metrics summarize behavior over time using counters, gauges, and histograms. Count requests and failures, track active connections, and record latency distributions; use bounded labels such as service, region, and route template rather than user IDs or raw URLs.
Logs capture individual events with structured fields. A payment timeout log might include timestamp, service, operation, error category, and trace ID, letting an engineer search for the events behind a rising failure rate without parsing free-form messages.
Traces describe an operation as a graph of spans across service boundaries. Propagate trace context through HTTP calls and message headers so a checkout trace can connect API handling, a database query, and a payment-provider call; asynchronous processing needs explicit context or span links.
Export telemetry through collectors to storage and query systems suited to each signal. Batch exports, bound buffers, and define drop policies so a telemetry outage does not block application requests; expose dropped-data counts and collection lag as well.
Use service-level indicators and objectives to turn measurements into actionable alerts. An alert about checkout failures should link to relevant dashboards, traces, and logs; separately detect missing telemetry so a silent service is not mistaken for a healthy one.
A worked example
Diagnosing a slow checkout
Suppose a checkout service handles 2,000 requests per second with an objective that 99% finish within one second. Record request counts, failures, and latency histograms by route and region. At a 1% head-sampling rate, roughly 20 requests per second are selected for tracing, though rare failures can still be missed. When slow checkouts exceed the allowed proportion, an alert links to the latency dashboard. Available traces show payment calls consuming most of the request time, and structured logs reveal provider timeouts. Track connection-pool saturation to distinguish an external slowdown from local resource exhaustion. If telemetry export stalls, bounded buffers protect checkout traffic while a separate collection-health check reports the visibility gap.
Trade-offs to say out loud
- Diagnostic detail versus cost
- Verbose logs and complete traces improve investigations but consume application resources, network bandwidth, storage, and query capacity.
- Label flexibility versus cardinality
- Every distinct metric label combination creates another time series, so unbounded dimensions such as request IDs can overwhelm the monitoring backend.
- Sampling versus coverage
- Sampling reduces trace volume but can miss rare failures, while retaining traces based on their final outcome requires buffering and coordinated collection.
- Telemetry durability versus application availability
- Bounded asynchronous export protects application traffic during monitoring outages but can lose diagnostic data precisely when failures occur.
- Useful context versus privacy
- Request details can accelerate debugging, but sensitive fields require redaction, access controls, and limited retention rather than indiscriminate collection.
- Fast detection versus alert noise
- Aggressive thresholds detect problems sooner but can page on harmless spikes, so alerts should reflect sustained user impact and have a clear response.
When to reach for it
- A request crosses several services or asynchronous workers, making the source of latency or failure hard to locate.
- The design includes retries, queues, or eventual consistency, so successful API responses do not prove that work completed.
- You need to defend availability, latency, or freshness targets and explain how operators will detect violations.
- The interview asks you to design a metrics, tracing, monitoring, or alerting platform.
How to bring it up in the interview
- Name the user outcome first: 'For notifications, I would measure delivery outcome and time to delivery, not just queue ingestion success.' Interviewers may probe what counts as success when a provider only acknowledges acceptance.
- Assign each signal a job: metrics detect and quantify the problem, traces locate work across boundaries, and logs explain specific events. Explain how trace IDs and consistent resource attributes connect them.
- State your cardinality budget, sampling policy, and retention assumptions. Expect questions about a tenant creating millions of series or a rare error disappearing from sampled traces.
- Explain alert semantics, including the denominator, evaluation window, and missing-data behavior. For latency across instances, aggregate compatible histogram buckets rather than averaging instance-level percentiles.
- Describe how observability fails: bounded export queues, telemetry drop counters, collector health checks, and an independent way to detect a monitoring outage.
Common mistakes
- Listing monitoring tools without specifying what is measured, which failure it reveals, or what action follows.
- Treating metrics, logs, and traces as interchangeable, or assuming traces automatically include useful application events.
- Using request IDs, user IDs, or arbitrary URLs as metric labels instead of keeping that context in appropriately controlled logs or traces.
- Monitoring CPU and API errors while ignoring queue age, processing freshness, and end-to-end business outcomes.
- Treating missing samples as zero errors, or using sampled diagnostic telemetry as a complete audit trail.
Where it comes up
Interview questions where observability is part of a strong answer, and why.
- Design a metrics monitoring system like DatadogThe core design must preserve useful measurements under cardinality growth, ingestion failures, and missing data while producing correct alerts.
- Design a distributed tracing systemThe core design connects cross-service operations through context propagation, coherent sampling, and searchable spans without harming application traffic.
- Design an on-call escalation system like PagerDutyThe alert-to-human part of observability requires durable escalation, duplicate-alert handling, acknowledgements, and honest treatment of uncertain notification delivery.
Questions people ask
What is the difference between observability and monitoring?
Monitoring usually checks known conditions, such as an error rate exceeding a threshold. Observability is the broader ability to investigate system behavior, including failures you did not predict in advance. Useful monitoring is part of an observable system.
What is the difference between logs, metrics, and traces?
Logs describe individual events, metrics summarize measurements over time, and traces connect related operations across components. A latency metric can reveal a slowdown, a trace can locate the slow dependency, and a log can explain the specific error.
Do you need to collect every log and trace?
No: collection volume should follow diagnostic needs, cost limits, and privacy requirements. Use selective logging, trace sampling, and explicit retention policies, while documenting what those choices can hide. Required audit records need a separate completeness and durability policy rather than ordinary diagnostic sampling.