How the interviewer opens it
“Design a metrics and monitoring system like Datadog. It should collect performance metrics from thousands of servers, power dashboards, and alert operators when a service or cluster is unhealthy.”
The prompt is vague on purpose. The first few minutes are yours: narrow it to the core features and agree on the numbers before you draw anything.
What to clarify first
- Signal types and scope
- Scale
- Sample sizing
- Read:write mix
- Latency
- Retention and queries
- Availability, consistency, and durability
- Timestamps and late data
- Labels and cardinality
- Alert behavior
The answers stay with the interviewer. In a SysRound round, Alex answers them as an interviewer would.
Scope of a strong answer
Core requirements
- Collect timestamped metrics with host, service, cluster, and other bounded labels
- Store metrics and query time ranges with filtering, grouping, and aggregation
- Support dashboards at several time resolutions
- Evaluate alert rules and send notifications
Usually out of scope
- Raw log ingestion and search
- Distributed tracing and application instrumentation libraries
- Building a dashboard editor
- Billing and account management
- Cross-region disaster recovery
- Arbitrary historical backfills
What interviewers probe
- Retries, late samples, and outage recovery
- Agents must keep collecting through failures, but replayed or out-of-order samples can corrupt aggregates. Waiting for every late sample would violate the freshness and alerting targets.
- Series cardinality and query fan-out
- Sample throughput is not the only scaling limit. New label combinations create index and memory pressure, and one broad selector can turn a cheap dashboard refresh into a fleet-wide scan.
- Aggregation semantics and irreversible rollups
- Metrics are not interchangeable numbers. Averages, cumulative counters, and histogram percentiles require different state, and an incorrect rollup cannot be repaired once raw data expires.
- Alert correctness when data or workers disappear
- An unhealthy service, a broken collector, and an overloaded query system can all look like missing metrics. Alert state must survive evaluator failures without flapping or repeatedly paging.
Common mistakes
- Designing a log search engine instead of a numeric time-series system.
- Discussing sample throughput but never estimating active series, label-index size, or series churn.
- Claiming exactly-once behavior because a durable queue is present, without explaining replay effects on storage and rollups.
- Applying sum or average uniformly to gauges, counters, and histogram percentiles.
- Allowing dashboard scans to exhaust the same capacity needed for ingestion and alerts.
- Treating missing or partial data as a healthy zero and resolving alerts during a monitoring outage.
The bar by level
- Senior (L5/E5)
- Drives sizing and access-pattern choices; handles tenant cardinality limits, idempotent ingestion, bounded late data, counter resets, histogram aggregation, and resolution-aware queries. Provides durable alert state and notification retry behavior, and protects collection and alert capacity from broad dashboard queries.
- Staff (L6+/E6+)
- Defines correctness and failure contracts across the entire pipeline, including partial queries, replay, late rollup revisions, retention, and alert state transitions. Distinguishes sample-rate, cardinality, and query-cost bottlenecks; plans backlog recovery and tenant isolation; explains where information is irreversibly lost and which independent failure domains are needed to monitor the monitoring system.