# Prometheus for system design interviews > Prometheus collects numeric metrics, stores them as time series, and evaluates queries and alert rules over them. It fits operational monitoring, such as tracking request rates, error rates, latency, and resource usage, but is not a durable record of every business event. Source: https://sysround.com/tech/prometheus Updated: 2026-11-05 ## How it works Applications expose metrics over HTTP, usually through a client library or an exporter that translates another system's statistics. Prometheus discovers targets through static configuration or service discovery, including Kubernetes, and periodically scrapes them. Each metric name plus its complete set of labels identifies a time series. The ingestion path appends samples to the local time-series database. Recent samples live in the head block, backed by a write-ahead log for crash recovery; older data is persisted in immutable blocks and compacted. Blocks contain compressed sample chunks and an index for finding series by labels. PromQL selects series by labels and computes rates, aggregations, and time-window operations. An instant query evaluates an expression at one point in time; a range query evaluates it repeatedly across a time interval. Queries run against the server's own data unless an additional distributed query layer is introduced. A rule evaluator periodically runs recording rules and alerting rules. Recording rules save expensive or frequently reused expressions as new series. Alerting rules produce alerts that Prometheus sends to Alertmanager, which handles grouping, deduplication, routing, silences, and inhibition. Optional remote write sends samples to an external backend for longer retention or centralized querying. Federation instead lets one Prometheus server scrape selected series from another, often for aggregated cross-cluster views. Neither mechanism turns the local Prometheus server into an automatically sharded, replicated database. ## What it gives you in a design - **Pull-based collection and target health.** Prometheus controls scrape frequency and records whether each scrape succeeds through the up metric. This makes target reachability visible, but network access from the scraper to targets must be designed explicitly. - **Dimensional metrics.** Labels support breakdowns such as service, region, status class, and route template. Every distinct label combination creates another series, so label design directly affects memory, storage, and query cost. - **PromQL aggregation and rates.** PromQL supports counter rates, ratios, joins, and time-window calculations. For counters, apply rate before aggregating across instances so resets are detected separately for each source. - **Latency distributions.** Histograms support latency distributions and aggregatable quantile estimates; classic histograms expose cumulative bucket series. Client-computed summary quantiles generally cannot be combined into a valid fleet-wide percentile. - **Recording and alerting rules.** Recording rules reduce repeated query work, while alerting rules express operational conditions over metrics. A for duration requires a condition to remain active across evaluations, helping filter brief spikes. - **Exporter ecosystem.** Exporters make infrastructure and third-party services observable without changing their internal implementations. Exporter availability and collection cost become part of the monitoring system's failure and capacity model. ## Scaling, replication and failure Scale first by controlling active series, label churn, scrape frequency, and expensive queries. Estimate ingestion as samples per scrape divided by scrape interval, summed across targets; classic histogram buckets and label combinations can multiply that volume substantially. Recording rules reduce repeated query work but also create additional stored series. Partition scraping by cluster, region, team, or target group when one server is insufficient. Standalone Prometheus does not automatically distribute storage and queries across servers. Systems such as Thanos or Grafana Mimir can provide broader query and storage capabilities, with different architectures and operating costs. For high availability, run independent Prometheus replicas scraping the same targets. They do not synchronously replicate local databases or use consensus, so their samples and rule evaluations can differ. A compatible query layer or remote backend can deduplicate replica data; Alertmanager can deduplicate equivalent alerts when replica-specific labels are removed from the alert identity. Scrape failures create observation gaps, and restarting a target does not recover measurements that were never collected. The WAL supports recovery of locally ingested data, not replay of missed scrapes. Remote write can buffer and retry during an outage, but it is not an unlimited durable queue: prolonged failures can exceed the recoverable window, so monitor lag and consult the deployed version's behavior. ## Limits and defaults worth knowing - **Default scrape interval:** The global scrape_interval defaults to 1 minute. It is configurable globally and per scrape job. - **Default scrape timeout:** scrape_timeout defaults to 10 seconds and cannot exceed the scrape interval. It is configurable. - **Default local retention:** If neither time-based nor size-based retention is configured, local storage defaults to 15 days of retention. Retention is configurable; local storage is not replicated by Prometheus itself. - **Default query timeout:** The query.timeout flag defaults to 2 minutes. It is configurable and does not guarantee that a large query will finish within that time. - **Default lookback delta:** The query lookback defaults to 5 minutes and is configurable. Instant selectors use the latest eligible sample within this window; a staleness marker can make a series disappear sooner. ## Use it when - You need operational metrics and alerts for request rate, errors, latency, saturation, and service availability. - Targets can expose scrape endpoints, and the monitoring network can reach them. - You need label-based aggregation across dynamic workloads such as Kubernetes pods. - You can bound label cardinality and tolerate gaps in monitoring data during collection failures. - You want local monitoring per cluster or region, optionally combined with a separate long-term metrics backend. ## Avoid it when - You need an exact, replayable ledger for billing, payments, or other business transactions. - Your primary data consists of logs, trace spans, or arbitrary event payloads rather than numeric time series. - You need a separate metric series for every user, request, or transaction at large scale. - You require automatic distributed storage, replication, and global queries from standalone Prometheus without additional components. ## What interviewers ask about it ### How would you estimate Prometheus capacity? Estimate active series from metric families and actual label combinations, then derive samples per second from scrape frequency. Account for series churn, query concurrency, recording rules, retention, and histogram buckets; validate memory and disk requirements with representative workloads rather than a universal per-server limit. ### What happens when a scrape fails? Prometheus records up as 0 for the failed target and does not receive its application samples for that scrape. A later counter scrape can still capture cumulative growth if the counter survived, but intermediate gauge values and event timing are lost. ### How do you make monitoring highly available? Use independent Prometheus replicas and a highly available Alertmanager deployment, with monitoring for the monitoring system itself. Normalize replica-specific alert labels for deduplication, and use a compatible backend or query layer if users need a deduplicated view of both replicas. ### How do you distinguish an unhealthy service from missing telemetry? Alert separately on application symptoms, scrape failures, and missing expected series using expressions such as absent or absent_over_time. A target removed from service discovery may disappear rather than remain as up equal to 0, so compare against an expected inventory or another independent signal when necessary. ### How would you compute fleet-wide p99 latency? With classic histograms, calculate per-bucket counter rates, aggregate across instances while preserving the le label, and apply histogram_quantile. The result is an estimate whose accuracy depends on bucket boundaries; averaging instance-level p99 values is not a valid replacement. ### When would you use federation instead of remote write? Use federation to let another Prometheus scrape selected raw or preaggregated series, often for a hierarchical overview. Use remote write to forward samples to a compatible backend for centralized storage or querying; neither path should be assumed to provide unlimited buffering or automatic deduplication. ## Common mistakes - Putting user IDs, request IDs, raw URLs, or unbounded error messages in labels instead of using bounded dimensions and linking to logs or traces. - Treating HA replicas as a synchronously replicated database, or summing both replicas and double-counting the same workload. - Averaging percentiles or applying rate after aggregating counters across instances. - Assuming no alert means the service is healthy without checking scrape health, missing series, rule evaluation, and notification delivery. ## Alternatives - **Grafana Mimir:** Choose it when Prometheus-compatible metrics need a shared, horizontally scalable backend with multi-tenancy and long retention. It adds distributed-system components and operational complexity. - **Thanos:** Choose it when independent Prometheus deployments need a unified query view, replica deduplication, and object-storage-backed historical data. - **Managed metrics services:** Choose a managed service when reducing storage and monitoring operations matters more than full infrastructure control. Evaluate ingestion pricing, cardinality limits, query behavior, and data-residency requirements. ## Interview questions where it comes up - [Design a metrics monitoring system like Datadog](https://sysround.com/questions/design-metrics-monitoring-system): Prometheus is a concrete reference for scrape-based ingestion, label cardinality, time-series storage, query evaluation, and missing-data-aware alerts. - [Design a control plane for a distributed database](https://sysround.com/questions/design-database-control-plane): Prometheus can monitor fleet health, resource saturation, replication lag, and reconciliation failures, while authoritative lifecycle state remains in the control plane's durable store. - [Design an on-call escalation system like PagerDuty](https://sysround.com/questions/design-pagerduty): Prometheus and Alertmanager can supply grouped alerts, while durable escalation deadlines, acknowledgments, and provider-delivery handling belong in the escalation service. ## FAQ ### Is Prometheus push or pull? Its normal collection model is pull: Prometheus scrapes HTTP endpoints on a schedule. Pushgateway supports limited cases such as service-level batch-job outcomes, but it is not a general replacement for scraping and requires explicit lifecycle management for retained metrics. ### What is the difference between Prometheus and Grafana? Prometheus collects and stores metrics, runs PromQL queries, and evaluates alerting rules. Grafana provides dashboards and visualization across Prometheus and other data sources, and can also evaluate alerts. Grafana is not the storage engine for Prometheus metrics. ### Does Prometheus support long-term storage? Local retention is configurable, so Prometheus can retain data longer if disk capacity allows. However, its local database is neither clustered nor replicated, and longer retention does not address those limits. Designs needing durable shared history commonly add a remote backend or an object-storage-based system such as Thanos. ## Related concepts - [Observability](https://sysround.com/concepts/observability) - [Time-series databases](https://sysround.com/concepts/time-series-databases) - [Service discovery](https://sysround.com/concepts/service-discovery) - [SLIs, SLOs and SLAs](https://sysround.com/concepts/sli-slo-sla) - [Write-ahead log](https://sysround.com/concepts/write-ahead-log) --- SysRound (https://sysround.com) is an AI practice tool for system design interviews: a voice or typed AI interviewer, a shared whiteboard, then feedback on every skill and a model answer. Its feedback is AI-generated and can be wrong.