# Log aggregation > Log aggregation collects logs from many application instances into a shared system for searching and investigating failures. In system design interviews, the key is to make logs useful across services without letting logging failures slow down the application or exhaust its disks. Source: https://sysround.com/concepts/log-aggregation Updated: 2026-11-05 ## How log aggregation works Applications emit structured records, usually JSON, with a timestamp, severity, service, environment, and message. Add request or trace IDs so an engineer can connect a checkout failure to related records from the payment and inventory services; avoid logging secrets or sensitive payloads. An agent on each host or node reads application files or container output, adds metadata, and ships records in batches. Bounded memory buffers and local disk spooling absorb brief outages; file offsets and rotation handling determine whether records are skipped or resent after a restart. Collectors accept batches, authenticate senders, and apply limits. At larger scale, a durable message queue separates ingestion from parsing and indexing, allowing downstream workers to fall behind temporarily without immediately blocking collection. Processing workers parse records, normalize fields, redact sensitive data, and route malformed records to a bounded quarantine. Retries can create duplicates, so either tolerate duplicate search results or use stable record IDs and deduplication where its cost is justified. Store recent logs in a searchable backend, commonly partitioned by time and sometimes tenant. Index selected fields and message text for interactive queries; keep compressed older logs in object storage when slower retrieval is acceptable. Users narrow searches by time range, service, severity, and request ID before searching message text. Track ingestion delay and dropped records separately: an empty query result may mean there were no errors, or it may mean the logging pipeline stopped delivering them. ## Worked example: Sizing a centralized logging pipeline Suppose 500 application instances each emit 200 records per second, averaging 500 bytes per record. That is 100,000 records per second, about 50 MB/s or 4.32 TB/day before compression, indexing, and replication. A ten-minute downstream outage creates about 30 GB of backlog at that rate. Agents batch records into collectors, which acknowledge only after a replicated queue accepts them. Indexers retain three days of searchable logs and archive compressed records for 30 days. Provision recovery capacity above the live arrival rate so the backlog can drain. If agent spools fill, drop designated low-priority diagnostic records and report the loss; required audit records need a separate durability policy. ## Trade-offs - **Application isolation versus log completeness.** Bounded asynchronous logging protects request latency, but prolonged outages eventually require dropping records, blocking producers, or providing more durable buffering. - **Search speed versus indexing cost.** Indexing more fields improves interactive searches but increases CPU, storage, and write amplification, especially with uncontrolled dynamic field names. - **Retention versus cost.** Short searchable retention with compressed archives lowers cost but makes older investigations slower and may require restoring data before querying. - **Delivery reliability versus complexity.** Durable buffering and retries reduce loss after acceptance but add operational work and duplicate records, without guaranteeing that every application-generated record survives. - **Diagnostic detail versus privacy.** Rich context helps debugging but requires source-side data minimization, redaction, access controls, and explicit deletion and retention policies. - **Shared infrastructure versus tenant isolation.** Shared collectors and indexes improve utilization, but per-tenant quotas and resource limits are needed to keep one noisy producer from degrading everyone else's logs. ## When to use it in a system design interview - The design has many replicas or short-lived containers, making individual machine logs difficult to retrieve. - An incident investigation needs to correlate records across several services using a request or trace ID. - The interview asks for a centralized logging platform with high write volume, searchable recent data, and longer-term archives. - Operational requirements include debugging failures after the original application instance has disappeared. ## How to bring it up in the interview - Start with requirements: peak records per second, average record size, searchable retention, archive retention, acceptable loss, and ingestion-to-search delay. - Draw the path from application to agent to collector, then add a durable queue only if burst absorption, replay, or independent consumer scaling justifies it. - State exactly what an acknowledgment means: accepted into memory, persisted locally, or replicated durably; interviewers will probe what survives each kind of crash. - Explain what happens when the backend is unavailable or local disks fill: bounded buffers, severity-aware dropping, quotas, and alerts should be explicit rather than implied. - Defend the query model and partitioning strategy, then expect follow-ups about expensive broad searches, noisy tenants, access control, and missing or late records. ## Common mistakes - Making synchronous network logging part of every request's critical path without an explicit latency and failure policy. - Claiming a durable queue prevents all log loss, even before an agent has read or durably buffered a record. - Indexing every JSON field by default and ignoring uncontrolled field growth, indexing overhead, and retention costs. - Treating timestamps from different machines as a reliable global order instead of accounting for clock skew and recording ingestion time separately. - Using diagnostic logs as the sole financial ledger or compliance audit trail without the required completeness, integrity, and access guarantees. ## FAQ ### What is the difference between log aggregation and centralized logging? The terms are often used interchangeably. Log aggregation emphasizes collecting records from many sources, while centralized logging usually includes their shared storage, search, retention, and access controls. Centralized access does not require a single physical server. ### How is log aggregation different from metrics and distributed tracing? Logs capture individual events and their details, metrics summarize numeric measurements over time, and traces connect operations along a request's path. They complement each other: a metric can reveal an error spike, a trace can locate the slow dependency, and correlated logs can explain the failure. ### How do you prevent log loss in a centralized logging system? Reduce loss with durable local spooling, rotation-aware agents, replicated ingestion, retries, and enough capacity to recover from outages. Define durability at each handoff, because records can still disappear before reaching durable storage. With finite buffers and a producer that cannot block, loss remains possible during a long enough outage. ## Related concepts - [Observability](https://sysround.com/concepts/observability) - [Distributed tracing](https://sysround.com/concepts/distributed-tracing) - [Inverted index](https://sysround.com/concepts/inverted-index) - [Backpressure](https://sysround.com/concepts/backpressure) --- SysRound (https://sysround.com) is an AI practice tool for system design interviews: a voice or typed AI interviewer, a shared whiteboard, then feedback on every skill and a model answer. Its feedback is AI-generated and can be wrong.