# Apache Kafka for system design interviews > Apache Kafka is a distributed, durable event log that lets producers publish records and multiple applications consume them independently. It fits designs that need asynchronous processing, replay, or stream processing, especially when several downstream systems need the same events. Source: https://sysround.com/tech/kafka Updated: 2026-11-05 ## How it works A topic is divided into partitions, each an ordered, append-only log. Every record receives an offset within its partition, not a global sequence number. Producers choose a partition directly or through a partitioner, commonly using a business key such as order_id to keep related records together. Brokers store partitions as log segments with indexes that help locate offsets and timestamps. Producers batch records, optionally compress them, and send them to partition leaders. Consumers fetch batches from brokers; batching, sequential storage access, and the operating system page cache are important parts of Kafka's architecture. Each partition has one leader and a configurable set of replicas. Followers copy the leader's log, and the in-sync replica set, or ISR, tracks replicas sufficiently caught up to participate in normal failover. With acks=all, the producer waits for acknowledgment from all current ISR members; min.insync.replicas determines the minimum ISR size required to accept these writes. In a conventional consumer group, each partition is assigned to at most one consumer at a time. Different groups read the same topic independently, while consumers within a group divide its partitions. Committed offsets record a group's progress, usually in Kafka's internal __consumer_offsets topic, but do not prove that an external side effect completed. Modern Kafka uses KRaft controllers to maintain cluster metadata through a Raft quorum. Controllers manage metadata and partition leadership, while brokers serve record traffic. This metadata consensus mechanism is separate from the leader-follower replication protocol used for topic data. ## What it gives you in a design - **Independent consumers and replay.** Analytics, indexing, fraud detection, and other applications can consume the same events using separate groups. A group can reset its offsets to replay available history without asking producers to resend it. - **Key-based ordering.** Kafka preserves log order within a partition, not across a topic. Key related events together and preserve that order during processing if the business requires sequential updates. - **Retention and log compaction.** Delete-based retention removes old log segments according to time or size settings. Compaction eventually retains the latest value for each key, with tombstones representing deletions; it is not immediate deduplication or a guarantee that every intermediate event remains replayable. - **Idempotent producers.** Producer IDs and per-partition sequence numbers let brokers suppress duplicate appends caused by producer retries. This does not deduplicate the same business operation submitted again as a new application-level send. - **Transactions and stream processing.** Transactions can atomically publish records across Kafka partitions and commit consumed offsets. Together with read_committed consumers, they support exactly-once processing for appropriately configured Kafka-to-Kafka pipelines, not arbitrary external side effects. - **Connectors and change data capture.** Kafka Connect provides a framework for moving records between Kafka and external systems. A database outbox captured through CDC can publish business events without an unsafe database-plus-Kafka dual write, though connector delivery guarantees still need examination. ## Scaling, replication and failure Partitions distribute storage and traffic across brokers and bound useful parallelism within a conventional consumer group. Estimate ingress bytes, retention storage, replication overhead, and consumer read traffic separately. Adding brokers does not automatically redistribute all existing partitions; placement and reassignment need planning. A heavily used key can overload one partition even when other brokers have spare capacity. More partitions do not split that key's traffic. Salting or splitting the key can spread load, but then ordering and aggregation must be reconstructed; increasing a topic's partition count can also change key-to-partition mapping. For a common durability-oriented configuration, use replication factor 3, min.insync.replicas=2, acks=all, and replicas spread across failure domains. If the ISR drops below two, writes fail rather than accept weaker durability. Safe leader election avoids promoting an out-of-sync replica, trading availability for protection against acknowledged-data loss under replica failures. Slow consumers build lag until they catch up or retention removes the records they need. Monitor lag in both offsets and time, disk headroom, replication health, and recovery throughput. Cross-region replication is commonly asynchronous, so regional failover needs an explicit data-loss window, duplicate-handling strategy, and plan for translating consumer progress. ## Limits and defaults worth knowing - **Ordering boundary:** Ordering is per partition. Kafka provides no total order across all partitions in a topic. - **Consumer-group parallelism:** For a conventional consumer group, at most one consumer owns a partition at a time. With P partitions, at most P consumers can actively consume assigned partitions in that group. - **Default time retention:** The broker default log.retention.hours is 168 hours, or 7 days, for delete-based retention when not overridden. Retention is configurable, operates on segments, and is not an exact per-record expiration deadline. - **Default record-batch size limit:** The broker's default message.max.bytes is approximately 1 MiB and limits a record batch, after compression when enabled. Larger batches require coordinated topic or broker, producer, and consumer size settings. - **Default replication settings:** The broker defaults default.replication.factor and min.insync.replicas are both 1. These are configurable and should not be mistaken for a production durability policy. - **Replay boundary:** Replay is limited to records still retained. For compacted topics, older values for a key may already have been removed, so replay does not necessarily recover the full event history. ## Use it when - Several independent downstream applications need the same durable event stream and must track their own progress. - Consumers need to rebuild indexes, projections, or aggregates by replaying retained events. - The design includes continuous aggregation, enrichment, joins, or other stream processing. - Traffic bursts should be buffered durably while downstream systems catch up within a defined retention window. - Related events need per-key ordering, but a single global order is unnecessary. ## Avoid it when - The main requirement is scheduled delivery, per-message priority, or individual job acknowledgment and redelivery. A task queue or scheduler is often a more direct fit. - The system needs synchronous request-response behavior or low-latency point lookups rather than sequential event consumption. - A small workload needs a simple background queue and cannot justify Kafka's operational complexity. - The proposal assumes Kafka alone can make a database update and an external payment or email exactly once. ## What interviewers ask about it ### How would you choose a partition key and partition count? Start with the ordering boundary, such as an order or account, then estimate peak traffic, key skew, and required consumer parallelism. Leave growth headroom, but account for the replication, metadata, and operational cost of more partitions. Explain that adding partitions can remap keys and requires care if ordering must survive the transition. ### What happens if a consumer crashes after processing but before committing its offset? Another consumer can read the record again from the last committed offset, producing at-least-once processing. Make the effect idempotent, for example by recording an event ID in the same database transaction as the business update. Committing first instead can lose work if the consumer crashes before applying the effect. ### Does acks=all mean every replica has stored the record? It means all replicas currently in the ISR have acknowledged it, not every configured replica. With min.insync.replicas=1, a degraded partition can still accept a write with only its leader in the ISR. State the replication factor, minimum ISR, acknowledgment mode, and leader-election policy together. ### Does Kafka provide exactly-once delivery? Idempotent production prevents duplicate appends from retries, while transactions can atomically commit Kafka output and input offsets. Consumers using read_committed can avoid seeing aborted transactional records. External effects still need an atomic integration mechanism, idempotency, or reconciliation. ### How do retries affect ordering and poison-message handling? Retrying in place preserves partition processing order but can block unrelated keys sharing that partition. Retry topics and dead-letter topics let processing continue, but may reorder events for the affected key. Choose explicitly between strict ordering and progress, and define how failed records are inspected and replayed. ### How do you publish events reliably after a database transaction? Write the business change and an outbox row in the same database transaction, then publish the outbox through CDC or a relay. The relay can deliver duplicates, so consumers should remain idempotent. Two independent writes to the database and Kafka leave a failure window regardless of which write happens first. ## Common mistakes - Claiming topic-wide ordering, or assuming partition order automatically survives concurrent asynchronous processing inside a consumer. - Treating offset commits as proof that database writes, emails, or payments happened exactly once. - Assuming consumers remove messages or that retention waits for every consumer to finish. - Adding Kafka without naming the partition key, consumer groups, durability settings, lag budget, and retry policy. ## Alternatives - **RabbitMQ:** Often a better fit for work queues that need flexible routing, individual acknowledgments, and job redelivery rather than a retained log with independent replay. - **Amazon SQS:** Fits managed asynchronous work distribution when visibility timeouts and low operational overhead matter more than Kafka-style retained replay and stream processing. - **Apache Pulsar:** Worth considering when durable streaming and queue-style subscriptions are both important, or when its separation of serving and storage fits the operating model. - **Database outbox with workers:** Can be enough for modest asynchronous workloads whose tasks must be created atomically with database changes, without operating a separate streaming platform. ## Interview questions where it comes up - [Design a distributed message log like Kafka](https://sysround.com/questions/design-kafka): Kafka is a concrete reference for partitioned logs, replication, safe failover, independent offsets, and retention. - [Design an ad click aggregator](https://sysround.com/questions/design-ad-click-aggregator): Kafka can buffer click events for replayable aggregation while forcing explicit decisions about duplicates, hot keys, and late events. - [Design a notification system](https://sysround.com/questions/design-notification-system): Kafka can distribute notification events to independent channel processors, with explicit trade-offs around retries, ordering, and idempotent delivery. - [Design Facebook post search](https://sysround.com/questions/design-facebook-post-search): A Kafka-backed change stream can decouple durable post updates from asynchronous search indexing and support index rebuilds. - [Design Google Analytics](https://sysround.com/questions/design-google-analytics): Kafka provides a retained ingestion stream for independent sessionization, aggregation, and archival consumers. - [Design a large data migration to the cloud](https://sysround.com/questions/design-cloud-data-migration): Kafka can retain CDC events during a bulk copy, provided the design tracks source positions, retention headroom, and cutover correctness. ## FAQ ### Is Kafka a message queue or a database? Kafka is primarily a distributed event log. Consumer groups provide queue-like work sharing, but consumption does not delete records, and multiple groups can replay the same retained data. It can support event-sourced systems, but it is not a general-purpose query database. ### What is the difference between a Kafka topic and a partition? A topic is a named stream of records, and partitions divide that stream for storage and parallel processing. Each partition has its own offsets, leader, replicas, and order. A topic with multiple partitions has no single global offset or total order. ### How does Kafka handle backpressure? Consumers pull records at their own pace, so slower processing usually appears as growing consumer lag rather than immediately blocking producers. That buffer is finite because retention and disk capacity are finite. Use bounded processing queues, pause and resume consumption where appropriate, and scale or shed load before consumers fall behind the retained history. ## Related concepts - [Partitioning strategies](https://sysround.com/concepts/data-partitioning) - [Database replication](https://sysround.com/concepts/replication) - [Message ordering](https://sysround.com/concepts/message-ordering) - [Delivery guarantees](https://sysround.com/concepts/delivery-guarantees) - [Transactional outbox pattern](https://sysround.com/concepts/outbox-pattern) --- SysRound (https://sysround.com) is an AI practice tool for system design interviews: a voice or typed AI interviewer, a shared whiteboard, then feedback on every skill and a model answer. Its feedback is AI-generated and can be wrong.