Redis vs Kafka: System Design Trade-offs

Side by side

Dimension	Redis	Kafka
Primary role	Redis: in-memory data structures and messaging	Kafka: partitioned, replicated event log
Messaging models	Pub/Sub for live delivery; Streams for retained messages	Retained topics with independent consumer groups
Durability	Streams depend on persistence and replication settings; Pub/Sub has no retained history	Depends on replication, producer acknowledgments, and replica settings
Replay and retention	Streams replay retained entries; capacity is typically memory-bound	Consumers seek within retained logs; disk-based retention
Consumption and recovery	Streams use acknowledgments and pending-entry recovery	Consumers track offsets; recovery resumes from committed progress
Ordering and scaling	Stream-ID order within one stream; shard across stream keys	Offset order within one partition; scale across partitions
State access	Direct key lookups, counters, sets, and sorted sets	Build a materialized view for application lookups
Slow consumers	Pub/Sub cannot catch up; Streams require retained entries and recovery logic	Consumers accumulate lag but must catch up before retention expires

The differences that matter

Redis is not one messaging model. Pub/Sub is useful for transient signals when a missed message is acceptable or recoverable elsewhere. Streams retain entries and support consumer groups, acknowledgments, and claiming abandoned work. A durable worker design must use Streams or another durable queue, not assume Pub/Sub stores messages.

Retention changes the capacity plan. Standard Redis Streams keep their working dataset in memory, with persistence configured separately. Kafka stores retained logs on disk and lets multiple consumer groups progress independently. Estimate event bytes per second, retention duration, replication, and recovery time before choosing either.

Failure recovery is part of the application design. With Redis Streams, workers must acknowledge completed entries and recover pending entries left by failed workers. With Kafka, consumers must coordinate processing with offset commits. Both can redeliver work after failures, so database writes, payments, and notifications still need idempotency.

They often belong in different layers rather than competing for one slot. Kafka can carry activity events to analytics, search indexing, and fraud consumers, while Redis holds the latest counters or cached results. Kafka does not replace Redis's direct state access, and a Redis cache does not replace a replayable event history.

Choose Redis when

You need shared state such as expiring sessions, rate-limit counters, presence, or leaderboards rather than an event log.
You need live signals where missed delivery is acceptable, such as a refresh hint backed by periodic state reads.
You need a bounded retained work stream, already operate Redis, and can implement pending-entry recovery, safe trimming, and appropriate persistence.
You need short-lived event history and have verified that its memory footprint and failure behavior fit the requirements.

Choose Kafka when

Several independent services need to consume the same events at their own pace.
Consumers must replay retained history to rebuild derived state or recover from processing bugs.
The event backlog or retention window makes memory-based storage an unattractive capacity choice.
You need a partitioned ingestion backbone for analytics or stream processing, with explicit consumer lag and recovery planning.

A worked example

Analytics history versus a live dashboard

Suppose a service produces 20,000 activity events per second at 500 bytes each. That is about 864 GB per day, or 2.6 TB for three days before replication, compression, and overhead. Analytics, fraud detection, and dashboard aggregation each need independent progress, and analytics must replay after a bad deployment. Kafka fits that retained event backbone. A dashboard consumer can update Redis counters so requests read current totals without scanning events. Redis Pub/Sub could send optional refresh hints, but clients should fetch current state after reconnecting. Keeping the entire three-day history in standard Redis Streams would require a substantial memory budget and an explicit persistence and failover plan.

In the interview

Start with the requirement: transient fan-out, durable work distribution, retained event history, or shared state. If you propose Redis for messaging, explicitly name Pub/Sub or Streams.
Quantify event rate, payload size, retention, consumer count, and maximum consumer outage. Expect a follow-up on backlog size and how quickly a recovered consumer can catch up.
Describe the failure timeline: a worker performs a side effect, then crashes before acknowledging or committing progress. Explain how idempotency prevents duplicate effects.
State the ordering boundary and partition key. Kafka orders within a partition; Redis Streams order entries within a stream, but parallel workers can complete out of order. Expect questions about hot keys and per-entity serialization.
Name the durability configuration rather than saying either system is simply durable. Discuss Redis persistence and failover loss windows, or Kafka replication, acknowledgments, minimum in-sync replicas, and behavior when replicas are unavailable.

Common mistakes

Comparing Redis Pub/Sub with Kafka without mentioning Redis Streams, or treating all Redis messaging as durable.
Claiming either system gives exactly-once external side effects automatically. Kafka transactions have a defined scope; an external database or provider still needs a coordination or idempotency strategy.
Assuming acknowledgment deletes a Redis Streams entry, or consuming a Kafka record deletes it. Retention and trimming are separate from consumer progress.
Choosing by generic speed claims while ignoring retention cost, consumer recovery, partition skew, and acceptable data loss.

Where it comes up

Interview questions where choosing between Redis and Kafka comes up.

Questions people ask

Can Redis replace Kafka?

Sometimes, if the requirement is transient Pub/Sub or a bounded retained stream whose capacity and recovery needs fit Redis Streams. It is not a drop-in replacement for a partitioned log with substantial retention and many independent consumers. Compare the messaging mode and workload, not just the product names.

What is the difference between Redis Streams and Kafka?

Both retain ordered entries and support consumer groups, but their storage and consumption models differ. Redis Streams use stream IDs, pending-entry tracking, acknowledgments, and explicit recovery of abandoned work. Kafka uses partition offsets and consumer-group assignments, with disk-based retention independent of consumption.

Is Redis faster than Kafka?

There is no useful universal answer because a Redis key lookup, Pub/Sub delivery, and a replicated Kafka append do different work. Latency depends on batching, persistence, replication, payload size, and load. In an interview, compare configurations that satisfy the same durability and delivery requirements rather than quoting unrelated benchmarks.