How it works
Producers write records containing a partition key and a data payload. Kinesis hashes the partition key into a shard's hash range, so records with the same key go to the same shard while that shard layout is unchanged. Each shard is an ordered sequence of records with assigned sequence numbers; there is no stream-wide order.
Kinesis stores records durably across three Availability Zones within an AWS Region. Records remain available for a configured retention period, regardless of whether consumers have read them. Reading a record does not remove it, so analytics, fraud detection, and archival applications can maintain independent progress.
Consumers read each shard through shared-throughput polling or enhanced fan-out subscriptions. Shared-throughput consumers compete for a shard's read capacity, while enhanced fan-out gives each registered consumer dedicated read throughput. Applications must still handle duplicate processing, failures, and checkpoint recovery.
The Kinesis Client Library distributes shard processing across workers and uses DynamoDB for leases and checkpoints. A checkpoint records how far processing has completed, not whether every external side effect happened exactly once. When shards split or merge, consumers must respect parent-child shard relationships to preserve processing order.
Provisioned mode makes shard capacity an explicit planning decision. On-demand mode manages capacity as traffic changes, but it does not remove partition-key skew, service quotas, or the possibility of throttling during abrupt traffic changes. Amazon Managed Service for Apache Flink can process Kinesis records when the design needs stateful windows, joins, or event-time handling.
What it gives you in a design
- Replayable event retention
- Consumers can start from retained historical positions and reprocess records without affecting other consumers. Retention is time-based, so it is a recovery window rather than permanent archival storage.
- Partitioned ordering
- Shards provide an ordered log, and partition keys let related events share that log. Concurrent writes, retries, and batch publishing can still make arrival order differ from business-event order.
- Independent consumer applications
- Different applications can maintain their own positions over the same records. Enhanced fan-out reduces read-capacity contention between applications, at an additional cost.
- Managed regional durability
- AWS manages replication across Availability Zones and the underlying storage infrastructure. Applications still need retry logic, replay-safe processing, and a separate plan for regional disaster recovery.
- Provisioned and on-demand capacity
- Provisioned mode exposes shard capacity for predictable workloads and explicit sizing. On-demand mode reduces capacity-management work, but key distribution and burst behavior still matter.
- AWS processing integrations
- Lambda can consume stream records, and Managed Service for Apache Flink supports stateful stream processing. These integrations simplify deployment but do not automatically make writes to downstream systems exactly once.
Scaling, replication and failure
Size ingestion using both bytes per second and records per second, then account for uneven key distribution. In provisioned mode, add shards by splitting hash ranges and reduce them through merges. More workers help only when the application has enough shards or supported processing concurrency to keep them busy.
A hot partition key still maps to one shard at a time. Adding capacity elsewhere does not fix it; consider salting the key, pre-aggregating events, or separating unusually large tenants. Salting trades away a single ordered path for that entity, so explain how downstream processing handles the resulting merge.
Scale reads separately from writes. Shared polling consumers divide shard read capacity, while enhanced fan-out provides dedicated throughput per registered consumer per shard. Monitor consumer lag and ensure recovery throughput exceeds the incoming event rate, otherwise a backlog cannot shrink.
Plan for at-least-once processing: producer retries can create duplicate records, and consumer recovery can repeat work after the last checkpoint. Use stable event IDs and idempotent or transactional sink updates. Regional replication is not automatic cross-Region recovery, so define replication, deduplication, and recovery-point expectations separately.
Limits and defaults worth knowing
Check the current documentation before relying on an exact value; many of these are configurable.
| Retention | The default retention period is 24 hours. It is configurable up to 365 days, with additional charges for longer retention. |
|---|---|
| Standard shard write capacity | A shard supports sustained writes of up to 1 MB per second or 1,000 records per second. Plan against both limits and allow headroom for skew. |
| Shared-throughput reads | A shard provides up to 2 MB per second of shared read throughput. GetRecords is limited to five calls per second per shard, shared across polling consumers. |
| Enhanced fan-out reads | Each registered enhanced fan-out consumer receives up to 2 MB per second of dedicated read throughput per shard. Consumer-count quotas depend on the stream mode and applicable service quotas. |
| Shard iterator lifetime | A shard iterator expires five minutes after it is returned. Polling clients must advance iterators and recover from expiration; the Kinesis Client Library manages this behavior. |
Use it when
- An AWS-based design needs durable event ingestion with several independent downstream applications.
- Consumers need replay after failures, deployment mistakes, or changes to aggregation logic.
- Ordering matters within an entity or partition, but global ordering is unnecessary.
- The workload is a continuous stream of clicks, telemetry, metrics, or location events rather than individually acknowledged work items.
- Managed infrastructure is more valuable than Kafka protocol compatibility or detailed broker control.
Avoid it when
- The requirement is a task queue with per-message acknowledgement, visibility timeouts, delayed delivery, and straightforward dead-letter routing.
- All events must have one global order at a throughput that exceeds a single shard's capacity.
- The system requires native Kafka APIs, portability across clouds, or control over broker storage and replication settings.
- Records must remain queryable indefinitely without a separate archive or database.
What interviewers ask about it
Follow-ups you can expect once Kinesis is on your whiteboard, and the core of a strong answer.
- How would you choose the partition key?
- Start with the required ordering boundary, such as a device or account, then estimate the traffic of the busiest key rather than just the average. If one key can exceed shard capacity, change the ordering requirement, salt the key, or pre-aggregate before publishing.
- Does Kinesis guarantee event ordering?
- Each shard exposes records in sequence order, but that is not necessarily the producer's intended business order. Concurrent writers and retries can change arrival order, and PutRecords does not guarantee request ordering; strict per-entity processing may require serialized publishing plus application sequence numbers.
- How do you avoid double-counting after a crash?
- Give events stable IDs and make the sink update idempotent, for example by atomically recording an event ID with its aggregate update. Checkpointing after a write avoids skipping work but leaves a crash window where the write can be repeated.
- What happens when a consumer falls behind?
- Track processing lag, throttling, and downstream latency, then identify whether compute, read capacity, or the sink is the bottleneck. Increase effective processing capacity and keep retention longer than the expected outage plus catch-up time, because unread records expire too.
- What changes when a shard splits?
- The parent shard eventually closes and its hash range is divided among child shards. A consumer that needs ordering must finish the parent before processing its children; the Kinesis Client Library coordinates these dependencies.
- How do producers handle partial batch failures?
- PutRecords can accept some entries and reject others, so inspect each entry's result and retry failed entries with backoff and jitter. Retrying only failures reduces unnecessary duplicates, but retries can change relative ordering and uncertain outcomes still require deduplication.
Common mistakes
- Calling Kinesis a work queue and assuming that reading acknowledges or deletes an individual record.
- Claiming exactly-once processing because the consumer stores checkpoints, without coordinating downstream side effects.
- Sizing from aggregate throughput while ignoring hot keys, competing readers, and the extra capacity needed to catch up after an outage.
- Treating on-demand capacity as unlimited instantaneous scaling or assuming multi-AZ replication provides cross-Region disaster recovery.
Alternatives
- Apache Kafka or Amazon MSK
- Choose this when Kafka compatibility, its ecosystem, portability, or broker-level configuration matters more than Kinesis's managed API and operating model.
- Amazon SQS
- Choose this for work distribution with message acknowledgement through deletion, visibility timeouts, and dead-letter queues rather than independent replayable readers.
- Amazon Data Firehose
- Choose this when the main requirement is managed delivery to destinations such as S3, with buffering and optional transformation, rather than application-controlled shard consumption.
- Google Cloud Pub/Sub
- Choose this for a Google Cloud design that favors managed topics and subscriptions over explicit shard-oriented capacity and consumption.
Where it comes up
Interview questions where Kinesis is a natural fit, and why.
- Design an ad click aggregatorKinesis can ingest click events for windowed aggregation, with partition skew, duplicate delivery, and replay-safe billing counts as central trade-offs.
- Design a metrics monitoring system like DatadogIt provides a durable ingestion layer between telemetry producers and independent aggregation, storage, and alerting consumers.
- Design Google AnalyticsIt supports replayable collection of page and session events while exposing tenant skew, late-event handling, and retry-safe counting.
- Design a top K trending systemIt can transport interaction events to streaming ranking workers that must handle duplicate events and time-window expiration.
- Design a distributed message log like KafkaIt is a concrete managed example for discussing partition ordering, independent consumer progress, resharding, and retention under slow readers.
- Design an A/B testing platformIt can carry exposure and outcome events to attribution consumers, where deduplication and replay are necessary for trustworthy experiment results.
Questions people ask
What is Amazon Kinesis used for?
Amazon Kinesis Data Streams ingests and retains continuous event data for independent consumers. Common uses include click analytics, telemetry processing, anomaly detection, and feeding real-time aggregates. It is the event transport and replay layer, not the database that serves analytical queries.
What is the difference between Amazon Kinesis and Kafka?
Both support partitioned, replayable event streams with independent consumers. Kinesis Data Streams is an AWS-managed service with shard-based capacity and AWS-specific APIs, while Kafka provides a broader ecosystem and more deployment and configuration control. Neither choice removes the need to handle skew, lag, duplicates, and sink correctness.
What is the difference between Kinesis Data Streams and Amazon Data Firehose?
Data Streams exposes a retained log that applications consume and replay using their own positions. Data Firehose, formerly Kinesis Data Firehose, focuses on managed delivery to configured destinations. A design can use Data Streams for live processing and Firehose to deliver a copy into S3.