AWS Enterprise Architecture: Real-Time Streaming

In a nutshell

Imagine a DVR quietly recording a live broadcast. It writes every frame down in order, keeps the tape for a while, and lets two very different people watch the same recording without fighting over it: someone in the control room reacting to what is on screen right now, and an editor next month scrubbing back through the archive to cut a highlight reel. Neither one blocks the other, and you can always rewind. That DVR tape is a real-time stream, and getting the recording right — durable, ordered, replayable, shared — is the whole game. On AWS the tape is Amazon Kinesis Data Streams.

Real-time streaming is how a system reacts to events while they still matter — a fraud spike, a crashing game build, a sensor reading that is physically impossible — instead of finding out tomorrow from a batch report. The trap almost every team falls into is treating the stream like a queue to drain as fast as possible with one do-everything worker. The architecture in this article does the opposite: it puts a durable log in the middle and fans it out to two paths that never touch each other — a hot path that lights up dashboards in seconds, and a cold path that lands every raw event cheaply in S3 for months. Because the log is replayable, any downstream store you lose or change, you simply rebuild by replaying the tape.

If the coming talk of shards, partition keys, iterator age, and enhanced fan-out is new to you, don’t worry — the next few sections teach every one of those primitives from zero, with worked numbers, before the reference architecture leans on them. Read top to bottom the first time; come back to the component table and Terraform as a reference later.

Level: Advanced reference with a beginner on-ramp · Time: ~61 min read

Before you start, it helps to know:

The basics of AWS Lambda (a function that runs in response to an event) and Amazon S3 (object storage). If Lambda is fuzzy, skim AWS Lambda deep dive.
What a queue is and why at-least-once delivery, ordering, and a dead-letter queue matter — see SQS, SNS & fan-out.
Rough familiarity with JSON events and IAM roles. No streaming experience assumed.

After this lesson you will be able to:

Explain what a shard, a partition key, and iterator age are — and size a stream straight from a throughput number.
Choose between Kinesis Data Streams, Amazon Data Firehose, Amazon MSK, and DynamoDB Streams for the ingest layer, and say why.
Decide when a stateless Lambda is enough and when you need stateful windowing in Managed Service for Apache Flink.
Reason about ordering, at-least-once delivery, backpressure, DLQs, and replay well enough to design a pipeline that doesn’t fall over on Monday morning.

Real-time streaming on AWS goes wrong in a very specific way: a team picks Kinesis because “it’s the streaming service,” writes one fat Lambda that does ingest and enrichment and indexing and archival, hard-codes a shard count that was right for the demo, and then discovers six weeks later that a Monday-morning traffic spike is throttling producers, the Lambda is retrying the same poison-pill record forever, and OpenSearch is on fire because every record is being indexed one at a time. None of that is Kinesis’s fault. It is the absence of an architecture — a deliberate separation between the durable log, the processing tier, the hot operational store, and the cheap analytical landing zone, each scaling on its own axis. This article is that architecture, built end to end on Kinesis Data Streams, Lambda, Kinesis Data Firehose, and Amazon OpenSearch Service.

The single most important idea here is that the stream is a buffer and a contract, not a queue you drain as fast as possible. Kinesis Data Streams gives you a replayable, ordered-per-key, durable log. Everything downstream — the hot path that lights up OpenSearch dashboards in seconds, and the cold path that lands raw events in S3 for cheap reprocessing — reads from that same log independently. Get that split right and the system is boring in the best way: producers never feel consumer pressure, a slow consumer never backs up a fast one, and you can rebuild any derived store by replaying the log.

The business scenario

Picture an operator who has events but no way to act on them while they still matter. This is the same shape at 30 engineers and at 3,000.

The early version: a digital-native business — a payments processor, a multiplayer game studio, a connected-fitness brand, an ad-tech exchange — emits a firehose of events. Transactions authorise, players score, devices report heart rate, bids clear. Today those events trickle into a transactional database and someone runs a query the next morning. The business is steering by a rear-view mirror, and every interesting question is asked in the past tense.

Then the questions that the batch world structurally cannot answer start arriving, and they are the questions the business actually cares about:

Payments / fintech: “Decline rates for one card BIN range spiked 4x in the last 90 seconds. Is it a fraud attack, an issuer outage, or our own gateway degrading? We need the alert now, and we need to slice the last 15 minutes by issuer, merchant, and geography to triage.” End-of-day reconciliation finds this tomorrow; the fraud ring is gone by then.
Gaming: “A new build went out 10 minutes ago. Crash events from one device class just went vertical. Roll back before the review-bomb, and show me the live funnel of session-start → match-found → match-complete so I can see exactly where players are dropping.”
IoT / connected product: “These 200 devices on firmware 4.2 are reporting temperatures that are physically impossible — sensor fault or a real safety issue? Surface the anomaly on the ops wall in seconds, and keep two years of raw telemetry so we can prove what happened and retrain the model.”
Ad-tech / clickstream: “Win-rate on one exchange just collapsed. Is it a bid-shading change on their side or a latency regression on ours? We bill on this; minutes of blindness are real money.”

Every one of these shares the same structural requirement, and it is the requirement that defines the architecture: one event stream must simultaneously feed a sub-second operational view and a durable analytical archive, and the two must not interfere. The operations dashboard needs the last minutes-to-hours of events, indexed and queryable interactively, with alerting (the hot path). The data scientist, the analyst, and the compliance officer need every raw event, kept cheaply for months or years, reprocessable when the model or the schema changes (the cold path). Bolt both onto one engine and they fight: a heavy analytical scan starves the live dashboard; a burst of ingestion stalls everyone; a schema change forces you to re-ingest from producers you do not control.

The scale-invariance is the reason this belongs in an architecture center. A 30-person startup runs this with a 4-shard stream, a single small OpenSearch domain, and one Firehose. A global platform runs the identical topology with on-demand or 500-shard streams, a multi-AZ OpenSearch cluster with dedicated masters and UltraWarm, and Firehose dynamic partitioning into a partitioned data lake. The shape — durable log, fan-out to hot and cold, independent scaling — never changes. Only the dials move.

The promise to the business: events become actionable in seconds, nothing is ever lost, and any derived view can be rebuilt from the source of truth without going back to the producers.

Architecture overview

The architecture is a single durable log fanned out to two paths that scale independently, plus a thin alerting/serving layer on top. Read it as four tiers.

Tier 1 — Ingest into a durable, replayable log (Kinesis Data Streams). Producers — application SDKs, the Kinesis Producer Library (KPL), mobile/IoT via API Gateway → Kinesis, or other AWS services — write records into a Kinesis data stream. Each record carries a partition key (e.g. device_id, card_bin, session_id). Kinesis hashes that key to a shard; all records with the same key land on the same shard in order, which is what gives you per-entity ordering without a global bottleneck. The stream is the contract: it durably retains every record (default 24h, extendable to 365 days) across three Availability Zones, and — critically — it lets multiple independent consumers read the same records at their own pace. This is the property the whole design hinges on.

Tier 2 — The hot path: stream processing with Lambda. A Lambda function is wired to the stream as an event source mapping. The Lambda service polls the stream, batches records per shard, and invokes your function with up to one concurrent invocation per shard (or far more with parallelization factor / enhanced fan-out — covered below). The function does the low-latency work: validate and parse, enrich (look up a device profile, a merchant category, a geo from IP), compute lightweight derived fields, drop or quarantine bad records, and bulk-index the results into Amazon OpenSearch Service. Because the function is invoked with a batch per shard, it indexes hundreds of documents in a single OpenSearch _bulk call rather than one-by-one — the difference between a healthy cluster and a melted one. Failures are handled by checkpointing semantics: a batch that errors is retried, and poison records are siphoned to an on-failure destination (an SQS DLQ or SNS) instead of blocking the shard forever.

Tier 3 — The cold path: the analytical landing zone (Kinesis Data Firehose). Independently of the Lambda, a Kinesis Data Firehose delivery stream consumes the same Kinesis data stream. Firehose is the fully-managed, zero-code, buffer-and-deliver service: it accumulates records by size or time (e.g. 128 MB or 60 s), optionally transforms them with its own Lambda, optionally converts JSON to Parquet/ORC using a Glue Data Catalog schema, and writes compressed, partitioned objects to S3. With dynamic partitioning, it lays records out as s3://lake/events/event_type=auth/dt=2026-06-09/hour=14/... so Athena/EMR/Redshift Spectrum can prune partitions efficiently. This is your immutable, cheap, query-anytime source of truth — and your replay buffer’s permanent twin. Firehose can also deliver to OpenSearch directly, but in this reference we keep Firehose on the S3/lake path and Lambda on the OpenSearch path, so the two stores fail and scale independently.

Tier 4 — Serving, search and alerting (Amazon OpenSearch Service). OpenSearch holds the hot, operational slice — typically the last hours-to-days of enriched events in time-based indices (events-2026.06.09), rolled and lifecycled by ISM. Engineers and ops teams query it interactively through OpenSearch Dashboards (live funnels, decline-rate-by-issuer panels, device-fault heat maps), and alerting monitors fire when a threshold trips (“declines for BIN X > 4x baseline over 90 s”) into Slack/PagerDuty/SNS. Anomaly detection (the built-in RCF-based feature) catches the spikes you didn’t write a rule for. As data ages, ISM rolls indices from hot nodes to UltraWarm (S3-backed) and then cold storage, so a single domain serves seconds-fresh dashboards and keeps weeks of searchable history without paying hot-node prices for all of it.

The end-to-end data path, following one authorization event from source to action:

A payment authorises. The producing service writes a JSON record ({card_bin, merchant_id, amount, result, ts, geo}) with partition_key = card_bin to the Kinesis data stream. Kinesis routes it to the shard owning that BIN, appends it durably across 3 AZs, and acknowledges.
Hot path: the Lambda event source mapping has been polling that shard; it delivers a batch of ~200 records to the function. The function enriches each (issuer name from BIN, merchant category, country from geo), computes is_decline, and issues one OpenSearch _bulk request indexing all 200 into events-2026.06.09. Latency from authorise to “visible on the dashboard” is single-digit seconds.
Alert: an OpenSearch alerting monitor running every 60 s over the last 5 minutes sees decline rate for BIN X cross its threshold and pages the on-call fraud analyst with a deep link straight into the filtered dashboard.
Cold path (parallel, unaware of all the above): Firehose, reading the same stream, has been buffering. At 60 s it converts the batch to Parquet using the Glue schema and writes s3://lake/auth/dt=2026-06-09/hour=14/part-….parquet. That object is now permanent, cheap, and queryable by Athena.
Triage & history: the analyst slices the last 15 minutes live in OpenSearch to isolate the attack to three merchants, while a data scientist runs an Athena query over three months of the same events in S3 to see whether this BIN has a history — no impact on the live cluster, because they’re hitting entirely different stores.
Replay (the safety net): a week later the fraud model adds a feature. Rather than ask producers to resend, the team replays: a new Lambda (or an EMR/Flink job) reads the Kinesis stream from a past sequence number, or re-processes the raw S3 Parquet, and rebuilds the derived OpenSearch index from scratch.

The diagram, in words. On the left, a stack of producers (app SDKs/KPL, API Gateway for mobile/IoT, AWS services) all pointing at a single tall cylinder in the center-left labelled Kinesis Data Streams, drawn as a set of parallel horizontal lanes (the shards). From that cylinder, two arrows fan out to the right, and the fact that they’re two separate arrows off the same log is the whole picture. The top arrow goes to a Lambda box (the hot path) which arrows into an OpenSearch Service domain; above OpenSearch sits OpenSearch Dashboards and an Alerting → PagerDuty/SNS badge. The bottom arrow goes to a Kinesis Data Firehose box (the cold path) which arrows into an S3 bucket drawn with partition folders, with Athena / Redshift Spectrum / EMR reading from S3 below. A small Glue Data Catalog sits between Firehose and S3 (schema for Parquet) and is also wired to Athena. A dotted “replay” arrow loops from S3 and from the Kinesis cylinder back into a reprocessing box, emphasising that any store is rebuildable. Cross-cutting boxes underneath — IAM, KMS, VPC, CloudWatch — touch every tier. The defining visual: one log, two non-interfering consumers, each scaling on its own axis.

Streaming primitives, explained (the parts the architecture assumes you know)

The overview above named shards, partition keys, iterator age, and enhanced fan-out as if they were obvious. They aren’t — and every sizing mistake and 3 a.m. page in a streaming system traces back to misunderstanding one of them. This section teaches each primitive from zero, with the arithmetic you actually use, so the component table and the Terraform later read like plain English.

The shard is the unit of everything

A Kinesis data stream is nothing but a set of shards. A shard is simultaneously the unit of capacity, the unit of ordering, and the unit of parallelism — that triple duty is the source of most confusion, so hold all three in your head at once. Each shard gives you a fixed, published budget:

Direction	Per-shard limit	Hit first when…
Write (ingest)	1 MB/s or 1,000 records/s	Large records → MB cap; tiny records → record-count cap
Read — standard (shared)	2 MB/s, shared across all standard consumers; `GetRecords` ≤ 5 calls/s, ≤ 10 MB or 10,000 records per call	Multiple pollers competing
Read — enhanced fan-out	2 MB/s per shard, dedicated per consumer (push)	Never — each EFO consumer gets its own pipe

Two facts fall out immediately. First, writes are capped by whichever ceiling you hit first — 1 MB/s or 1,000 records/s. A stream of 200-byte IoT pings saturates the record limit long before the byte limit; a stream of 200 KB images saturates bytes at five records/s. Second, standard read throughput is shared: add a second polling consumer and the two now split 2 MB/s per shard. That single sentence explains why the reference architecture reaches for enhanced fan-out — more on that below.

A PutRecords batch call carries up to 500 records and 5 MB total, each record up to 1 MB. Batch aggressively; a PutRecord-per-event producer wastes API calls and money.

Partition keys and the hash ring (worked example)

Every record you write carries a partition key — a string you choose (device_id, card_bin, session_id). Kinesis runs that string through MD5 to get a 128-bit number and drops it onto a ring; the shards divide the ring into equal arcs. Same key → same number → same shard, in order, forever. That is the entire mechanism behind “ordered per key.”

Work it through with four shards. The ring [0, 2¹²⁸) splits into four equal arcs, one per shard:

key "device-8f2a91"  → MD5 → 0x2c… → arc 1  → shard-000001
key "device-8f2a91"  → MD5 → 0x2c… → arc 1  → shard-000001   (same key, same shard, ordered)
key "device-01b7c4"  → MD5 → 0xb9… → arc 3  → shard-000003

Because MD5 spreads its inputs uniformly, key distribution equals key variety. Use a high-cardinality key like device_id (millions of distinct values) and traffic fans out evenly across all shards. Use a low-cardinality key like region (say four values) and you light up at most four points on a ring of billions — the arithmetic guarantees that one or two shards carry almost everything. That is a hot shard: it throttles at its 1 MB/s / 1,000 rec/s ceiling while its siblings idle, and the consumer for that shard becomes the whole pipeline’s bottleneck. You cannot fix a hot shard downstream — the damage is done the instant the producer picks the key. When you genuinely need per-entity ordering, the entity is your key. When you don’t, choose the highest-cardinality field you have (or append a random suffix) purely for spread.

Sizing a stream from a number (worked example)

You never guess a shard count — you compute it from throughput. Take the reference example later in this article: 6,000 authorizations/second, ~1.2 kB each.

Bytes/s   = 6,000 × 1,200 B      = 7.2 MB/s   →  7.2 / 1.0  = 8 shards (byte-bound)
Records/s = 6,000                             →  6,000 / 1,000 = 6 shards (record-bound)
Shards    = max(8, 6) = 8, then add headroom  →  provision 8 shards

Always take the max of the byte and record calculations, then add headroom (aim to run each shard near ~70% of its ceiling so a spike has somewhere to go). Eight shards give ~8,000 records/s of write headroom against a 6,000/s peak — comfortable. This is exactly the number the Meridian Pay example lands on, and now you can see why.

On-demand vs provisioned (with the trade)

You pick a capacity mode when you create the stream, and you can switch modes live without downtime.

	On-demand	Provisioned
You manage	Nothing — no shard count	The shard count (and resharding)
Scaling	Automatic: baseline 4 MB/s in / 8 MB/s out, auto-grows to 2× the trailing-30-day peak, up to 200 MB/s / 200,000 rec/s per stream (a raisable soft quota)	You split/merge shards yourself (or via `UpdateShardCount`)
Billing	Per GB ingested and per GB retrieved	Per shard-hour + per million PUT payload units (25 KB each)
Best for	New, spiky, or unpredictable load; getting to production fast	Steady, well-understood load — meaningfully cheaper per GB at scale

The rule the reference architecture follows: launch on-demand, then move to provisioned once the traffic shape is known. Meridian ran on-demand for four weeks, saw a steady ~6k peak, switched to eight provisioned shards, and cut the Kinesis bill ~55% — because on-demand’s convenience premium stops paying for itself the moment your load is predictable.

Retention and replay — the property everything else depends on

A stream keeps every record for a retention period: default 24 hours, extendable to 7 days (extended retention, a per-shard-hour surcharge), and up to 365 days (long-term retention, billed per GB-month plus a small retrieval fee when you read data older than 24 hours). Records are not deleted when a consumer reads them — every consumer has its own position, and the record sits in the log until it ages out. That retention window is your replay buffer. It is the reason you can rebuild any derived store (a corrupted OpenSearch index, a new model feature) without ever going back to the producers. Set it to give yourself enough room to reprocess: seven days is a common, comfortable default.

Iterator age — the one metric to watch

If you learn a single Kinesis metric, learn iterator age: GetRecords.IteratorAgeMilliseconds for standard consumers (and SubscribeToShard.MillisBehindLatest for enhanced fan-out). It is the age of the last record your consumer just read — literally how far behind real time you are. Zero means caught up. A number that climbs means your consumer is falling behind: a hot shard, a slow function, read contention, or a downstream (OpenSearch) applying backpressure. Alarm on iterator-age-rising and you get the earliest possible warning that the hot path is losing the race; everything else (throttle errors, DLQ depth) is a second-order symptom.

KPL and KCL — the libraries you’ll hear named

Two client libraries do heavy lifting so you don’t hand-roll it:

KPL (Kinesis Producer Library) — producer side. It aggregates many small application records into a single ≤1 MB Kinesis record (a large throughput and cost win, since you’re billed per 25 KB PUT payload unit), batches them through PutRecords, and handles retries and rate-limiting. Consumers deaggregate transparently (the KCL and Lambda do it for you).
KCL (Kinesis Client Library) — consumer side. It leases shards using a DynamoDB lease table, checkpoints each consumer’s progress, rebalances shards across worker instances, and follows resharding automatically. KCL 2.x can consume via enhanced fan-out. Lambda’s event source mapping is essentially a fully managed KCL-style consumer you never have to operate — which is why the hot path in this architecture is a Lambda, not a fleet of KCL workers.

Standard consumers vs enhanced fan-out (EFO)

This is the primitive the overview kept promising. With standard consumption, every consumer pulls and they all share the shard’s 2 MB/s read budget. One consumer is fine; add a Firehose delivery stream, a replay job, and a Lambda all polling the same shards and they starve each other — iterator age climbs across the board. Enhanced fan-out flips the model: each registered consumer gets its own dedicated 2 MB/s per shard, and records are pushed to it over an HTTP/2 SubscribeToShard stream at roughly 70 ms latency. You can register up to 20 EFO consumers per stream. It costs extra (per consumer-shard-hour plus per GB retrieved), so the pattern is: EFO for the one latency-critical consumer (the hot-path Lambda), standard polling for the throughput-tolerant ones (Firehose, batch replay). That is precisely the split the reference architecture makes.

Resharding — changing capacity on a provisioned stream

When a provisioned stream’s load changes you reshard. You split a shard (one becomes two, doubling capacity across that key range) or merge two adjacent shards (two become one, halving it). UpdateShardCount automates the underlying splits/merges toward a target shard count (it is rate-limited — roughly a doubling or halving per operation, with a capped number of operations per day). On-demand streams reshard themselves. The subtle part is ordering across a reshard: when a shard splits, the parent shard is closed and its records must be drained to the end before the two child shards take over — that parent→child lineage is exactly how per-key order survives a capacity change. The KCL and Lambda’s event source mapping honor that lineage for you; if you ever write a raw consumer, respecting it is on you.

Component breakdown

Component	AWS service	Role in the pipeline	Key configuration choices
Durable log / ingest	Kinesis Data Streams	Replayable, ordered-per-key buffer; the single source of truth and contract	On-demand mode for spiky/unknown load; provisioned shards once load is steady (cheaper at scale). Partition key chosen for even distribution. Retention 24h → up to 365 days. Server-side KMS encryption on.
Producers	KPL / AWS SDK / API Gateway proxy / agents	Get events into the stream efficiently	KPL for high-throughput aggregation+batching; API Gateway → Kinesis integration for browser/mobile/IoT where you don’t ship the SDK. Always set a meaningful partition key.
Hot-path processor	AWS Lambda (event source mapping)	Low-latency enrich, transform, route, and bulk-index to OpenSearch	Batch size 100–500; batch window 1–5 s; parallelization factor up to 10 per shard; bisect-on-error + maximum-retry + on-failure SQS/SNS DLQ; enhanced fan-out consumer for fast, dedicated read.
Cold-path delivery	Kinesis Data Firehose	Zero-code buffer → transform → Parquet → partitioned S3 landing zone	Buffer 128 MB / 60 s; record format conversion to Parquet via Glue schema; dynamic partitioning by `event_type`/date/hour; GZIP/Snappy compression; S3 error-output prefix for failed records.
Operational analytics store	Amazon OpenSearch Service	Sub-second search, live dashboards, alerting, anomaly detection over hot data	Multi-AZ with dedicated master nodes (3) at scale; time-based indices + ISM rollover; UltraWarm + cold tiers for history; in-VPC deployment; fine-grained access control.
Analytical lake	Amazon S3 + Glue Data Catalog	Immutable raw archive + schema registry for query-in-place	Lifecycle to S3 Intelligent-Tiering/Glacier; partitioned layout; Glue crawler or Firehose-registered schema; the durable twin of the stream.
Lake query engines	Athena, Redshift Spectrum, EMR	Ad-hoc and heavy analytics over months/years of events — off the hot store	Athena for ad-hoc/serverless; Spectrum for BI joins to curated dims; EMR/Spark/Flink for replay and ML feature jobs.
Cross-cutting	IAM, KMS, CloudWatch, VPC, X-Ray	Identity, encryption, observability, isolation	Least-privilege roles per tier; CMKs per stream/domain; iterator-age & throttle alarms; private subnets + endpoints.

A few component-level decisions carry disproportionate weight:

Partition key design is the single highest-leverage choice in the whole system. The key determines both ordering and parallelism. Pick a key with too few distinct values (say, region with 4 values) and you get hot shards: most traffic crams onto one shard, that shard’s records throttle at its 1 MB/s · 1000 rec/s write limit while others sit idle, and the consumer for that shard becomes your bottleneck. Pick a high-cardinality, evenly-distributed key (device_id, session_id, card_bin if BINs are well spread) and load fans out across shards smoothly. When you genuinely need per-entity ordering, the entity is your key; when you don’t, use a high-cardinality key (or a random suffix) purely for distribution. There is no fixing a bad partition key downstream — it is decided at the producer.

Why two consumers off one stream instead of one Lambda that does everything. A Kinesis stream supports multiple consumers, and using that is the architecture, not an optimisation. If a single Lambda both indexed to OpenSearch and wrote to S3, then OpenSearch back-pressure (a cluster yellow, a bulk reject) would stall the same iterator that’s responsible for archival — you’d risk losing your durable copy because your operational copy was unhealthy. By giving Firehose its own consumer, the archive keeps flowing no matter what OpenSearch is doing, and vice-versa. They share a source of truth and share nothing else.

Standard consumers vs. enhanced fan-out. Each shard’s read throughput (2 MB/s) is shared across all standard (polling) consumers. With both a Lambda and Firehose polling, plus any replay job, you can starve reads and watch iterator age climb (your consumers fall behind real time). Enhanced fan-out (EFO) gives a consumer its own dedicated 2 MB/s per shard pipe with push delivery and ~70 ms latency. The reference uses EFO for the latency-sensitive Lambda hot path and standard consumption for Firehose (which is throughput-, not latency-, sensitive), so the live dashboard stays fast even when the archive is busy.

Firehose does the boring-but-critical lake hygiene for free. Hand-rolling JSON→Parquet conversion, buffering to avoid the S3 small-file problem, and Hive-style partitioning is a surprising amount of code to get right and keep right. Firehose does all three declaratively: it buffers to large objects (killing small-file query tax), converts to columnar Parquet against a Glue schema (10x cheaper Athena scans), and dynamic-partitions by fields you choose. Letting the managed service own the cold path is what keeps the team’s code limited to the business logic in the hot-path Lambda.

Ingest options: choosing the front door

This reference uses Kinesis Data Streams as the front door, and for a keyed, replayable, multi-consumer log it is the right default on AWS. But “get events into the platform” has four other credible answers, and picking the wrong one is a common early mistake. Here is the whole ingest family and the one question that separates them.

A rename to know: what this article calls Kinesis Data Firehose is now officially Amazon Data Firehose — same service, new name (AWS dropped the “Kinesis” prefix in 2024). Older consoles, docs, IAM actions (firehose:*), and Terraform resources (aws_kinesis_firehose_delivery_stream) still use the legacy name, which is why you’ll see both. They are the same thing.

Option	What it is	Ordering	Replay / retention	Reach for it when…
Kinesis Data Streams	Durable, keyed, replayable log with many independent consumers	Per shard (per key)	Yes — 24 h to 365 days	You need a stream to process: multiple consumers, per-key order, replay. The default here.
Amazon Data Firehose (was Kinesis Data Firehose)	Zero-code delivery pipe: buffer → transform → convert → deliver	None (delivery, not a log)	No — it delivers, it doesn’t retain for replay	You only need to land data in S3 / Redshift / OpenSearch / Splunk with buffering, Parquet conversion, and dynamic partitioning — no custom consumers.
Amazon MSK (managed Kafka)	Fully managed Apache Kafka	Per partition (per key)	Yes — configurable, incl. log compaction	You have an existing Kafka ecosystem, need compaction or consumer-group semantics, or want multi-cloud portability.
DynamoDB Streams	Change-data-capture feed of table item mutations	Per partition key	24 h	The events you care about are already DynamoDB writes — capture CDC without a separate producer.
Kinesis Video Streams	Durable ingest for time-encoded media (video/audio/binary)	Per fragment, time-indexed	Configurable	You’re streaming camera/media frames for playback or ML (Rekognition), not JSON business events.

The dividing question is: do you have a stream to process or just data to deliver? If two different audiences must consume the same events at their own pace, with replay — that’s a log, and you want Kinesis Data Streams or MSK. If you simply need events to arrive in a store, cheaply and hands-off — that’s delivery, and Firehose is purpose-built for it. In this architecture the answer is both, which is why Firehose sits downstream of the Kinesis stream (consuming it) rather than replacing it: the stream provides the replayable log and the fan-out; Firehose provides the boring, correct, zero-code landing into the lake.

Firehose, specifically, earns its place by owning three things that are surprisingly hard to hand-roll and keep correct:

Buffering — it accumulates records to a size or time threshold (e.g. 128 MB or 60 s) before writing, so you get large objects instead of the millions of tiny files that make a data lake slow and expensive to query (the “small-file problem”).
Transformation & format conversion — an optional transform Lambda for light shaping, plus native JSON → Parquet/ORC conversion against a Glue Data Catalog schema, which makes downstream Athena scans roughly 10× cheaper (columnar + compressed).
Dynamic partitioning — it writes Hive-style partitions like event_type=auth/dt=2026-06-09/hour=14/ derived from fields in the record, so query engines prune partitions instead of scanning everything.

MSK vs Kinesis is the comparison teams agonize over most. Kinesis wins on operational simplicity (no brokers, no ZooKeeper/KRaft to run), native Lambda/Firehose integration, and on-demand auto-scaling. MSK wins when you already live in Kafka: existing producers/consumers, the connector ecosystem, log compaction, fine-grained consumer-group control, and portability across clouds. If you’re greenfield on AWS and thinking in AWS-native terms, Kinesis is less to run; if Kafka is already your event backbone, MSK lets you keep it managed. That trade-off — and how a Kafka backbone looks on AWS — is explored in Confluent Cloud Kafka as an event backbone.

DynamoDB Streams deserves a special call-out because it removes a producer entirely. If your source of truth is a DynamoDB table, its stream emits an ordered, per-partition-key feed of every insert/modify/remove, which a Lambda can fan into exactly the hot/cold split described here — no application code changes to “start producing.” That change-data-capture pattern is covered in DynamoDB Streams change data capture. The mental model stays identical: a durable, keyed, ordered log with independent consumers — only the log is a table’s change feed instead of a Kinesis stream.

Implementation guidance

Region, accounts, and isolation. Put the streaming stack in the account that owns the workload, but keep the analytical lake (S3 + Glue) in a data account so analysts and the platform team get governed access without touching production streaming. In a multi-account org (AWS Organizations / Control Tower), the Kinesis stream and Lambda live in prod-app, Firehose assumes a role to write into the data-lake account’s bucket, and OpenSearch lives in prod-app (operational) or a shared observability account.

Infrastructure as Code (Terraform sketch). Everything here is declarative; do not click streams or domains into existence. The core resources and the wiring that people most often get wrong:

# 1. The durable log — on-demand to start; switch to PROVISIONED with a shard count once load is known.
resource "aws_kinesis_stream" "events" {
  name             = "events-${var.env}"
  retention_period = 168                       # hours; 7 days of replay headroom
  stream_mode_details { stream_mode = "ON_DEMAND" }
  encryption_type  = "KMS"
  kms_key_id       = aws_kms_key.stream.arn
}

# 2. Enhanced fan-out consumer for the latency-sensitive hot path.
resource "aws_kinesis_stream_consumer" "hot" {
  name       = "lambda-hot-efo"
  stream_arn = aws_kinesis_stream.events.arn
}

# 3. Lambda event source mapping — the bits that save you at 3 a.m.
resource "aws_lambda_event_source_mapping" "hot" {
  event_source_arn                   = aws_kinesis_stream_consumer.hot.arn   # EFO ARN
  function_name                      = aws_lambda_function.enricher.arn
  starting_position                  = "LATEST"
  batch_size                         = 300
  maximum_batching_window_in_seconds = 2
  parallelization_factor             = 4         # >1 invocation per shard, order kept per key
  bisect_batch_on_function_error     = true      # isolate the poison record
  maximum_retry_attempts             = 5
  maximum_record_age_in_seconds      = 3600
  function_response_types            = ["ReportBatchItemFailures"]  # partial-batch checkpointing
  destination_config {
    on_failure { destination_arn = aws_sqs_queue.dlq.arn }
  }
}

# 4. Firehose cold path — same stream, independent consumer; JSON -> Parquet -> partitioned S3.
resource "aws_kinesis_firehose_delivery_stream" "lake" {
  name        = "events-lake-${var.env}"
  destination = "extended_s3"
  kinesis_source_configuration {
    kinesis_stream_arn = aws_kinesis_stream.events.arn
    role_arn           = aws_iam_role.firehose.arn
  }
  extended_s3_configuration {
    role_arn   = aws_iam_role.firehose.arn
    bucket_arn = aws_s3_bucket.lake.arn
    buffering_size     = 128
    buffering_interval = 60
    dynamic_partitioning_configuration { enabled = true }
    prefix              = "events/event_type=!{partitionKeyFromQuery:event_type}/dt=!{timestamp:yyyy-MM-dd}/hour=!{timestamp:HH}/"
    error_output_prefix = "errors/!{firehose:error-output-type}/dt=!{timestamp:yyyy-MM-dd}/"
    data_format_conversion_configuration {
      enabled = true
      output_format_configuration { serializer { parquet_ser_de {} } }
      schema_configuration {
        role_arn      = aws_iam_role.firehose.arn
        database_name = aws_glue_catalog_database.lake.name
        table_name    = aws_glue_catalog_table.events.name
      }
    }
  }
}

The high-value, frequently-missed lines: function_response_types = ["ReportBatchItemFailures"] lets the function return which records in a batch failed so Lambda re-drives only those rather than the whole batch (no more replaying 299 good records to retry 1 bad one); bisect_batch_on_function_error plus on_failure DLQ guarantees a poison pill never wedges a shard; and parallelization_factor is how you scale processing concurrency beyond one-per-shard while still preserving per-partition-key order. On the Firehose side, dynamic_partitioning + the !{partitionKeyFromQuery:...} prefix is what produces a properly partitioned, Parquet, query-cheap lake with zero custom code.

Networking and identity wiring.

OpenSearch in a VPC, never public. Place data nodes in private subnets across AZs; reach Dashboards through a bastion/SSO proxy or via a private ALB. Enable fine-grained access control and map IAM roles to OpenSearch roles, so the hot-path Lambda’s role can write to events-* indices and nothing else.
VPC endpoints (PrivateLink) for Kinesis, Firehose, S3 (gateway endpoint), and STS so data and control-plane traffic never traverse the public internet. The Lambda runs in the VPC to reach OpenSearch; give it an endpoint to Kinesis so it can still read the stream.
Least-privilege IAM, one role per tier. The producer role gets kinesis:PutRecord* on exactly that stream. The Lambda execution role gets kinesis:GetRecords/GetShardIterator/SubscribeToShard/DescribeStream* on the stream/consumer, es:ESHttpPost/Put on the OpenSearch domain, and sqs:SendMessage on the DLQ. The Firehose role gets read on the stream, write on the lake bucket, glue:GetTable* for the schema, and kms:GenerateDataKey/Decrypt on the relevant keys. No tier shares a role.
KMS CMKs for the stream, the OpenSearch domain, and the S3 lake. Producers and consumers need explicit kms:Decrypt/GenerateDataKey grants — a forgotten KMS grant is the most common “it deployed but nothing flows” failure.

Schema discipline. Producers should emit a versioned envelope ({schema_version, event_type, payload, ts}). Register the canonical schema in AWS Glue Schema Registry; producers using the KPL/SDK serializer validate against it, so a malformed producer is rejected at the edge rather than poisoning the lake. The hot-path Lambda routes on event_type, and Firehose dynamic-partitions on it — one field doing double duty.

Stream processing: Lambda, Flink, and Spark

The reference architecture processes the hot path with a Lambda, and for stateless per-batch work — validate, enrich, tokenize, bulk-index — that is the right, cheap, simple choice. But “stream processing” is a spectrum, and the moment you need to compute something across events (a per-minute count, a running average, a join between two streams, a user session) a stateless Lambda stops being enough. This section maps the three engines and the streaming concepts that decide between them.

The three engines

Engine	Model	State	Reach for it when…
AWS Lambda (event source mapping)	Stateless, per-batch	None built in (externalize to DynamoDB)	Enrich/transform/route/index each batch independently. Simplest and cheapest. This article’s hot path.
Amazon Managed Service for Apache Flink	True streaming, event-at-a-time	Rich, managed operator state	Windowed aggregations, stream-to-stream joins, sessionization, event-time correctness, exactly-once.
Spark Structured Streaming on EMR	Micro-batch (and continuous)	Managed, large-scale	You already run Spark/EMR, want one codebase for batch + stream, or heavy stateful/ML feature jobs at higher latency.

Another rename to know: Amazon Managed Service for Apache Flink is the current name for what was Kinesis Data Analytics (specifically the Apache Flink flavor). Same service, renamed in 2023. You’ll still see “Kinesis Data Analytics”/“KDA” in older material and some ARNs.

The dividing line is state. Lambda’s contract is “here is a batch, do something to it, I’ll forget you existed.” That is perfect for stateless work and it is why the hot path here is a Lambda. The instant you need to remember something between batches — a count over the last minute, whether this session is still open, the last value seen per device — you either bolt an external store onto Lambda (workable for simple cases, awkward as it grows) or move to Flink, which manages that state for you, durably and at scale.

Windowing: turning an infinite stream into finite answers

A stream never ends, so any aggregation must be scoped to a window. Three shapes cover almost everything:

Tumbling — fixed size, non-overlapping. “Count authorizations per 1-minute window.” Every event lands in exactly one window. The default for periodic metrics.
Sliding — fixed size, overlapping by a slide interval. “5-minute decline rate, recomputed every 1 minute.” An event can belong to several windows at once. Use it for smoothed, frequently-updated rolling metrics.
Session — dynamic, defined by a gap of inactivity. “Group a user’s events until they’re quiet for 30 minutes.” The window’s length depends on the data, not the clock. The natural fit for user/device sessions and funnels.

Event time vs processing time (and why it bites)

There are two clocks in every streaming system, and confusing them produces subtly wrong numbers:

Processing time — the wall-clock at the operator when it handles the event. Simple and low-latency, but non-deterministic: reprocess the same data on a faster machine and you get different window boundaries. A network blip that delays events shifts them into the “wrong” minute.
Event time — the timestamp embedded in the event, when it actually happened at the source. Deterministic and correct even when events arrive late or out of order — which, across mobile clients, retries, and multiple shards, they always do. Event time is what you want for anything a human will trust (“declines in the 14:03 minute”).

Event time only works if the engine knows when it has probably seen everything for a window. That’s what a watermark is: a marker that flows with the stream asserting “no more events with timestamp ≤ T will arrive (probably).” When the watermark passes a window’s end, the window fires. Allowed lateness keeps a window open a little longer to absorb stragglers; events later than watermark + lateness are dropped or routed to a side output. Watermarks are the single concept that makes out-of-order streams produce correct, on-time results — and they are a first-class feature in Flink, not something you get from a bare Lambda.

Exactly-once, checkpointing, and what “guarantee” really means

Kinesis delivery is at-least-once: on a retry, a reshard, or a producer resend, a consumer can legitimately see the same record twice. So “exactly-once” is never free — it’s something the processing layer provides on top. Flink does it with two mechanisms working together:

Checkpointing — Flink periodically takes a consistent distributed snapshot of all operator state plus the source positions (a barrier-based algorithm), and writes it to durable storage. On failure it restores the last snapshot and replays the sources from exactly those positions, so state is recovered without double-counting. Managed Service for Apache Flink runs these checkpoints for you and also supports snapshots/savepoints for clean restarts and version upgrades.
Transactional sinks — checkpointing gives exactly-once state; end-to-end exactly-once also needs the output to be idempotent or transactional (a two-phase commit sink, or a deterministic record id the sink dedupes on). Without that, your state is exact but your writes can still duplicate.

For the Lambda hot path in this architecture, exactly-once is handled the pragmatic way: make the OpenSearch write idempotent by using a deterministic document _id (e.g. a hash of the event’s natural key + sequence number), so re-indexing the same record overwrites rather than duplicates. That is the everyday answer when you don’t need a full stateful engine.

Lambda vs Kappa: two ways to shape the whole platform

Zoom out from the engines to the architecture patterns, and there are two classic blueprints (the naming is unfortunate — “Lambda architecture” has nothing to do with AWS Lambda the service):

Lambda architecture — run two layers in parallel: a speed layer (real-time, approximate, low-latency — here, Kinesis → Lambda/Flink → OpenSearch) and a batch layer (complete, accurate, high-latency recompute over all history — here, S3 + Athena/Spark), merged at query time. It’s accurate but you maintain two codebases and reconcile two systems.
Kappa architecture — run one path: everything is a stream, and “reprocessing” just means replaying the log through the same streaming code. It requires a durable, replayable log (which Kinesis retention and the S3 Parquet twin both give you).

This reference is essentially kappa-flavored: one durable log, derived stores rebuilt by replay, no separate batch codebase to keep in sync. The cold path’s S3 Parquet isn’t a competing “batch layer” — it’s the same events, kept cheaply and long, replayable through the same processing logic when a schema or model changes. That is the elegance the durable log buys you, and it’s why the “replay to rebuild” loop appears again and again in the sections that follow.

Enterprise considerations

Security & Zero Trust. Treat every tier as independently authenticated and encrypted. Encryption in transit is TLS everywhere (Kinesis/Firehose/OpenSearch HTTPS endpoints); encryption at rest is KMS CMKs on the stream, the domain, and the bucket. OpenSearch runs in-VPC with fine-grained access control, document- and field-level security where PII lives (mask pan/card_number, restrict geo to authorised roles), and SAML/SSO for Dashboards users — no shared admin logins. Apply least-privilege per-tier IAM as above, and use SCPs to forbid creating public OpenSearch domains or unencrypted streams org-wide. For regulated data, strip or tokenize sensitive fields in the hot-path Lambda before indexing, and let only the S3 lake hold the raw (encrypted, access-controlled) copy. Nothing trusts the network; every hop carries identity.

Cost optimization. The cost levers are specific and mostly about not over-paying for the hot tier:

Kinesis mode: on-demand is convenient and right for spiky/unknown load, but it costs meaningfully more per GB than provisioned at steady state. Once you know your sustained throughput, switch to provisioned and size shards to ~70% of capacity; you can flip modes without downtime. This is frequently the biggest single saving.
OpenSearch tiering: keep only days of data on expensive hot nodes; roll older indices to UltraWarm (S3-backed, a fraction of the cost) and then cold via ISM. Right-size with gp3 storage and reserved instances for the steady-state cluster. Most teams keep 10x more data hot than anyone ever queries.
Firehose + Parquet makes the cold path almost free to query: columnar + compressed + partitioned means Athena scans (and bills) a fraction of the bytes. Lifecycle S3 to Intelligent-Tiering and Glacier for cold history.
Lambda is billed on GB-seconds; right-size memory (which also sets CPU), keep the function lean, and let batching amortise invocations. A bigger batch window trades a little latency for fewer invocations and fewer, larger OpenSearch bulk calls.

Scalability. Each tier scales on its own axis, which is the point. The stream scales by shard count (provisioned) or automatically (on-demand) up to the quota; watch WriteProvisionedThroughputExceeded and resharding (split/merge). Lambda scales to one concurrent invocation per shard times the parallelization factor — so 50 shards × factor 4 = up to 200 concurrent enrichers — and EFO removes read contention. Firehose scales transparently. OpenSearch scales by adding data nodes and shards (size shards to ~10–50 GB, avoid shard explosion). The governing metric for “is the hot path keeping up?” is iterator age / GetRecords.IteratorAgeMilliseconds — if it climbs, consumers are falling behind real time and you add shards, raise the parallelization factor, or move to EFO.

Reliability & DR (RTO/RPO). The durable log is the resilience story. Kinesis replicates synchronously across three AZs, so an AZ loss is a non-event for ingest. The replay window (retention up to 365 days) is your logical RPO for derived stores: if OpenSearch is corrupted or lost, you have not lost data — you rebuild the index by replaying the stream and/or reprocessing the S3 Parquet, so RPO for the derived/operational store is effectively zero up to the retention window, and RTO is “how long a reprocess takes.” For OpenSearch itself, take automated snapshots to S3 (hourly) for a fast restore. For regional DR, the lake’s S3 bucket uses Cross-Region Replication; the stream can be mirrored to a second region with a small consumer-and-re-put bridge or you stand the pipeline up in the DR region and replay from replicated S3. Pin concrete numbers to it: hot-path component failure RTO in minutes (Lambda/OpenSearch auto-recover); full operational-store rebuild RTO in the low hours via replay; data-loss RPO ≈ 0 within retention. The poison-pill DLQ guarantees one bad record can never take down a shard — reliability at the record level, not just the cluster level.

Observability. Instrument all four tiers, but watch these signals specifically: stream iterator age and throughput-exceeded (the canary for falling behind / hot shards); Lambda errors, throttles, iterator age, and DLQ depth (a non-empty DLQ means poison records to investigate); Firehose delivery freshness and S3 delivery failures; OpenSearch cluster status (green/yellow/red), _bulk rejections / 429s, JVM memory pressure, and free storage. Wire CloudWatch alarms on iterator-age-rising and DLQ-not-empty as your two highest-signal pages. Enable X-Ray on the Lambda to see enrich-then-index latency. And use OpenSearch’s own alerting/anomaly-detection for the business signals (decline spikes, crash bursts) — the platform that holds the data is also the one that watches it.

Governance. Tag every resource by data-classification, owner, cost-center, and env. Enforce org-wide guardrails with SCPs (no unencrypted streams, no public OpenSearch). Catalog the lake in Glue / Lake Formation so analyst access to historical events is governed with row/column controls, decoupled from the operational store. Keep producer schemas in the Glue Schema Registry with compatibility rules so a producer change can’t silently break the lake. Retention is policy: stream retention, OpenSearch ISM, and S3 lifecycle each encode “how long, where” for their tier, and they should be reviewed together.

Reference enterprise example

Meridian Pay is a fictional mid-market payments processor: ~1,400 merchants, peaks of 6,000 authorizations/second on Black-Friday-class days, ~1.2 kB JSON per event, and a hard requirement from their sponsor bank to detect fraud-pattern anomalies and gateway degradation in near-real-time. Their old world was a read-replica of the auth database plus a nightly job; fraud was found at T+1 day and the bank was unhappy.

What they built. They stood up the reference exactly as above:

Kinesis Data Streams, started in on-demand mode while load was unknown, then — after four weeks of data showed a steady ~6k peak / ~900 average — moved to provisioned with 8 shards (each shard handles 1 MB/s · 1000 rec/s ingest; 8 gives ~8k rec/s headroom at peak with the partition key being card_bin, which spreads evenly across ~3,000 active BINs). The mode switch alone cut their Kinesis bill by ~55%. Retention set to 7 days for replay headroom.
Hot-path Lambda with an enhanced fan-out consumer, batch_size = 300, maximum_batching_window = 2 s, parallelization_factor = 4. It enriches each auth (issuer name and country from BIN, merchant category, is_decline, tokenizes the PAN so no card number ever reaches OpenSearch) and bulk-indexes ~300 docs per call into daily auth-YYYY.MM.DD indices. Partial-batch failures (ReportBatchItemFailures) re-drive only bad records; a poison auth goes to an SQS DLQ that an engineer triages. End-to-end authorise→dashboard latency measured at 3–5 s at peak.
OpenSearch Service: 3 dedicated master nodes + 6 r6g data nodes across 3 AZs, in-VPC, fine-grained access control with field-level security hiding the (tokenized) PAN from analyst roles. ISM keeps 2 days hot, rolls 14 days to UltraWarm, then cold-stores to 90 days. An alerting monitor evaluates decline-rate-by-BIN every 60 s over a 5-minute window; anomaly detection runs on per-issuer auth volume.
Firehose, consuming the same stream independently, buffers 128 MB / 60 s, converts to Parquet via a Glue schema, and dynamic-partitions into s3://meridian-lake/auth/dt=…/hour=…/ in the separate data account. Athena and Redshift Spectrum serve the analysts and the bank’s monthly reporting off S3 — never touching the live cluster.

The decisions that mattered. They explicitly chose two consumers over one mega-Lambda after a load test showed that an OpenSearch _bulk 429 during a spike stalled the combined function and started backing up archival — unacceptable for a regulated copy of record. Splitting Firehose onto its own consumer made the S3 archive immune to OpenSearch health. They chose EFO for the hot path after iterator age crept up under combined polling load; the dedicated 2 MB/s pipe dropped iterator age back to near-zero. And they sized OpenSearch hot capacity for 2 days, not 90 — UltraWarm holds the rest — after realising analysts queried >7-day-old data via Athena anyway.

The incident that proved it. Eleven weeks in, an issuer had a partial outage. Decline rate for that issuer’s BIN range went vertical. The OpenSearch monitor paged the on-call within 90 seconds; the analyst opened the deep-linked dashboard, sliced the last 15 minutes, and confirmed it was issuer-side (one BIN range, all merchants, all geographies) rather than a Meridian gateway fault — and routed affected traffic to a fallback issuer. Total time from spike to mitigation: under 7 minutes, versus the next-day discovery their old system offered. A month later, when the fraud team added two features to the model, they replayed three days of auths from S3 Parquet to rebuild a new derived index — no producer changes, no data loss, a few hours of EMR.

The outcome. Fraud-pattern and outage detection moved from T+1 day to seconds. The bank got its near-real-time assurance. Steady-state cost landed around $3,100/month (Kinesis provisioned ~$300, Lambda ~$250, Firehose ~$200, OpenSearch ~$2,000, S3/Athena/Glue ~$350) — roughly one-third of an early quote that had put 90 days of data on hot OpenSearch nodes. The whole pipeline is ~600 lines of Terraform plus one Lambda; nobody hand-writes Parquet, partitioning, or buffering, because Firehose owns all three.

When to use it

Use this architecture when you have a continuous event stream that must be actionable in seconds and retained cheaply for later, where the operational view and the analytical archive have genuinely different access patterns. The sweet spot: fraud/risk monitoring, IoT/device telemetry, clickstream and product analytics, gaming telemetry, ad-tech, application/security observability, and any “decline/anomaly/spike — tell me now and let me dig later” problem. It shines precisely because the hot and cold paths scale and fail independently, and because the durable log makes every derived store rebuildable.

Trade-offs to go in with eyes open. You are running and right-sizing an OpenSearch cluster, which is the operationally heaviest piece here (shard sizing, JVM pressure, version upgrades) — budget for that expertise or lean harder on managed alarms. Kinesis introduces shard math and partition-key thinking that a simpler queue doesn’t; get the key wrong and you fight hot shards. And there is a real latency floor — this is seconds-fresh, not microseconds; it is not a low-latency transactional path.

Anti-patterns to avoid. Do not write one Lambda that does ingest, enrich, index, and archive — you couple the durability of your record-of-truth to the health of your dashboard, exactly the failure Meridian load-tested away. Do not index to OpenSearch one document per record — always _bulk per batch. Do not pick a low-cardinality partition key for convenience; hot shards will find you at peak. Do not keep all history on hot OpenSearch nodes because it’s easy — that’s the line item that triples the bill. Do not skip the DLQ / partial-batch-failure wiring; a single poison record will otherwise wedge a shard and silently stall the pipeline. And do not treat Firehose-to-OpenSearch as a substitute for the Lambda hot path when you need enrichment and custom routing — Firehose’s transform Lambda is fine for light shaping, but the dual-consumer split (Lambda→OpenSearch, Firehose→S3) is what keeps the stores independent.

Alternatives, and when they win.

Managed Service for Apache Flink (KDA) instead of Lambda when you need true stateful stream processing — windowed aggregations, joins across streams, exactly-once sinks, sessionization. Lambda is perfect for stateless per-batch enrich-and-index; reach for Flink the moment you need sustained windowed state or stream-to-stream joins.
Amazon MSK (managed Kafka) instead of Kinesis when you have existing Kafka investment/ecosystem, need very long retention with compaction, sub-shard-granularity consumer-group semantics, or multi-cloud portability. Kinesis wins on operational simplicity, native Firehose/Lambda integration, and on-demand auto-scaling; MSK wins on Kafka-ecosystem fit and fine-grained control.
DynamoDB Streams / Kinesis-from-DynamoDB when the events you care about are already table mutations — capture change data without a separate producer.
EventBridge instead of Kinesis when you have discrete, low-to-moderate-volume business events needing rich routing/filtering to many targets, rather than a high-throughput ordered firehose. EventBridge is routing-shaped; Kinesis is log-shaped. Choose by whether you think in events to route or a stream to process.
OpenSearch Ingestion / Data Prepper as an alternative to the Lambda for the OpenSearch path when you want a managed, config-driven pipeline (with its own buffering and transforms) instead of owning function code — a reasonable swap that keeps the dual-consumer shape.

The decision rule in one line: if you have a high-volume, keyed event stream that two very different audiences must consume — one in seconds, one over months — without interfering, this Kinesis → (Lambda → OpenSearch | Firehose → S3) reference is the AWS-native answer, and the durable log underneath it is what lets you sleep.

Going deeper

Everything above is enough to build the pipeline. This section is for the reader who has to operate it under load, pass a design review, or debug it at 3 a.m. — the internals, edge cases, and numbers behind the primitives.

Ordering and delivery guarantees, precisely

Kinesis gives strict ordering per shard, which — because a partition key maps to a fixed shard — means ordering per key. Each record gets a sequence number that strictly increases within a shard; there is deliberately no global order across shards (that’s how the log scales horizontally). Two consequences bite in practice:

Producer-side reordering. PutRecords can partially fail: 3 of 500 records get throttled while 497 succeed. A naive “retry the failures” resends them after later records for the same key already landed — reordering that key. If strict per-key order matters, either use PutRecord sequentially with the SequenceNumberForOrdering parameter, or accept the KPL’s ordering behavior and design consumers to tolerate it. Most teams discover this only when an out-of-order event corrupts a running total.
At-least-once, everywhere. Delivery is at-least-once, so duplicates are normal, not exceptional — from producer retries, consumer retries, and resharding transitions. The correct posture is idempotent consumers: dedupe on a natural key or sequence number, use deterministic sink ids (the OpenSearch _id trick), and never assume “processed once.” Anyone who assumes exactly-once by default ships a double-counting bug.

Backpressure and throttling — what actually happens

“The stream is throttling” is really two different failures on two different sides:

Write side. Exceed a shard’s 1 MB/s / 1,000 rec/s and producers get ProvisionedThroughputExceededException; the CloudWatch signal is WriteProvisionedThroughputExceeded. If it’s concentrated on one shard, you have a hot partition key, not a capacity shortage — adding shards won’t help until you fix the key. If it’s spread across all shards, you’re genuinely at capacity: reshard or move to on-demand.
Read side. Standard consumers sharing 2 MB/s per shard throttle each other; the symptom is rising iterator age, sometimes with ReadProvisionedThroughputExceeded. This is where enhanced fan-out earns its cost — it removes read contention entirely by giving each consumer its own pipe.

The elegant part is how backpressure propagates safely. If OpenSearch slows down, the hot-path Lambda’s _bulk calls take longer, so each batch takes longer, so the event source mapping reads the shard more slowly, so iterator age rises — a visible, alarmable signal — but nothing is lost, because the durable log holds the backlog up to the retention window. Backpressure in a well-built streaming system converts into latency and a metric, never into dropped data. That is the whole reason the buffer sits in the middle.

Enhanced fan-out internals and cost

EFO isn’t magic; it’s a different transport. Instead of consumers polling GetRecords, a registered EFO consumer opens a long-lived HTTP/2 connection and calls SubscribeToShard; Kinesis then pushes records to it, per shard, over a dedicated 2 MB/s channel at ~70 ms typical latency. You may register up to 20 EFO consumers per stream. Cost has two components you must model: a consumer-shard-hour charge (you pay for every shard your consumer is subscribed to, every hour) plus a per-GB data-retrieval charge. Concretely: one EFO consumer on an 8-shard stream is 8 shards × 730 h ≈ 5,840 consumer-shard-hours/month on top of data retrieval — which is why EFO roughly doubles the base Kinesis line item and why you reserve it for the one latency-critical consumer rather than sprinkling it everywhere.

The KPL aggregation gotcha

The KPL’s aggregation — packing many user records into one Kinesis record to save money — is invisible only if every consumer deaggregates. Lambda and the KCL do it automatically. But if some tool reads the raw stream without KPL deaggregation (a hand-rolled consumer, certain third-party connectors), it sees the aggregated blob, not your records, and quietly mangles everything. Rule: if you aggregate on the producer, guarantee every consumer deaggregates — or don’t aggregate.

Glue Schema Registry and safe schema evolution

The lesson mentions registering schemas; here’s the machinery that makes it worth it. The AWS Glue Schema Registry stores versioned schemas (Avro, JSON Schema, or Protobuf). Producers serialize with a schema, and the registry embeds only a compact schema-version id in each record (not the whole schema — cheap on the wire); consumers use that id to deserialize. The payoff is enforced compatibility modes, checked at registration so a breaking change is rejected before it can poison the lake:

Mode	Guarantees	Safe change
BACKWARD (default)	New schema can read data written with the previous schema	Delete a field, add an optional field
FORWARD	Previous schema can read data written with the new schema	Add a field, delete an optional field
FULL	Both directions, adjacent versions	Add/delete only optional fields
NONE	No checks	Anything (you own the risk)

There are TRANSITIVE variants that check against all prior versions, not just the adjacent one. Wire the registry into the KPL/KCL, Firehose, Flink, or Lambda, pick BACKWARD for most event streams, and a producer that tries to ship an incompatible change fails at deploy time instead of silently breaking every downstream reader at 2 a.m.

A cost model you can defend

Streaming bills surprise people because the hot store dominates and the stream itself is cheap. Representative us-east-1 figures (illustrative — always price your own region/usage; rates change):

Kinesis provisioned: ~$0.015 per shard-hour → 8 shards × 730 h ≈ $88/month for shards, plus per-million PUT payload units (25 KB each). EFO and extended retention add to this — together roughly the ~$300/month the reference example cites.
Amazon Data Firehose: billed per GB ingested (with a small uplift for format conversion) — typically the smallest line item.
Amazon OpenSearch Service: the dominant cost, because hot data nodes are expensive. The lever isn’t the node type; it’s how much data you keep hot. Keeping 90 days hot instead of 2 (with UltraWarm/cold for the rest) is the difference between the reference’s ~$2,000/month and the ~$6,000 early quote it replaced.
S3 + Athena: near-free to store (pennies/GB-month, lifecycle to Intelligent-Tiering/Glacier) and cheap to query because Firehose wrote Parquet — Athena bills per TB scanned, and columnar + compressed + partitioned means you scan a fraction of the bytes.

The one-line cost lesson: right-size the hot tier, let Parquet make the cold tier almost free, and the stream is a rounding error.

DR framing in one paragraph

The durable log is the disaster-recovery story, so put numbers on it. Kinesis replicates synchronously across three Availability Zones — an AZ loss is a non-event for ingest. Because you can replay up to the retention window, the RPO for any derived store is effectively zero within that window: lose OpenSearch and you rebuild it from the stream or the S3 Parquet rather than from producers. RTO is “how long a reprocess takes” — minutes for a component that auto-recovers, low hours for a full operational-store rebuild. For regional DR, replicate the S3 lake with Cross-Region Replication and stand the pipeline up in the second region from replicated Parquet. The record-level safety net is the poison-pill DLQ: one bad record can never wedge a shard, so reliability holds at the record grain, not just the cluster grain.

Practice challenges

Work these in order — they climb from arithmetic a beginner can do on paper to design decisions a senior engineer defends in review. Try each before opening the solution.

Challenge 1 — Size the stream (Beginner). A connected-fitness product emits 12,000 device readings/second, each ~500 bytes. How many provisioned shards do you need, and which limit is binding?