AWS Enterprise Architecture: Event-Driven Serverless

The phrase “event-driven serverless” gets sold as a billing model — “you only pay when code runs” — and that framing causes most of the bad architectures I review. Teams reach for Lambda to save money, wire every function to call the next one synchronously, and end up with a distributed monolith billed by the millisecond: the same tight coupling as before, now spread across forty functions, each holding a connection open and waiting on the one in front of it. The win of this architecture is not the pricing. The win is that the network of facts becomes the integration layer. Services stop calling each other; they emit events about what happened, and other services react. The broker — not a shared database, not a synchronous API mesh — becomes the contract. Done well, you can delete a consumer, add three new ones, or replay yesterday’s traffic into a brand-new service, and nobody upstream knows or cares.

This article is a concrete AWS reference architecture for getting that right. The protagonists are AWS Lambda for stateless compute that scales to zero; Amazon EventBridge as the central event bus, schema registry, and cross-account router; Amazon SQS and SNS for the durable buffering and fan-out that make Lambda reliable under load; Amazon DynamoDB as the single-digit-millisecond operational store with Streams as a first-class event source; and AWS Step Functions for the long-running, multi-step business transactions (sagas) that you must never try to cram into a single function. The running domain is an order-and-fulfilment platform for an omnichannel retailer, because order processing is the canonical workload where commands, events, sagas, idempotency, and read models all show up at once and you cannot hand-wave any of them away.

In a nutshell

Picture the pass in a busy restaurant kitchen. A waiter clips an order ticket to the rail and immediately walks away to serve the next table — they do not stand there waiting for the steak to cook. The grill station, the salad station, and the dessert station each watch the rail, pick up the tickets that concern them, and work at their own pace. Nobody is blocked on anybody. During a dinner rush the tickets simply queue up on the rail and the stations drain them as fast as they can; when the room is empty the stations do nothing and cost nothing. That ticket rail is an event bus, the stations are your services, and the tickets are events. Event-driven serverless architecture is that same idea in AWS: instead of one service phoning another and waiting on the line, a service announces a fact that happened (“OrderCreated”) to a broker, and any number of other services react to it independently.

Contrast that with the way most systems start life: a single program where checkout calls payment, which calls inventory, which calls the email service, all inside one HTTP request while the customer’s browser spins. That is a phone call — synchronous, blocking, and only as fast and as reliable as its slowest link. This lesson is about replacing those phone calls with tickets on a rail, using the AWS primitives built for it: Lambda (code that runs only when an event arrives, then vanishes), EventBridge, SNS, and SQS (the rails, mailboxes, and megaphones that move events around), DynamoDB (a fast database whose change-log itself becomes a source of events), and Step Functions (the head chef who runs a multi-step recipe with a plan for what to do when a step fails).

Why should a beginner care? Because this shape is how modern systems survive traffic spikes without falling over, let separate teams ship without stepping on each other, and cost almost nothing when idle. It is also the single most misunderstood architecture in the cloud — “serverless” sounds like a billing trick, but the real prize is the decoupling. Get the mental model right early and much of the rest of your cloud career gets easier.

Level: Advanced · Time: ~58 min read

Prerequisites — know these first:

• The three messaging services and what each is for — SNS, SQS & EventBridge messaging fundamentals.
• How Lambda is triggered, scales, and is billed — Lambda deep dive: runtimes, triggers, concurrency.
• Comfort reading JSON and a little Python; you do not need to have built any of this before.
• Optional but useful for the deeper half: the bus itself — EventBridge: buses, schema, pipes — and orchestration — Step Functions orchestration & error handling.

After this lesson you will be able to:

• Explain the difference between a command and an event, and between choreography and orchestration, and say which AWS service fits each.
• Choose correctly between SQS, SNS, EventBridge, and Kinesis for a given piece of work, instead of forcing everything through one broker.
• Design an at-least-once pipeline that is safe because its consumers are idempotent, with dead-letter queues on every hop.
• Explain how a Lambda event source mapping batches records and reports partial failures, and why that one setting fixes the most common correctness bug in the pattern.
• Recognise the failure modes — poison messages, retry storms, downstream throttling — and name the specific setting that contains each one.

The business scenario

Lakeside Outfitters is a fictional but representative company: an outdoor-gear retailer doing roughly USD 60M in annual GMV across a website, an iOS/Android app, 32 physical stores, and two marketplace integrations (Amazon and a regional outdoors marketplace). They run on a three-year-old Django monolith backed by a single PostgreSQL RDS instance and a fleet of EC2 instances behind an Application Load Balancer. On a quiet Tuesday the monolith is fine. The problem is the days that actually matter to the business.

The pain that triggered the rebuild:

• Promotions take the whole site down. During a seasonal sale, checkout, payment capture, inventory decrement, loyalty accrual, fraud scoring, marketplace inventory sync, and the confirmation email all run synchronously inside one HTTP request. When the payment processor slows from 250 ms to 5 seconds, request workers pile up, the RDS connection pool (capped at 100) exhausts, and the entire application — including catalogue browsing, which needs none of that — falls over. One slow downstream takes everything down with it.
• Inventory is wrong everywhere at once. Store POS, the website, and both marketplaces each write stock counts on their own schedule into the same inventory table with their own locking assumptions. Oversells on fast-moving SKUs are routine, and customer service absorbs the cancellations and goodwill credits.
• Every new channel is a quarter-long project. Onboarding a marketplace means threading its calls through the monolith’s request path. The blast radius of any change is the whole application, so releases are monthly and stressful.
• Spiky, unpredictable load. Traffic is near zero at 3 a.m. and 40x baseline when an email campaign lands at 9 a.m. They are paying for a fleet sized for the peak and idle the rest of the day, yet the peak still knocks them over.

The business goals are not “go serverless.” They are: survive a 40x traffic spike on promo days without checkout failing; never oversell; onboard a new sales channel in under two weeks; let four product squads release independently; and stop paying for idle capacity at night. Those goals are precisely what make this event-driven rather than a lift-and-shift to Lambda. Spike survival and the never-oversell guarantee both demand that work be accepted in milliseconds and processed reliably out of band, against a single authoritative stock record that every channel reacts to instead of races against.

Critically, this architecture scales down as cleanly as it scales up. A 15-person startup shipping 800 orders a day deploys the exact same shape — pay-per-request Lambda, on-demand DynamoDB, a default EventBridge bus, standard SQS — for a few hundred dollars a month, and grows into the provisioned-concurrency, multi-Region version without re-drawing the diagram. That is what makes it a reference architecture and not a hyperscaler special.

Architecture overview

The system separates three planes that monoliths smear together: the synchronous request plane (a human is waiting), the asynchronous command plane (work that must happen reliably but not while the user waits), and the event notification plane (facts that many independent consumers react to). Each plane gets the AWS primitive whose semantics fit, instead of forcing every interaction through one broker.

The end-to-end flow for placing an order:

1. Ingress and edge. A client (web, mobile, store POS, marketplace webhook) hits Amazon CloudFront (global CDN, TLS, AWS WAF for the OWASP rules and rate-based blocking) in front of Amazon API Gateway. API Gateway terminates the API, validates the JWT against an Amazon Cognito user pool (or a Lambda authorizer for partner API keys) using a cached authorizer result, enforces per-key usage plans and throttling, and validates the request body against a JSON Schema model so malformed payloads never reach compute. This is the only public door.

2. Accept fast, decouple immediately. For the write path, API Gateway does not invoke a “do everything” Lambda. It uses a direct service integration to put the validated request onto the command plane — either PutEvents to EventBridge or SendMessage to an SQS queue — via an IAM role, with no Lambda in the hot path at all. The client gets a 202 Accepted with an order ID in well under 100 ms. The order is now durably captured; everything else happens out of band. (Read paths — catalogue, order status — do invoke Lambda, but against DynamoDB read models, not the write path.)

3. The command lands and the order is created. An OrderIngest Lambda consumes the command from SQS (which gives it batching, retries, and a dead-letter queue for free), validates business rules, and performs a conditional write to the Orders DynamoDB table — attribute_not_exists(PK) keyed on a client-supplied idempotency key — so a retried or duplicated submission can never create two orders. This is where idempotency is enforced, not hoped for.

4. DynamoDB Streams turns the write into an event. The committed order write appears on the DynamoDB Stream. A thin OrderEventPublisher Lambda (or an EventBridge Pipe — more on that below) reads the stream and emits a well-typed OrderCreated event to the EventBridge custom bus. This is the transactional-outbox pattern done natively: the event is published because and only because the database commit succeeded, so the store and the bus can never disagree.

5. Fan-out to independent reactors. On the EventBridge bus, content-based rules route OrderCreated to every interested consumer, each on its own SQS queue (the “fan-out with buffering” pattern, SNS-style but on EventBridge):

   • Inventory service reserves stock with a conditional UpdateItem and emits StockReserved or StockReservationFailed.
   • Loyalty service accrues points.
   • Notifications service sends the confirmation (via SNS → email/SMS).
   • Analytics archives the raw event to S3 through Kinesis Data Firehose.
   • Marketplace sync pushes the new stock level outward. Each consumer is a separate squad’s code, deployed independently, ignorant of the others. Adding a fifth is a new rule and a new queue — zero changes upstream.

6. The multi-step transaction runs as a saga. Order fulfilment is not one event; it is a sequence with money and physical goods at stake — reserve stock, capture payment, allocate from a warehouse, generate a shipping label, and compensate (refund, release stock) if any step fails. That orchestration lives in an AWS Step Functions state machine (Standard workflow), kicked off by the OrderCreated event. Step Functions owns the retries, timeouts, parallel branches, human-approval waits, and — crucially — the compensating transactions. None of that belongs in a tangle of Lambdas calling Lambdas.

7. Read models for queries. Consumers also project events into purpose-built DynamoDB read models — an OrderStatusView, an InventoryByStore view — so the synchronous read API serves single-digit-millisecond queries without ever touching the write path or doing cross-service joins. This is CQRS: the write model and the read models are different shapes, kept eventually consistent by the event stream.

Drawn out, the diagram is three horizontal bands. Top band (request plane): Client → CloudFront/WAF → API Gateway → (reads) Lambda → DynamoDB read models; (writes) direct integration → SQS/EventBridge, returning 202. Middle band (command + event plane): SQS → OrderIngest Lambda → Orders table → DynamoDB Stream → EventBridge custom bus, which fans out through rules to per-consumer SQS queues, each draining into its own Lambda. Bottom band (orchestration): the OrderCreated event also starts a Step Functions saga that calls the inventory, payment, and shipping services with built-in retry/compensation, writing terminal results back as events. Cross-cutting all three: CloudWatch + X-Ray for traces and metrics, DLQs on every async hop, and EventBridge Archive capturing every event for replay.

Core concepts: commands, events, and the four messaging primitives

Before wiring services together it pays to nail the vocabulary, because the whole architecture turns on two distinctions that beginners routinely blur.

Command vs event. A command is a request to do something — “PlaceOrder”, “ChargeCard”. It is addressed to exactly one handler, it expects the thing to happen, and the sender often cares about the result. An event is a statement that something already happened — “OrderPlaced”, “PaymentCaptured”. It is addressed to no one in particular; the emitter has moved on and does not know or care who is listening. Commands are imperative and point-to-point; events are past-tense and broadcast. The single biggest mindset shift in this whole lesson is to stop having services tell each other what to do and start having them announce what they did, letting others decide how to react. Get the tense right — past tense for events — and the coupling melts away on its own.

Producer, consumer, and the thing in the middle. A producer (or publisher) emits messages; a consumer (or subscriber) processes them. In between sits a broker, and AWS gives you four with genuinely different semantics. The art is matching the primitive to the shape of the work:

Read that table as four answers to four different questions. “I need to hand work to a pool of workers and not lose it if they’re slow” → SQS. “I need one message to reach many subscribers right now, or to text a human” → SNS. “I need to route business events to consumers based on what’s in the event, with a schema and replay” → EventBridge. “I have a firehose of records that must stay in order and be re-readable” → Kinesis. Forcing all four jobs through one service is the most common architectural smell in this space.

The three integration patterns, built from those primitives:

• Point-to-point — one producer, one queue, one consumer pool. A command lands on an SQS queue; one worker fleet drains it. This is the command plane (“process this order”).
• Publish/subscribe (fan-out) — one event, many independent reactors. OrderCreated hits the EventBridge bus (or an SNS topic) and inventory, loyalty, notifications, and analytics each get their own copy on their own queue. Adding a fifth reactor changes nothing upstream. This is the event plane and it is where the decoupling lives.
• Event streaming — an ordered, replayable log that consumers read at their own pace, keeping their own position. Kinesis (or DynamoDB Streams, or Kafka/MSK) for clickstream, IoT telemetry, or change-data-capture where order and replay matter more than per-message routing.

The reference architecture in this lesson uses all three at once, deliberately: SQS for the command plane, EventBridge fan-out for the event plane, and DynamoDB Streams (a log) as the bridge between the database and the bus. That is not over-engineering — it is matching three different kinds of work to the three primitives whose semantics fit, which is the entire discipline.

Component breakdown

A few of these choices carry the design and deserve the why, not just the what.

EventBridge is the bus, not SNS — but SNS still has a job. People ask why both. EventBridge gives you content-based routing on the event payload (a JSON rule like {"detail":{"orderValue":[{"numeric":[">",500]}]}}), a schema registry, 24-hour-plus retry with DLQ per target, and archive-and-replay. That makes it the right backbone for integration events between services. SNS is simpler, higher-throughput, and lower-latency for raw fan-out, and it is the natural fit for end-user notifications (it speaks email/SMS/push directly) and for the classic SNS→SQS fan-in when many queues need the same message at very high rates. Rule of thumb in this architecture: EventBridge for service-to-service business events; SNS for notifications and ultra-high-fanout pub/sub. Putting everything on one and ignoring the other is the common mistake.

SQS sits in front of nearly every Lambda for a reason. EventBridge can invoke Lambda directly, but a raw async invoke gives you only two internal retries and then the event is gone (unless a DLQ catches it). Routing EventBridge rule → SQS → Lambda instead buys you four things that matter under real load: a durable buffer that absorbs a 40x spike while Lambda concurrency catches up; batching (up to 10,000 records / 6 MB per invoke) that slashes invocation count and cost; controlled concurrency via the event-source-mapping maximumConcurrency, so a downstream database is not stampeded; and a first-class DLQ with maxReceiveCount for poison messages. This single pattern is the difference between “survives the campaign” and “melts at 9:01 a.m.”

DynamoDB single-table design with Streams as the outbox. The Orders and Inventory data lives in a DynamoDB single-table design (composite PK/SK, plus GSIs for access patterns like “orders by customer” and “stock by store”). Two properties make it the right core: conditional writes give you optimistic concurrency and idempotency without a lock, and Streams give you an ordered, exactly-once-per-shard change log that you turn into events. That last point is the transactional outbox solved with zero extra infrastructure — you do not need a separate outbox table and poller, because the table is the log.

Step Functions, not a Lambda chain, for the saga. The temptation is to have the payment Lambda invoke the shipping Lambda invoke the label Lambda. That recreates the synchronous monolith with worse failure modes (a 15-minute Lambda timeout ceiling, no built-in compensation, opaque debugging). A Standard state machine instead gives you durable execution that survives for up to a year, declarative Retry/Catch, parallel branches, .waitForTaskToken for steps that wait on a human or an external callback, and a visual execution history that is the difference between a 5-minute and a 5-hour incident postmortem. Use Express workflows only for the high-volume, short, idempotent orchestrations where you do not need the per-execution audit trail.

Implementation guidance

Compute. Functions are Python 3.13 (or Node 22) on ARM/Graviton2 — roughly 20% cheaper per GB-second and usually faster for this workload. Right-size memory with AWS Lambda Power Tuning (a Step Functions state machine that sweeps memory settings against real payloads); for these handlers the cost-optimal point is typically 512–1024 MB, where more memory buys proportionally more CPU and the function finishes faster and cheaper. Adopt Lambda Powertools (Python/TypeScript) on day one for structured logging, EMF custom metrics, tracing, and — importantly — its idempotency and batch-processing utilities, which save you from re-implementing both badly.

Idempotency is non-negotiable and lives in three places. At-least-once delivery is the law of this land: SQS, EventBridge, and DynamoDB Streams can all hand you the same message twice. (1) The write path enforces it structurally with the DynamoDB conditional attribute_not_exists on the idempotency key. (2) Side-effecting consumers (charge a card, send an email) wrap their handler with the Powertools idempotency utility backed by a DynamoDB idempotency table with TTL, so a replay is a no-op. (3) The saga makes each task idempotent and uses idempotency keys on external calls (e.g. the payment processor’s Idempotency-Key header) so a Step Functions retry never double-charges.

Always-on partial batch responses. When Lambda reads a batch of 10 SQS messages and message 7 fails, the naive behaviour re-delivers all 10 — re-processing the 9 that succeeded. Set ReportBatchItemFailures on the event-source mapping and return the failed message IDs in batchItemFailures; only the genuine failures are retried. Forgetting this is the single most common correctness bug I see in SQS→Lambda pipelines.

A few IaC snippets (Terraform) that capture the load-bearing wiring. First, the EventBridge rule that routes OrderCreated to a buffered consumer queue, with a DLQ and retry policy on the target (the part people omit):

resource "aws_cloudwatch_event_rule" "order_created" {
  name           = "order-created-to-inventory"
  event_bus_name = aws_cloudwatch_event_bus.orders.name
  event_pattern  = jsonencode({
    "source"      = ["lakeside.orders"],
    "detail-type" = ["OrderCreated"]
  })
}

resource "aws_cloudwatch_event_target" "to_inventory_queue" {
  rule           = aws_cloudwatch_event_rule.order_created.name
  event_bus_name = aws_cloudwatch_event_bus.orders.name
  arn            = aws_sqs_queue.inventory.arn

  dead_letter_config { arn = aws_sqs_queue.inventory_dlq.arn }
  retry_policy {
    maximum_retry_attempts       = 10
    maximum_event_age_in_seconds = 3600
  }
}

Second, the SQS → Lambda event-source mapping that pins concurrency and turns on partial-batch reporting (the two settings that keep a downstream store safe and retries correct):

resource "aws_lambda_event_source_mapping" "inventory_consumer" {
  event_source_arn                   = aws_sqs_queue.inventory.arn
  function_name                      = aws_lambda_function.inventory.arn
  batch_size                         = 10
  maximum_batching_window_in_seconds = 5
  function_response_types            = ["ReportBatchItemFailures"]

  scaling_config { maximum_concurrency = 20 } # protect the DynamoDB write path
}

Third, the direct API-Gateway-to-SQS integration that keeps Lambda out of the write hot path entirely (sub-100 ms 202, nothing to cold-start):

resource "aws_apigatewayv2_integration" "place_order" {
  api_id              = aws_apigatewayv2_api.public.id
  integration_type    = "AWS_PROXY"
  integration_subtype = "SQS-SendMessage"
  credentials_arn     = aws_iam_role.apigw_to_sqs.arn
  request_parameters  = {
    "QueueUrl"    = aws_sqs_queue.order_ingest.url
    "MessageBody" = "$request.body"
  }
}

Package functions with AWS SAM or the Serverless Framework for fast local iteration (sam local invoke, sam local start-api), but keep the shared, account-level platform — the buses, VPC, IAM boundaries, Organizations guardrails — in Terraform so the platform team owns it independently of the squads’ function repos. Wire CI/CD as CodePipeline → CodeBuild (or GitHub Actions → OIDC into AWS, no long-lived keys), deploying each function behind a Lambda alias with weighted/canary shifting and a CloudWatch alarm that auto-rolls-back on an error-rate breach.

Networking and identity. Most of this is internet-facing-AWS-managed and needs no VPC — Lambda, EventBridge, SQS, SNS, DynamoDB, and Step Functions are all reachable over IAM-authenticated AWS APIs, and keeping functions out of a VPC removes the ENI cold-start tax. The moment a Lambda must reach a private resource (an RDS replica, an internal microservice, the on-prem ERP over Direct Connect), attach it to private subnets and reach AWS services through VPC Gateway/Interface Endpoints (PrivateLink) so traffic never leaves the AWS network — there is a Gateway Endpoint for DynamoDB and S3 (free) and Interface Endpoints for SQS, SNS, EventBridge, and Step Functions. Identity is least-privilege IAM per function: each Lambda gets its own execution role scoped to exactly the one queue it drains and the one table partition it writes, never a shared “Lambda can do anything” role. Cross-account event flow (e.g. a central security account subscribing to all order events) is a resource policy on the EventBridge bus granting PutEvents/rule creation to specific account IDs — no credential sharing.

How Lambda actually consumes events

“Lambda is triggered by an event” hides a lot of machinery, and understanding it is the difference between a pipeline that quietly loses messages and one that doesn’t. There are three different ways an event reaches your function, and — this is the part that catches everyone — they have different error-handling contracts.

Three invocation models.

• Synchronous — the caller waits for the response (API Gateway → Lambda, an SDK Invoke). Retries are the caller’s problem; if the function errors, the error comes straight back.
• Asynchronous — the caller drops the event in Lambda’s internal queue and gets an immediate acknowledgement (SNS → Lambda, an EventBridge rule → Lambda, S3 → Lambda). Lambda retries the function twice on failure with backoff, then the event is gone unless you attach an on-failure destination or a DLQ.
• Event source mapping (poll-based) — Lambda runs a managed poller that reads from a source which cannot push (SQS, Kinesis, DynamoDB Streams, MSK/Kafka), assembles a batch, and invokes your function with it. This is where the reference architecture lives, and it has the richest error handling. The poller is Lambda-managed infrastructure — you do not pay for it and there is no server to run.

Batching. The poller does not invoke once per message; it hands your function a batch, so you amortise fixed costs (cold-start init, opening a database connection) across many records. Fewer, fuller invokes mean lower cost and higher throughput. The behaviour is tuned by a handful of event-source-mapping settings, and the two source families differ sharply:

Read the two right-hand columns carefully. SQS error handling is the queue’s job: a message that keeps failing is retried according to the queue’s visibility timeout, its receive count climbs, and at maxReceiveCount the queue itself moves it to the DLQ. There is no “maximum retry attempts” or “bisect” on an SQS mapping. Stream sources (Kinesis, DynamoDB Streams) are the opposite: because a stream is ordered, one poison record blocks its whole shard until it succeeds, ages out, or exhausts retries — so here you get MaximumRetryAttempts, MaximumRecordAgeInSeconds, BisectBatchOnFunctionError (repeatedly halve the batch to quarantine the one bad record), and an OnFailure destination to catch what’s dropped. Applying SQS’s mental model to a stream (or vice-versa) is how shards silently stall for hours.

Partial batch response, worked step by step. Say a batch of ten SQS messages arrives and the seventh one throws. The naïve handler lets the exception escape, so the whole batch’s messages become visible again on the queue, and all ten are redelivered — the nine that already succeeded get processed a second time. If those nine each sent a confirmation email, nine customers get a duplicate. The fix is one setting plus a few lines of code:

import json

def handler(event, context):
    failures = []
    for record in event["Records"]:
        try:
            process(json.loads(record["body"]))
        except Exception:
            # report ONLY this message; the rest are acked and deleted from the queue
            failures.append({"itemIdentifier": record["messageId"]})
    return {"batchItemFailures": failures}

with function_response_types = ["ReportBatchItemFailures"] on the mapping. Now only message seven returns to the queue, its receive count increments, and after maxReceiveCount attempts it lands in the DLQ. AWS Lambda Powertools’ BatchProcessor wraps this exact contract so you never hand-roll the loop. Forgetting partial batch responses is the single most common correctness bug in SQS→Lambda pipelines — the Implementation section flags it too, and now you know precisely why it bites.

Poison messages and the dead-letter queue. A poison message is one that will never succeed no matter how many times you retry it — malformed JSON, a reference to a record that was deleted, a bug that only that exact payload triggers. Without a ceiling it would loop forever, burning money and (on a stream) blocking every record behind it. The maxReceiveCount → DLQ on SQS (and MaximumRetryAttempts + OnFailure on streams) is that ceiling: after N tries the message is set aside in a dead-letter queue so the pipeline keeps flowing, an alarm on DLQ depth pages a human, and once the bug is fixed you redrive the DLQ (SQS has a built-in message-move task) back to the source queue to reprocess it. A pipeline without DLQs does not avoid poison messages — it just loses them silently, or wedges on them.

Delivery semantics: at-least-once, ordering, and why idempotency is the price of admission

Every broker in this architecture makes the same promise, and it is not the one beginners expect. The promise is at-least-once delivery: your message will be delivered, but occasionally more than once. SQS standard queues, EventBridge, SNS, and stream retries can all hand you a duplicate — a network blip acks late, a batch partially fails and replays, a visibility timeout lapses while you were still working. “Exactly-once” is largely a myth in distributed systems; the honest engineering response is not to chase it but to make processing the same message twice produce the same result as processing it once. That property is idempotency, and in this architecture it is not optional — skip it and you will double-charge a customer the first time a retry fires in production.

Three places idempotency lives — the Implementation section lists them; here is how each one actually works:

1. The write, made idempotent by a conditional put. The client generates an idempotency key (a UUID per order attempt) that rides along on the command. The ingest handler writes with a condition:

   table.put_item(
       Item={"PK": f"ORDER#{order_id}", "SK": "META", "status": "NEW"},
       ConditionExpression="attribute_not_exists(PK)")
   

   The first write wins. A duplicate submission raises ConditionalCheckFailedException, which the handler swallows as a no-op — the order already exists, nothing to do. No lock, no race. This is optimistic concurrency, and it is the cheapest idempotency you will ever buy.

2. The side effect, made idempotent by an idempotency store. A conditional write protects the database, but “charge the card” and “send the email” happen outside DynamoDB — you cannot condition-put an email. Here you wrap the handler with an idempotency layer keyed on the event’s stable ID, backed by a small DynamoDB table with a TTL:

   from aws_lambda_powertools.utilities.idempotency import (
       idempotent, DynamoDBPersistenceLayer, IdempotencyConfig)
   
   persistence = DynamoDBPersistenceLayer(table_name="idempotency")
   config = IdempotencyConfig(event_key_jmespath="detail.idempotencyKey")
   
   @idempotent(persistence_store=persistence, config=config)
   def handler(event, context):
       charge_card(event["detail"])          # runs exactly once per key
       return {"receipt": "ok"}
   

   On a replay, Powertools sees the key already marked COMPLETE and returns the stored result without re-running the body. It also writes an IN_PROGRESS marker so two concurrent copies of the same event cannot both charge.

3. The external call, made idempotent by the provider’s key. Payment processors and most serious APIs accept an Idempotency-Key header; pass the same key on a Step Functions retry and the processor de-dupes on its side. Idempotency is a chain — every link that has a side effect needs its own guarantee.

Ordering — and how much you actually need. At-least-once’s ugly sibling is out-of-order. EventBridge, SNS standard, and SQS standard make no ordering promise; StockReserved can arrive before OrderCreated. Two tools restore order when you genuinely need it:

• FIFO queues and topics (SQS FIFO, SNS FIFO). Messages that share a message group ID are delivered in order and processed exactly-once (deduplication within a 5-minute window). The catch is throughput: ordering is per-group, so you get parallelism across groups but serialisation within one — pick a group ID with enough cardinality (per-customer, per-order) to parallelise, never one global group that serialises everything.
• Kinesis / DynamoDB Streams partition keys. A stream keeps strict order per shard, and the partition key decides the shard. All events for order#123 share a key, land on one shard, and stay ordered relative to one another; different orders spread across shards for throughput. Same trade-off, same lever: choose a partition key with high cardinality or you create a hot shard.

The design instinct to cultivate: need the least ordering you can get away with. Global ordering is expensive and rarely required. Usually you need order only within one entity (one order, one account), which is exactly what a per-entity group ID or partition key gives you — full throughput everywhere else. Reaching for a single FIFO group “to be safe” throttles the whole system to one message at a time.

Enterprise considerations

Security and Zero Trust. Identity is the perimeter, because there is barely a network one. Every hop authenticates with IAM and authorizes with a scoped policy; there are no implicit trust zones. Concretely: the public edge has WAF + Cognito/JWT validation + request-schema validation, so unauthenticated or malformed traffic dies at the door. Every internal interaction is an IAM-signed AWS API call — a compromised inventory function holds a role that can drain one queue and conditional-write one table prefix, nothing more, so its blast radius is bounded by policy, not by hope. Data is encrypted with customer-managed KMS keys at rest (DynamoDB, SQS, SNS, S3) and in transit via TLS everywhere. Secrets (payment-processor keys, marketplace tokens) live in Secrets Manager with automatic rotation and are fetched at runtime, never baked into env vars in plaintext. Events on the bus carry no card numbers or raw PII — they carry references (an order ID, a tokenised payment handle), so the event log is not a liability. Guardrails are enforced org-wide with Service Control Policies (deny public S3, deny disabling CloudTrail, require KMS), and GuardDuty + Security Hub watch the whole account continuously.

Cost optimization. This is where the architecture’s economics shine, and where the naive version quietly bleeds money. The headline win is scale-to-zero: at 3 a.m. you pay essentially nothing — no idle EC2 fleet. You pay per Lambda invocation-ms, per million SQS/EventBridge messages, and per DynamoDB request unit. The non-obvious levers: (1) Batch aggressively — a Lambda that processes 10 SQS messages per invoke costs a tenth of one that processes them one at a time; the maximumBatchingWindow lets you trade a little latency for far fewer invocations. (2) Graviton + Power Tuning typically cuts compute 20–40% with no code change. (3) Move from DynamoDB on-demand to provisioned with auto-scaling once traffic is predictable enough — on-demand is perfect for spiky/unknown load but costs ~5–7x per request at sustained high volume; this is the single biggest line-item swing at scale. (4) EventBridge Pipes replace “glue” Lambdas (stream→transform→target) with a managed integration you do not pay per-invoke for. (5) Use Express Step Functions for the high-volume short orchestrations (priced by duration/memory, far cheaper than Standard’s per-state-transition charge) and reserve Standard for the durable, auditable, long-running ones. A realistic bill for this platform at ~3,000 orders/day with promo spikes lands around USD 1,800–2,500/month, versus the over-provisioned EC2 fleet it replaced.

Scalability. Each plane scales on its own axis. Lambda scales out to thousands of concurrent executions (default 1,000/account/Region, raised on request) and, with SQS as the buffer, a spike does not drop work — it queues it, and the consumers drain it as concurrency ramps. The two things you must actively manage: reserved/provisioned concurrency to protect latency-critical reads (and to cap functions that hit a fragile downstream), and the downstream you are protecting — set the event-source-mapping maximumConcurrency so 5,000 queued messages do not translate into 5,000 simultaneous writes against a database that tops out at 500. DynamoDB on-demand absorbs the spike natively (it adapts to traffic), which is exactly why it is the default here. The design’s superpower is that the write acceptance path (API GW → SQS) has effectively unbounded throughput and near-zero latency regardless of how backed-up the processing is — the customer’s 202 never slows down because fulfilment is busy.

Reliability and DR (RTO/RPO). Within a Region, every managed service here is already multi-AZ — SQS, EventBridge, DynamoDB, Step Functions, and Lambda all replicate across Availability Zones with no work from you, so a single-AZ failure is a non-event. The deliberate reliability work is at the message level: a DLQ on every async hop (EventBridge target, SQS consumer, Lambda async, Step Functions task) so nothing is ever silently lost; alarms on DLQ depth; and EventBridge Archive + Replay so you can re-drive events into a fixed or new consumer. Idempotent consumers make redelivery and replay safe. For multi-Region DR, the cost-effective default is warm standby: DynamoDB Global Tables replicate the source-of-truth data to a second Region (RPO of ~1 second), the same IaC deploys the stack there, and a Route 53 health-check failover repoints the API. Realistic targets: RPO ≈ seconds (Global Tables) and RTO ≈ 10–20 minutes (DNS failover + provisioned-concurrency warm-up). The subtle bit is the event bus: use a second custom bus in the DR Region and cross-Region EventBridge replication (bus-to-bus) so in-flight events are mirrored — otherwise you fail over the data but lose the events in flight.

Observability. Asynchronous, distributed systems are invisible without deliberate instrumentation, and “tail the logs” does not work when one order touches eleven functions. Three pillars: (1) A correlation/causation ID stamped on the first request and propagated through every event’s detail — Powertools does this — so you can reconstruct one order’s entire journey across all functions and queues. (2) AWS X-Ray active tracing on Lambda, API Gateway, and Step Functions for the service map and latency breakdown; X-Ray now traces across EventBridge and SQS hops, so the async graph is one trace, not eleven disconnected ones. (3) CloudWatch EMF custom business metrics (orders accepted, stock reservations failed, saga compensations triggered) plus the operational ones that actually page you: SQS ApproximateAgeOfOldestMessage (the truest “are we falling behind” signal), DLQ depth > 0, Lambda error rate and throttles, and Step Functions ExecutionsFailed. Step Functions’ visual execution history is itself a debugging tool — you see which state failed and why.

Governance. The platform team owns the buses, the schema registry, the IAM boundaries, and the Organizations guardrails as code; squads own their functions, queues, and rules within those guardrails. The EventBridge Schema Registry is the linchpin of governance here: every event type has a registered, versioned schema, consumers generate typed bindings from it, and a producer cannot silently change an event’s shape and break six consumers — schema evolution follows the same additive-only discipline as any public contract (add optional fields freely; never remove or retype one without a new version). CloudTrail captures every control-plane action org-wide; AWS Config rules enforce that every queue has a DLQ, every table has PITR enabled, and nothing is unencrypted; cost allocation tags per squad/service make the bill attributable so teams see their own spend.

Reference enterprise example

Lakeside Outfitters committed to the rebuild after a flagship Memorial Day sale: an email blast drove 38x normal traffic at 9 a.m., the RDS connection pool saturated within ninety seconds, checkout returned 500s for forty minutes, and they oversold a popular tent by 340 units because the website and two marketplaces all decremented the same row under contention. The post-incident number that got the CFO’s attention was USD 210,000 in lost orders plus goodwill credits from a single morning.

Their constraints were concrete: a four-squad engineering org (orders, inventory, fulfilment, growth), a hard mandate to keep the existing monolith serving catalogue browsing during the migration (no big-bang cutover), and a CFO ceiling of “the new platform must cost less at steady state than the EC2 fleet it replaces.” They migrated the write path first — the part that actually fell over.

What they built, mapped to this architecture:

• Ingress: CloudFront + WAF + API Gateway (HTTP API) with a Cognito JWT authorizer. The POST /orders route is a direct API-Gateway-to-SQS integration — no Lambda in the hot path — returning 202 with an order ID in a measured p50 of 41 ms / p99 of 88 ms, independent of backend load.
• Ingest + source of truth: OrderIngest Lambda (Python 3.13, ARM, 768 MB after Power Tuning) drains the SQS command queue in batches of 10 with ReportBatchItemFailures, conditional-writes to a single-table Orders DynamoDB (on-demand, PITR on), keyed on a client idempotency key. Duplicate submissions are silently no-ops.
• Outbox: an EventBridge Pipe reads the Orders DynamoDB Stream, filters to committed INSERT/MODIFY events, and publishes typed OrderCreated / OrderUpdated events to the lakeside.orders custom bus. No glue-Lambda to maintain.
• Fan-out: EventBridge rules route to four per-squad SQS queues (inventory, loyalty, notifications, analytics), each with a DLQ and a concurrency-pinned consumer. Inventory’s consumer does a conditional UpdateItem to reserve stock and emits StockReserved / StockReservationFailed — one authoritative stock record, reacted to, never raced against. Oversells went to zero.
• Saga: a Standard Step Functions workflow, triggered by OrderCreated, runs reserve-stock → capture-payment → allocate-warehouse → create-shipping-label, with Retry on transient errors, Catch → compensating refund + stock-release on hard failure, and .waitForTaskToken on a manual-review branch for orders flagged by fraud scoring. Every execution is visible end-to-end in the console.
• Notifications & marketplace sync: SNS handles the customer confirmation (email/SMS); a marketplace-sync consumer pushes new stock levels outward, reacting to StockReserved.
• Reads: a GET /orders/{id} Lambda serves an OrderStatusView DynamoDB read model (provisioned concurrency = 5 to kill cold-start tail latency on the customer-facing path).

The growth squad onboarded a third marketplace six weeks later by adding one EventBridge rule and one consumer Lambda — zero changes to orders, inventory, or fulfilment code, which is the entire point of the event backbone. The next seasonal sale drove 44x baseline traffic; checkout p99 stayed under 95 ms, the SQS queues peaked at ~9,000 messages and drained in under three minutes as Lambda concurrency ramped, and not a single order was lost or oversold. Steady-state cost settled at ~USD 2,100/month against the old fleet’s ~USD 5,400, clearing the CFO’s bar with room to spare. The one scar they earned: an early version emitted full customer addresses on the bus, which a security review flagged; they refactored to emit references and fetch PII inside the consumer that needed it — the “events carry references, not payloads” lesson, learned the way most teams learn it.

When to use it

Use this architecture when your workload is genuinely event-shaped: many independent reactions to business facts, spiky or unpredictable load, a need to onboard consumers without touching producers, and a tolerance for eventual consistency between services. Order processing, IoT ingestion, media pipelines, real-time fraud and notifications, SaaS activity feeds, and any “fan-out to N teams” integration are the sweet spot. It is also the right starting point for a small team precisely because it scales to zero — you pay for traffic, not for a fleet sitting idle, and you grow into the enterprise version without re-architecting.

The trade-offs are real and you should price them in. Eventual consistency is a feature, not a bug, but it is a cognitive tax: a customer may see “order placed” before the loyalty points appear, and your product and support teams must be fine with that. Debugging is harder than a monolith’s stack trace — which is exactly why the correlation-ID + X-Ray + Step Functions-history discipline above is not optional. And at-least-once delivery means you must build idempotency; if you skip it, you will double-charge a customer in production. There is no version of this architecture that is correct without idempotent consumers.

Anti-patterns to avoid:

• The distributed monolith. Lambda-A synchronously invokes Lambda-B invokes Lambda-C, each waiting on the next. You have rebuilt the monolith with worse latency, a 15-minute timeout ceiling, and no compensation. If a flow is a sequence of steps with rollback, it is a Step Functions saga, not a Lambda chain.
• One broker for everything. Forcing high-fanout notifications through EventBridge, or routing nuanced business events through raw SNS, or — worst — using a DynamoDB table as a message queue. Match the primitive to the semantic.
• EventBridge → Lambda with no SQS buffer for anything that can spike or whose downstream is fragile. You lose batching, controlled concurrency, and a real DLQ, and you stampede the database on the first campaign.
• “Eventual consistency” used to dodge a requirement that is actually strongly consistent. Moving money between two accounts in one atomic step is a transaction, not a saga of independent events — use a DynamoDB TransactWriteItems (or a Step Functions saga with explicit compensation), and be honest about which it is.
• Forgetting partial batch responses, which silently re-processes successful messages on every batch with one failure.

Alternatives, and when they win. If your workload is a steady, predictable, high-throughput stream (always-on at scale, not spiky), a containerised event-driven stack on ECS/EKS with Kafka (MSK) can be cheaper per unit and gives you Kafka’s log-replay and consumer-group semantics — at the cost of running and patching the platform. If you need strict global ordering and stream replay as a first-class primitive, Kinesis Data Streams (or MSK) beats SQS/EventBridge. If the system is genuinely a handful of synchronous request/response APIs with no fan-out and strong-consistency needs throughout, a modular service on Fargate or a well-factored monolith is simpler and you should not reach for an event bus at all — the operational and cognitive overhead of async only pays off when there are real, independent consumers reacting to real events. Choose the broker, the consistency model, and the compute on the shape of the work, not on a slide that says “serverless is cheaper.”

Going deeper

Everything so far gets a correct system running. This section is the senior-engineer layer — the internals, edge cases, and failure modes that separate a demo from a platform that survives a promo-day surge.

The transactional outbox, and why DynamoDB Streams is the outbox. The oldest bug in event-driven systems is the dual-write: your handler writes the order to the database and publishes OrderCreated to the bus as two separate operations. If the process dies between them, you have either saved an order nobody hears about or announced an order that was never saved. There is no ordering of the two writes that is safe. The transactional outbox fixes it by writing the event into the same database transaction as the business data (a row in an outbox table), then a separate process reads the outbox and publishes — so the event exists if and only if the data was committed. In DynamoDB you get this for free: the Stream is an ordered, durable log of every committed change, so you do not need a separate outbox table at all. A committed write appears on the stream; an EventBridge Pipe reads it and publishes. The store and the bus can never disagree because the bus is strictly downstream of the commit. If you ever do need an explicit outbox (spanning several tables, or an event richer than the row), write the business items and an outbox item together in one TransactWriteItems (up to 100 actions, fully ACID) and stream that.

EventBridge Pipes: source → filter → enrich → target. A Pipe is managed point-to-point plumbing that replaces the “glue Lambda” you would otherwise write to move records from a stream or queue onto the bus. Its four stages are worth memorising, because each removes code you would otherwise own and pay to run:

resource "aws_pipes_pipe" "orders_outbox" {
  name     = "orders-outbox"
  role_arn = aws_iam_role.pipe.arn
  source   = aws_dynamodb_table.orders.stream_arn
  target   = aws_cloudwatch_event_bus.orders.arn

  source_parameters {
    dynamodb_stream_parameters {
      starting_position = "LATEST"
      batch_size        = 100
    }
    filter_criteria {
      filter { pattern = jsonencode({ eventName = ["INSERT", "MODIFY"] }) } # skip REMOVE
    }
  }

  target_parameters {
    eventbridge_event_bus_parameters {
      source      = "lakeside.orders"
      detail_type = "OrderCreated"
    }
  }
}

The filter stage is a cost lever, not just a convenience: it runs before any enrichment or target invocation, so a Pipe that only cares about INSERTs never pays to process the REMOVEs. The enrich stage (a Lambda, a Step Functions workflow, or an API destination) lets you fetch the extra fields a downstream needs so the target event is complete — instead of a fan-out of consumers all calling back for the same data.

The claim-check pattern for large payloads. Every broker here caps message size at 256 KB — SQS, SNS, and EventBridge all land at 256 KB per message or entry. A 4 MB product-import blob or a base64-encoded image will not fit. The claim-check pattern: write the big payload to S3, put only the pointer (bucket + key, plus a checksum) on the bus, and let the consumer fetch it if it needs the body:

{ "detail": { "bucket": "lakeside-imports", "key": "catalog/abc.json",
              "sha256": "9f2c...", "sizeBytes": 5242880 } }

Consumers that only route on metadata never fetch it; the one that needs the body does a GetObject. The SQS Extended Client Library automates this for queues — offloading anything over a threshold to S3 and rehydrating it on receive. A bonus: your event log and every DLQ stay free of giant blobs, so archive and replay stay cheap.

Choreography vs orchestration — the decision, not the dogma. Choreography is the fan-out you have already seen: services react to events with no central conductor. It is loosely coupled and easy to extend, but no single place knows “where is order 123 in the process,” and one business flow’s logic ends up smeared across ten consumers. Orchestration (Step Functions) puts one state machine in charge of a sequence: it knows the current step, owns the retries and timeouts, and — the killer feature — runs compensating transactions when a later step fails (refund the payment, release the stock). The rule that keeps you sane: choreograph between bounded contexts, orchestrate within one business transaction. OrderCreated fanning out to inventory, loyalty, and analytics is choreography (independent reactions that do not roll back together). “Reserve stock → capture payment → allocate warehouse → ship, and undo it all on failure” is one transaction with money at stake — that is a Step Functions saga, never a chain of Lambdas invoking Lambdas.

Schema evolution: the event is a public contract. The moment a second team consumes your event, its shape becomes an API you cannot casually change. The EventBridge Schema Registry stores a versioned schema per event type (it can even infer schemas from live traffic) and generates typed code bindings, so a consumer breaks at compile time, not at 2 a.m. in production. The discipline is the same additive-only rule as any public API: add optional fields freely; never remove a field, rename one, or change its type or meaning without publishing a new version (a new detail-type like OrderCreated.v2, or a version attribute in the envelope) and running both until consumers migrate. Put a stable event ID and a version in every envelope from day one — retrofitting them after you have fifty consumers is a migration project.

Retry storms, backpressure, and protecting the fragile downstream. At-least-once plus automatic retries has a dark side: when a downstream slows down, every consumer retries, retries add load, and load slows the downstream further — a retry storm that turns a hiccup into an outage. Three levers contain it. (1) The event-source-mapping MaximumConcurrency caps how many consumers hit a database at once, so 9,000 queued messages become 20 concurrent writes, not 9,000 — the queue is your backpressure, letting work wait safely instead of stampeding. (2) Exponential backoff with jitter on retries spreads them out instead of synchronising every client into a thundering herd. (3) A DLQ ceiling stops any single message from retrying forever. The truest early-warning signal is ApproximateAgeOfOldestMessage climbing — alarm on message age, not just depth, because age is the real “are we falling behind” metric while depth can look fine as messages churn.

EventBridge Scheduler for time-driven events. Not every event comes from a service; some come from the clock. EventBridge Scheduler (the successor to scheduled rules) fires one-time or recurring events at scale — millions of schedules, per-schedule time zones, one-time “run at this exact timestamp,” and a flexible time window to smear load off the top of the hour:

resource "aws_scheduler_schedule" "nightly_reconcile" {
  name                         = "nightly-inventory-reconcile"
  flexible_time_window { mode = "OFF" }
  schedule_expression          = "cron(0 2 * * ? *)"
  schedule_expression_timezone = "Asia/Kolkata"

  target {
    arn      = aws_lambda_function.reconcile.arn
    role_arn = aws_iam_role.scheduler.arn
  }
}

Use it for the nightly reconciliation sweep, the “cancel unpaid orders after 30 minutes” timer (a one-time schedule created when the order is placed, then self-deleted), and campaign sends — instead of a cron box you have to keep patched and alive.

Observability across an async graph. The hardest part of this architecture is that no single stack trace spans one order’s journey across eleven functions and five queues. Three disciplines make it observable. (1) A correlation ID minted at ingress and copied into every event’s detail (and every log line) so you can search filter @message like /corr-abc123/ across all log groups and reconstruct the whole path; add a causation ID (the ID of the event that caused this one) and you can rebuild the causal tree, not just a flat list. (2) X-Ray active tracing on Lambda, API Gateway, and Step Functions — and X-Ray now propagates trace context through SQS and EventBridge, so the async hops stitch into one service map instead of eleven disconnected traces. (3) The metrics that actually page you are queue-shaped, not CPU-shaped: ApproximateAgeOfOldestMessage (falling behind), DLQ ApproximateNumberOfMessagesVisible > 0 (something is poisoned), Lambda Throttles and Errors, and Step Functions ExecutionsFailed.

Practice challenges

Work these in order; each builds on the last. Try before opening the solution. All ARNs and account IDs are placeholders.

1 (Beginner) — Pick the primitive. For each need, name the AWS service: (a) text a customer when their order ships; (b) hand incoming orders to a worker pool without losing any if the workers are slow; © route only orders over ₹50,000 to a fraud consumer based on the order’s contents; (d) keep a re-readable, ordered log of every click on the site.

<details><summary>Solution</summary>

(a) SNS — it speaks SMS/email/push directly. (b) SQS — durable buffer, one consumer pool, load-levelling. © EventBridge — a content-based rule on the payload, e.g. {"detail":{"total":[{"numeric":[">",50000]}]}}. (d) Kinesis Data Streams — an ordered, replayable log you read at your own offset.

Why: each service’s core semantic — mailbox, fan-out, content router, ordered log — maps to exactly one of these needs; matching them is the whole discipline. </details>

2 (Beginner) — Command or event? Label each, and rewrite any that are badly named: PlaceOrder, OrderShipped, SendInvoiceEmail, PaymentCaptured, DecrementInventory.

<details><summary>Solution</summary>

PlaceOrder = command, OrderShipped = event, SendInvoiceEmail = command, PaymentCaptured = event, DecrementInventory = command. Events are past-tense facts; commands are imperatives. If you find yourself emitting DecrementInventory onto a bus, you have smuggled a command into the event plane — emit OrderPlaced and let the inventory service decide to decrement.

Why: the tense test (past-tense fact vs imperative verb) is the fastest way to catch coupling sneaking back in. </details>

3 (Intermediate) — Fix the double-email bug. A Lambda reads batches of 10 from SQS and sends one email per message. Ops report that whenever a single message in a batch fails, some customers get duplicate emails. What is happening, and what are the two changes?

<details><summary>Solution</summary>

The handler lets an exception escape, so the whole batch returns to the queue and all 10 are redelivered — re-emailing the 9 that already succeeded. Fix: (1) set function_response_types = ["ReportBatchItemFailures"] on the event source mapping and return {"batchItemFailures":[{"itemIdentifier": id}, ...]} for only the failed messages; (2) make the send idempotent (Powertools idempotency keyed on the message or order ID) so even a legitimate redelivery cannot send twice.

Why: partial batch response stops reprocessing the innocent 9; idempotency covers the at-least-once redelivery that will still happen occasionally. </details>

4 (Intermediate) — Choreography or orchestration? You must run: reserve stock → capture payment → allocate warehouse → print label, and if payment fails, release the reserved stock. A teammate proposes: the inventory Lambda emits StockReserved, which triggers a payment Lambda, which emits PaymentCaptured, which triggers allocation, and so on. Critique it and propose the right shape.

<details><summary>Solution</summary>

This is a chain of Lambdas invoking each other via events — a distributed monolith with no place that owns “release the stock if payment fails.” There is no compensation, no single view of progress, and a failure midway leaves reserved stock stranded. Right shape: a Step Functions Standard saga with Retry on transient errors and Catch → a compensation branch (ReleaseStock, RefundPayment), kicked off by the OrderCreated event.

Why: a sequence of steps with rollback is one transaction — orchestrate it; do not smear it across event-triggered functions with no compensator. </details>

5 (Advanced) — Contain a poison message on a stream. A DynamoDB-Streams-triggered Lambda keeps failing on one malformed record, and the shard is now hours behind because that record blocks everything queued behind it. Which event-source-mapping settings fix it, and why does the SQS approach not apply?

<details><summary>Solution</summary>

On a stream source, set BisectBatchOnFunctionError = true (repeatedly halves the batch to isolate the one poison record), MaximumRetryAttempts (a finite ceiling), MaximumRecordAgeInSeconds (discard stale records), and an OnFailure destination (SQS/SNS) to capture what is dropped so you can inspect it. SQS’s maxReceiveCount → DLQ does not apply because a stream is ordered — a bad record blocks its shard rather than being independently redeliverable, so you need bisect plus a retry/age ceiling to get past it.

Why: ordered stream sources fail differently from queues; the poison record halts a shard, so you quarantine it rather than letting it loop. </details>

6 (Advanced) — Design the large-payload path. A new marketplace integration pushes 3–8 MB product-catalogue documents that must flow through the same EventBridge bus as everything else. EventBridge caps an entry at 256 KB. How do you carry these without abandoning the bus?

<details><summary>Solution</summary>

Claim-check. Write the big document to an S3 bucket, then put a small event on the bus carrying only the pointer — {"bucket":"...","key":"imports/abc.json","sha256":"...","sizeBytes":5242880} — plus whatever routing fields the rules need. Consumers that need the body do a GetObject; consumers that only route on metadata never fetch it. For SQS legs, the SQS Extended Client Library automates the offload and rehydrate. Add an S3 lifecycle rule to expire the payloads.

Why: the bus is for facts and routing, not blobs; S3 carries the bytes, the event carries a checksummed reference, and the 256 KB limit stops being a constraint. </details>

Common beginner mistakes

Distinct from a symptom→cause→fix table: these are mental-model errors — the wrong idea, why it bites, and the right model to replace it with.

• “Serverless is just a cheaper way to run my code.” The wrong model treats Lambda as billing-optimised EC2 and keeps the synchronous call graph. The bite: a distributed monolith billed by the millisecond — the same coupling as before, now spread across forty functions holding connections open, each waiting on the one in front. Right model: the prize is decoupling via events, not the price. If your functions still call each other synchronously in a chain, you have not gone event-driven; you have just re-hosted the monolith.

• “I’ll add idempotency later.” The wrong model assumes each message arrives exactly once. The bite: the first production retry double-charges a customer or sends a duplicate shipment. Right model: at-least-once is the contract from day one; idempotency is the price of admission, not a hardening pass. Build the conditional write and the idempotency store before you ship, not after the incident.

• “One broker for everything.” Forcing high-fanout notifications through EventBridge, nuanced business events through raw SNS, or — worst — using a DynamoDB table as a message queue. The bite: you fight the tool’s semantics forever (no routing here, no ordering there, no replay anywhere). Right model: match the primitive to the semantic — SQS to buffer, SNS to fan-out and notify, EventBridge to route business events, Kinesis to stream.

• “EventBridge can invoke my Lambda directly, so why add SQS?” The wrong model sees the queue as needless plumbing. The bite: a raw async invoke gives two retries then silence — no batching, no concurrency cap — so the first campaign stampedes your database and drops work on the floor. Right model: rule → SQS → Lambda buys a durable buffer, batching, MaximumConcurrency, and a real DLQ. The queue is the shock absorber.

• “Global ordering, to be safe.” Reaching for a single FIFO group or one Kinesis shard so everything stays in order. The bite: you have serialised the entire system to one-message-at-a-time and killed throughput. Right model: need the least ordering you can get away with — order per entity (per order, per customer) via a high-cardinality group or partition key, and run parallel everywhere else.

• “Events should carry all the data so consumers don’t have to call back.” Tempting — until the event carries a full customer record with their address and card details. The bite: your event log and every DLQ is now a PII liability, and payloads blow past 256 KB. Right model: events carry references and facts, not payloads — an order ID and a tokenised handle; consumers fetch what they need, and big blobs go to S3 via claim-check.

• “Eventual consistency is fine everywhere.” Using async events to move money between two accounts in one step. The bite: for a beat the books do not balance, and a failure leaves them permanently unbalanced. Right model: a genuinely atomic operation is a transaction (TransactWriteItems) or a saga with explicit compensation — be honest about which requirements are actually strongly consistent and which tolerate lag.

Glossary

• Event — an immutable statement that something already happened (OrderCreated), broadcast to whoever cares. Past tense, no addressee.
• Command — a request to do something (PlaceOrder), addressed to one handler, expecting an action.
• Producer / consumer — the service that emits a message; the service that processes it. Also called publisher/subscriber.
• Broker — the middleware that moves messages between producers and consumers (SQS, SNS, EventBridge, Kinesis).
• Event bus — EventBridge’s router: producers call PutEvents, rules match on content and forward to targets.
• Rule / event pattern — a JSON filter on the event payload that decides which targets a matching event reaches.
• Queue — a durable buffer (SQS) that holds messages until a consumer pulls them; a consumed message is deleted.
• Topic — a pub/sub channel (SNS) that pushes each message to all subscribers at once.
• Stream — an ordered, replayable log (Kinesis, DynamoDB Streams) where consumers track their own read position (offset).
• Fan-out — one event delivered to many independent consumers, each on its own queue or subscription.
• Point-to-point — one producer, one queue, one consumer pool.
• Event source mapping (ESM) — the Lambda-managed poller that reads from SQS/Kinesis/DynamoDB Streams/MSK, batches records, and invokes your function.
• Batch / batching window — records grouped into one invoke; the window is how long the poller waits to fill a batch, trading a little latency for fewer, fuller invokes.
• Partial batch response — returning only the failed message IDs (batchItemFailures) so successful records in a batch are not reprocessed.
• At-least-once delivery — the guarantee that a message arrives, possibly more than once — which is why idempotency is mandatory.
• Idempotency — processing the same message twice yields the same result as once; enforced by conditional writes and an idempotency store.
• Dead-letter queue (DLQ) — where a message goes after exhausting retries, so the pipeline keeps flowing and you can inspect or redrive it.
• Redrive — moving messages from a DLQ back to the source queue to reprocess them after a fix.
• Poison message — a message that will never succeed on retry; the DLQ ceiling contains it.
• FIFO — first-in-first-out queues/topics: ordered per message group ID, with built-in deduplication.
• Message group ID / partition key — the field that decides ordering scope: messages sharing it stay ordered (and, for streams, land on the same shard).
• Transactional outbox — publishing an event only if the database commit succeeded; here, DynamoDB Streams is the outbox.
• EventBridge Pipes — managed source → filter → enrich → target plumbing that replaces glue Lambdas.
• Claim-check — storing a large payload in S3 and putting only a pointer on the bus to stay under the 256 KB limit.
• Choreography — services reacting to events with no central coordinator.
• Orchestration (saga) — a Step Functions state machine owning a multi-step transaction, with retries and compensation.
• Compensating transaction — the undo step (refund, release stock) run when a later saga step fails.
• CQRS — separating the write model from purpose-built read models kept eventually consistent by the event stream.
• Correlation / causation ID — IDs propagated through every event to reconstruct one request’s path (and its causal tree) across services.
• Retry storm — retries amplifying load on a slow downstream until it fails; contained by concurrency caps, backoff with jitter, and DLQs.
• Backpressure — letting a queue absorb a spike so consumers drain at a safe rate instead of stampeding the downstream.