Event-Driven Order Processing with the Saga Pattern on AWS

In a nutshell

Imagine booking a holiday through a travel agent: they reserve a flight, then a hotel, then a hire car. There is no single “undo” button across three different companies — so if the car company has nothing available, the agent phones the hotel and the airline and cancels what was already booked. Each cancellation is a deliberate business action, not a database rollback. You end the call with either a complete trip or no trip at all — never charged for a flight to a holiday you can’t actually take.

That is the saga pattern, and this lesson builds it on AWS for an online store. An order has to reserve inventory, charge a card, and book shipping — three separate systems with no shared “commit.” A saga runs those steps one at a time, and if a later step fails it runs a compensating transaction for each earlier step in reverse (refund the card, release the inventory). The result: a customer is never left charged for something the shop can’t ship. The “travel agent” here is AWS Step Functions — a workflow engine that drives the steps forward on success and drives the cancellations backward on failure, keeping a durable record of exactly where every order got to.

The alternative style, choreography, has no central agent at all: each service listens for events and reacts on its own (payment hears “order placed,” inventory hears “payment done”). That is looser and scales beautifully, but nobody can answer “what state is order 8842 in?” This lesson teaches both, shows why an orchestrated saga wins when you need auditability and clean rollback, and wires in idempotency, dead-letter queues, and tracing so the whole thing is boringly reliable under a flash-sale spike.

Level: Intermediate · Time: ~40 min

Before you start, you should be comfortable with: basic AWS Lambda (a function that runs on an event), Amazon DynamoDB as a key-value store, and the idea of an event (a small message saying “something happened”). If messaging primitives are new, skim the sibling lessons on EventBridge and SQS/SNS fan-out, FIFO and DLQs first. You do not need prior Step Functions experience — we build it up from zero.

After this lesson you will be able to:

• Explain, to a teammate, why you cannot wrap a BEGIN…COMMIT around a payment API, a DynamoDB table, and a courier’s REST endpoint — and what a saga does instead.
• Choose between orchestration (Step Functions) and choreography (EventBridge/SNS/SQS) for a given workflow, and justify the trade-off.
• Design a correct compensating transaction for every forward step, and order the steps so irreversible actions come last.
• Make every step idempotent so at-least-once delivery cannot double-charge a customer, using DynamoDB conditional writes and idempotency keys.
• Read a Step Functions definition — Task, Choice, Parallel, Map, Retry, Catch — and pick the right service-integration pattern (request/response, .sync, or .waitForTaskToken).
• Add the outbox pattern, SQS FIFO ordering, dead-letter queues, and X-Ray tracing so the saga is durable, ordered, and observable in production.

A fast-fashion retailer — the kind that drops a new collection every week and runs a flash sale the moment an influencer posts — has a checkout problem that is quietly costing it millions. At peak, an order touches four systems before it ships: the payment gateway authorizes the card, the inventory service reserves the last size-medium of a viral hoodie, a loyalty ledger burns points, and the shipping service books a courier slot. When all four succeed, life is good. When the third one fails — the courier API times out at 2pm on launch day — the company has already charged the customer and decremented stock for an order that will never ship. The support queue fills with “you took my money and the item is gone” tickets, the finance team reconciles refunds by hand for a week, and the brand takes the reputational hit. The ask from the VP of Engineering is blunt: “Make order processing never leave a customer charged for something we can’t fulfill, and make it survive a 50× traffic spike.” This article is the reference architecture for building that on AWS, using the saga pattern to guarantee that a partial failure unwinds cleanly instead of stranding money and stock.

The pressures are the ones every commerce platform hits at scale. Consistency is the hard one: there is no distributed transaction across a third-party payment API, a DynamoDB inventory table, and a courier’s REST endpoint — you cannot wrap a BEGIN…COMMIT around the open internet. Spikiness means the system sits near-idle between drops and then takes 50× load in ninety seconds, which kills anything you have to pre-provision. Latency means a shopper watching a spinner will abandon the cart in seconds. And auditability means finance needs to prove, for every order, exactly which steps ran and which were compensated. The saga pattern answers the consistency problem head-on: instead of one atomic transaction, you model the order as a sequence of local transactions, each with a defined compensating action that semantically undoes it, and an orchestrator that drives the sequence forward — or, on failure, drives the compensations backward.

Why not the obvious approaches

The naive designs each fail in a way someone on the team will have lived through, so naming them matters.

A single synchronous request that calls payment, then inventory, then shipping in-line is the version most teams start with. It works in the demo and dies in production: if shipping throws after payment succeeded, you are now writing rollback logic by hand inside a catch block, across network calls that can themselves fail mid-rollback, with no durable record of where you got to. One Lambda timeout and the order is in an unknown state forever.

A two-phase commit (2PC) is the textbook “correct” answer and the wrong one here. 2PC needs every participant to support a prepare/commit protocol and hold locks until the coordinator decides — but a third-party payment gateway and a courier API expose no such protocol, and holding an inventory lock across a slow external call destroys throughput on launch day. Distributed locks plus external services plus a spike is a deadlock waiting to happen.

A pile of choreographed events with no orchestrator — payment publishes an event, inventory reacts, shipping reacts — scales beautifully and becomes unobservable. With compensations in the mix, the failure logic is smeared across five services and no single place tells you what state an order is in or why it rolled back. Choreography is excellent for loose coupling and miserable for a workflow finance has to audit.

The orchestrated saga threads the needle. A central state machine owns the sequence, calls each service, and on any failure walks the already-completed steps in reverse, invoking each one’s compensating transaction — refund the payment, release the inventory reservation, restore the loyalty points. The orchestration is durable, every transition is logged, and “what happened to order 8842?” has a single, queryable answer.

Architecture overview

The system has two halves that share infrastructure but run on different clocks: a synchronous intake path that accepts an order and returns fast, and the asynchronous saga execution that does the real work of coordinating payment, inventory, and shipping. Holding those apart is the first step to operating this well — the shopper should never wait on a courier API.

The defining property of the topology is that the saga is durable and explicit, not implicit in a chain of in-flight HTTP calls. A dedicated state machine holds the source of truth for “where is this order,” each step is an idempotent local transaction, and every step has a named compensator. Nothing about the happy path or the rollback lives in a Lambda’s ephemeral memory.

Intake path, following the control flow:

1. A shopper checks out from the storefront. Traffic hits Akamai at the edge for TLS termination, global anycast, and WAF/bot mitigation — critical when a flash sale draws bot-driven sneaker-style buying — before it reaches AWS. Customer identity is federated through Okta as the consumer IdP (with the internal ops console federating staff identity through Microsoft Entra ID), so the order request carries a verified identity claim.
2. The request lands on Amazon API Gateway, which validates the JWT via a Lambda/JWT authorizer, applies per-client rate limiting and usage plans, and forwards a clean CreateOrder command.
3. An intake Lambda does the minimum synchronous work: validate the cart, write an OrderRecord to DynamoDB in PENDING state with a generated orderId and an idempotency key derived from the client request, and publish an OrderSubmitted event to Amazon EventBridge. It returns 202 Accepted with the orderId immediately — the shopper sees “order received” in well under a second, while the heavy lifting happens behind the event.
4. An EventBridge rule matches OrderSubmitted and starts an execution of the AWS Step Functions saga state machine, passing the order payload. EventBridge is the seam that decouples intake from execution, lets other consumers (analytics, fraud, the data lake via Firehose) subscribe to the same event, and absorbs bursts.

Saga execution path, driven entirely by Step Functions as the orchestrator:

1. The state machine runs the forward sequence as a series of task states, each invoking a single-purpose Lambda that performs one local transaction against its service: AuthorizePayment (charge the card via the payment gateway), ReserveInventory (a conditional DynamoDB update that decrements stock only if available), BurnLoyaltyPoints (debit the loyalty ledger), and BookShipment (reserve a courier slot). Each task writes its outcome back to the order’s DynamoDB record so the item history is complete.
2. Between steps, the state machine evaluates the result. On success it advances; on a caught error it transitions into the compensation branch — a reverse sequence of Catch handlers that invoke the compensators for exactly the steps that completed: RefundPayment, ReleaseInventory, RestoreLoyaltyPoints. The machine only compensates what actually ran, which is why each forward task records its completion.
3. On full success the machine writes the order to CONFIRMED and emits an OrderConfirmed event back to EventBridge (fulfillment, email, the customer’s order history all react). On a compensated failure it writes CANCELLED_COMPENSATED and emits OrderFailed, so the shopper is told honestly and the card was never left charged.
4. Any Lambda invocation that exhausts its retries is routed to a per-step Amazon SQS dead-letter queue (DLQ). The order parks in a NEEDS_REVIEW state rather than silently failing, a CloudWatch alarm fires, and an operator picks it up — manual remediation is a designed outcome, not an accident.

Component breakdown

A few choices deserve the why, because they are the ones teams get wrong.

Why Step Functions as the orchestrator, not application code. You could write the saga loop in a Lambda and store progress in DynamoDB yourself. Teams that do end up re-implementing retries, timeouts, error catching, and an execution history — badly — and then cannot answer “show me the exact path order 8842 took.” Step Functions gives you durable state, declarative per-state Retry/Catch, visual execution history for every order, and built-in compensation flow. The orchestration logic becomes a reviewable artifact in version control instead of branching buried in a function.

Why every step must be idempotent. Retries are not optional in a distributed system — EventBridge delivers at-least-once, Lambda retries, and Step Functions retries — so any step can run twice. AuthorizePayment keys on the order’s idempotency token and asks the gateway for an idempotent charge so a double-invoke does not double-charge. ReserveInventory uses a DynamoDB conditional write so re-running it does not decrement stock twice. Idempotency is what makes at-least-once delivery safe; without it, the retries that give you reliability also give you double charges.

Why a DLQ per step, not one global one. When a worker exhausts retries, you want to know which step failed and to remediate it with the right context — a payment failure and a shipping failure are different operational tickets. Per-step DLQs preserve that, drive a specific alarm, and let an operator redrive just the affected messages once the downstream service recovers, rather than replaying an undifferentiated pile.

The saga, concretely

The whole design lives or dies on getting the compensations right. Every forward step needs a semantic inverse — not a literal undo, because you cannot un-send an email, but a business action that neutralizes it.

The order of compensation matters: you unwind in reverse, releasing inventory and refunding payment so the customer is made whole first. The Step Functions definition expresses this as a Catch on each task that routes to a compensation state which itself chains the relevant compensators. A trimmed Amazon States Language snippet shows the shape — note the per-state retry-then-catch:

"ReserveInventory": {
  "Type": "Task",
  "Resource": "arn:aws:lambda:...:function:ReserveInventory",
  "Retry": [
    { "ErrorEquals": ["States.TaskFailed"], "IntervalSeconds": 2,
      "MaxAttempts": 3, "BackoffRate": 2.0 }
  ],
  "Catch": [
    { "ErrorEquals": ["InventoryUnavailable"], "Next": "RefundPayment" },
    { "ErrorEquals": ["States.ALL"], "Next": "RefundPayment" }
  ],
  "Next": "BurnLoyaltyPoints"
}

RefundPayment runs because payment already succeeded; inventory never decremented, so it needs no release. That selective unwind is the entire point — and it is why each forward task stamps its completion onto the DynamoDB order record before the next begins.

Orchestration versus choreography: two ways to run a saga

The ## Why not the obvious approaches section already ruled out one synchronous call and 2PC. But there are genuinely two valid ways to build a saga, and choosing between them is the most consequential design decision you will make here. Both coordinate the same local transactions; they differ only in who holds the plot.

Orchestration puts a single coordinator in charge. One component (here, a Step Functions state machine) knows the whole recipe: call payment, then inventory, then shipping, and on failure call the compensators in reverse. The workers are dumb and single-purpose — ReserveInventory knows how to reserve inventory and nothing about what comes before or after it. The coordinator is the brain; the workers are the hands.

Choreography has no coordinator. Each service subscribes to events, does its bit, and emits its own event announcing what it did. Nobody owns the sequence — it emerges from the chain of reactions. The same order flow, done as choreography, looks like this:

1. Intake publishes OrderSubmitted to an EventBridge bus.
2. The payment service has a rule matching OrderSubmitted; it charges the card and emits PaymentAuthorized (or PaymentFailed).
3. The inventory service has a rule matching PaymentAuthorized; it reserves stock and emits InventoryReserved (or InventoryUnavailable).
4. The shipping service matches InventoryReserved, books the courier, and emits ShipmentBooked.
5. Compensation is also choreographed: the inventory service subscribes to ShipmentFailed and, on hearing it, releases the reservation and emits InventoryReleased; the payment service subscribes to both InventoryUnavailable and InventoryReleased and issues a refund.

Notice what happened to the failure logic: it is now smeared across five services. To answer “why did order 8842 roll back?” you have to reconstruct the story from events scattered across five CloudWatch log groups. There is no single place that says “this order is currently compensating.” That is the defining weakness of choreography for a workflow finance must audit — and exactly why this architecture chose orchestration.

Here is the honest, side-by-side trade-off. Neither column is “better”; they optimize for different things.

A useful rule of thumb: orchestrate within a bounded context, choreograph between them. Inside the order-fulfillment boundary — where payment, inventory, and shipping must succeed or unwind together — orchestration’s central control and clean compensation earn their keep. But the edges of this system are already choreographed, and correctly so: intake emits OrderSubmitted to EventBridge, and analytics, fraud scoring, the data lake, and email all subscribe independently without the saga knowing they exist — choreography doing what it does best, fan-out to loosely coupled consumers. The two styles are not rivals; a mature platform uses orchestration for the transactional core and choreography for the notification fabric around it (the event-bus mechanics behind that fabric — buses, rules, schema registry, Pipes — are the subject of the EventBridge lesson linked in the prerequisites).

One subtlety beginners miss: choreography can still implement a saga. People sometimes say “we use choreography, so we don’t do sagas” — that is a category error. A saga is what you are doing (a sequence of local transactions with compensations); orchestration and choreography are how you coordinate it. You can compensate correctly in either style. Orchestration just makes the compensation logic something you can point at, review, and watch execute.

Step Functions as the orchestrator, in depth

The component table lists Step Functions as the orchestrator with “Standard workflow; Catch/Retry per state.” A beginner needs the mental model under that one line, because the state machine is where the saga actually lives.

A Step Functions state machine is a JSON document written in Amazon States Language (ASL). It is a directed graph of states; each state does one thing and names the next. The state types you will use in a saga:

Two of these deserve a closer look because they change how the saga scales.

Map — iterating over line items. A real basket has many products, each needing its own reservation; Map runs the reservation sub-flow once per item. It has two flavours, and the difference is a production trap. Inline Map (the default) keeps every iteration in the parent execution’s history and caps concurrency at 40 — fine for a basket of a dozen items. Distributed Map spins each iteration into its own child execution, scales to thousands of parallel iterations, and can read its input array straight from S3 — but does not fold every child event into the parent history. Reach for Distributed Map only when the fan-out is large; for a shopping basket, Inline Map is right.

Parallel and the compensation fan-in. Parallel runs branches concurrently, but if any branch fails the whole state fails, and your Catch must then compensate every branch that had already succeeded. Parallelism buys latency but complicates rollback — use it where steps are genuinely independent (charging the card while validating the shipping address) and keep the strictly-ordered money/stock steps sequential.

Retry and Catch, the two-line safety net. The original snippet showed a ReserveInventory state with a Retry then a Catch. Read that as: “try me again on a transient blip; if I still fail, jump to compensation.” The evaluation order is fixed and worth memorizing: a state runs, and on error Step Functions first walks the Retry array (retrying with backoff until MaxAttempts is spent), and only then walks the Catch array to route the still-failing error somewhere. A few fields carry the weight:

• ErrorEquals matches error names. Use your own domain errors (InventoryUnavailable) for expected business failures you do not want to retry, and the built-ins (States.TaskFailed, States.Timeout, States.ALL) for infrastructure blips you do.
• BackoffRate and MaxDelaySeconds shape exponential backoff; add JitterStrategy: "FULL" so a thundering herd of retries after a downstream recovers doesn’t stampede it again.
• Catch.ResultPath decides where the error object is placed in the state data, so the compensation branch can read why it is compensating.

A critical distinction the beginner must internalize: retry is forward recovery; compensation is backward recovery. You Retry a step you believe will succeed if tried again (a 500 from a healthy service under load). You Catch into compensation when the step has definitively failed (the card was declined; the item is genuinely out of stock). Retrying a hard business failure just wastes time before the inevitable rollback — which is why InventoryUnavailable is caught, not retried.

Here is a fuller, still-trimmed skeleton showing the shape a real saga takes — a Choice gate, the forward chain, and a Catch that routes to a compensation state. ARNs are placeholders.

{
  "Comment": "Order saga (trimmed)",
  "StartAt": "FraudGate",
  "States": {
    "FraudGate": {
      "Type": "Choice",
      "Choices": [
        { "Variable": "$.fraudScore", "NumericGreaterThan": 90, "Next": "ManualReview" }
      ],
      "Default": "AuthorizePayment"
    },
    "AuthorizePayment": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": { "FunctionName": "AuthorizePayment", "Payload.$": "$" },
      "Retry": [
        { "ErrorEquals": ["States.TaskFailed", "States.Timeout"],
          "IntervalSeconds": 2, "MaxAttempts": 3, "BackoffRate": 2.0,
          "JitterStrategy": "FULL" }
      ],
      "Catch": [
        { "ErrorEquals": ["CardDeclined"], "Next": "FailOrder", "ResultPath": "$.error" },
        { "ErrorEquals": ["States.ALL"], "Next": "FailOrder", "ResultPath": "$.error" }
      ],
      "Next": "ReserveInventory"
    },
    "ReserveInventory": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": { "FunctionName": "ReserveInventory", "Payload.$": "$" },
      "TimeoutSeconds": 10,
      "Catch": [
        { "ErrorEquals": ["States.ALL"], "Next": "RefundPayment", "ResultPath": "$.error" }
      ],
      "Next": "BookShipment"
    },
    "RefundPayment": {
      "Type": "Task",
      "Resource": "arn:aws:states:::lambda:invoke",
      "Parameters": { "FunctionName": "RefundPayment", "Payload.$": "$" },
      "Next": "FailOrder"
    },
    "FailOrder": { "Type": "Fail", "Error": "SagaCompensated" }
  }
}

Standard versus Express — the choice that sets your bill and your audit trail. Step Functions offers two workflow types, and picking wrong is a common and expensive mistake.

The order saga runs as Standard because finance wants the visual execution history for every order and the flow may pause on an external payment webhook (well beyond five minutes if the gateway is slow). The honest cost note from the enterprise section stands: Standard costs more per transition but is the audit trail. Reserve Express for genuinely high-volume, short, replay-safe sub-flows. And note the semantics: Express is at-least-once, so a workflow body can run twice — one more reason every step is idempotent.

Service-integration patterns — how a Task actually calls the world. This is the piece most tutorials skip, and it is what makes Step Functions more than a fancy for loop. A Task can integrate with a downstream in three ways, chosen by a suffix on the resource ARN:

1. Request/Response (default) — call the service, take the immediate response, move on. This is how a plain lambda:invoke works. Good for fast, synchronous steps.
2. Run-a-Job — .sync — call a long-running service and block the state until the job finishes, e.g. ecs:runTask.sync or glue:startJobRun.sync. Step Functions handles the polling for you and advances only when the job reports done. Use it when a step launches a container or batch job that takes minutes.
3. Wait-for-Callback — .waitForTaskToken — the powerful one for sagas. Step Functions generates a task token, hands it to your worker, and pauses the execution — for up to a year — until something calls SendTaskSuccess/SendTaskFailure with that token. This is exactly how you model an asynchronous payment gateway: AuthorizePayment submits the charge and returns the token; when the gateway’s webhook later fires, a small Lambda validates it and calls SendTaskSuccess(token), and only then does the saga proceed. No polling, no burning a Lambda for minutes — the execution sits durably parked; pair it with HeartbeatSeconds so a webhook that never arrives times out into compensation.

That callback pattern is why “the saga waited three minutes for the bank” costs you nothing: the execution is suspended state, not a running function. The deeper mechanics of .sync versus .waitForTaskToken and the full error taxonomy are covered in the sibling lesson on Step Functions distributed orchestration and error handling.

Idempotency, exactly-once delivery, and the outbox pattern

The component notes already state that “every step must be idempotent.” This section makes that concrete, because it is the single hardest thing to get right and the thing that turns a demo into a system that survives production.

At-least-once is the default, and it composes downward. EventBridge and SQS standard queues both deliver at-least-once; Lambda retries failed async and stream invocations; Step Functions retries tasks. Every one of those hops can deliver the same message twice. The senior insight most beginners resist: “exactly-once” at any single hop does not add up to end-to-end exactly-once. Even Standard Step Functions’ exactly-once workflow execution does not stop the Lambda it calls from running twice under a Retry. So don’t chase end-to-end exactly-once by stacking exactly-once hops — instead assume at-least-once everywhere and make every step idempotent, so running it twice is indistinguishable from running it once. Idempotency is what makes at-least-once safe — the load-bearing property of the whole design.

What “idempotent” means, per step. An operation is idempotent if applying it twice has the same effect as applying it once. Concretely for our steps:

• AuthorizePayment attaches the order’s idempotency key to the charge request. Every serious payment API accepts an idempotency key and will return the original result — not a second charge — if it sees the same key again. Two invocations, one charge.
• ReserveInventory uses a DynamoDB conditional write that both decrements stock and records that this order already reserved it, in one atomic operation. Re-running it is a no-op because the condition fails the second time.
• RefundPayment and ReleaseInventory (the compensators) must be idempotent too — a rollback branch can also be retried. Releasing the same reservation twice must not credit stock twice.

Here is the inventory reservation as a real conditional write. It decrements stock only if there is stock and this order hasn’t already claimed it, using a reservations set on the item. Attribute values are illustrative.

# ReserveInventory worker (representative — not a live run)
import boto3
from botocore.exceptions import ClientError

ddb = boto3.client("dynamodb")

def reserve(sku: str, order_id: str, qty: int):
    try:
        ddb.update_item(
            TableName="inventory-prod",
            Key={"sku": {"S": sku}},
            UpdateExpression="SET available = available - :q "
                             "ADD reservedBy :o",
            ConditionExpression="available >= :q AND "
                                "(attribute_not_exists(reservedBy) OR NOT contains(reservedBy, :ov))",
            ExpressionAttributeValues={
                ":q":  {"N": str(qty)},
                ":o":  {"SS": [order_id]},
                ":ov": {"S": order_id},
            },
        )
        return "RESERVED"
    except ClientError as e:
        if e.response["Error"]["Code"] == "ConditionalCheckFailedException":
            # Either out of stock, or THIS order already reserved (a retry).
            # Idempotent: treat an existing reservation by this order as success.
            return "ALREADY_RESERVED_OR_UNAVAILABLE"
        raise

The ConditionalCheckFailedException doing double duty — “out of stock” and “already reserved by me” — is the crux. The worker distinguishes them with a follow-up read and either reports success (idempotent retry) or raises InventoryUnavailable (real business failure, caught into compensation). One conditional write gives you both stock safety and idempotency.

The dual-write problem, and the outbox pattern that fixes it. Now a subtler bug. The intake Lambda does two things: it writes the OrderRecord to DynamoDB and it publishes OrderSubmitted to EventBridge. Those are two separate systems with no shared transaction. What if the DynamoDB write succeeds and then the process crashes before the EventBridge publish? You now have an order in the database that no saga will ever pick up — a silently stuck order. Flip it — publish first, then crash before the write — and a saga starts for an order that doesn’t exist. This is the dual-write problem, and it bites every event-driven system.

The fix is the outbox pattern. Instead of writing the record and then publishing an event, you write the record and an “outbox” event in a single atomic DynamoDB transaction, then let a separate relay publish the event afterwards:

# Intake: atomic write of order + outbox event (representative)
ddb.transact_write_items(TransactItems=[
    {"Put": {"TableName": "orders-prod",
             "Item": {"orderId": {"S": order_id}, "status": {"S": "PENDING"}}}},
    {"Put": {"TableName": "orders-prod",
             "Item": {"orderId": {"S": f"OUTBOX#{order_id}"},
                      "eventType": {"S": "OrderSubmitted"},
                      "published": {"BOOL": False}}}},
])

Both items commit or neither does — DynamoDB TransactWriteItems is all-or-nothing. Then DynamoDB Streams carries every new outbox item to a tiny relay Lambda that publishes it to EventBridge and marks it published. Now the guarantee is airtight: the event is published if and only if the order was durably written, because they were the same transaction. The relay itself is at-least-once (a stream record can be re-delivered), so the event may be published twice — which is fine, because we already made the saga idempotent. The dual-write problem is converted into a duplicate-delivery problem, and we already solved that. (This is the same DynamoDB-Streams-as-CDC mechanism taught in the streams lesson — the outbox is one of its highest-value uses.)

Ordering: when sequence matters, and when it doesn’t. Within our saga, Step Functions enforces order — inventory strictly follows payment. But the notification fabric around it uses queues, and standard SQS and EventBridge do not guarantee order. Usually that’s fine (does it matter whether the analytics or the email event lands first? No). But some consumers need per-order ordering — a downstream that must process OrderConfirmed before OrderShipped for the same order. That is what SQS FIFO queues are for:

• A FIFO queue preserves order within a message group, identified by MessageGroupId. Set MessageGroupId = orderId, and every message for order 8842 is delivered strictly in order, one at a time — while different orders still flow in parallel. Ordering is per group, not global, which is exactly what you want: strict order where it matters, parallelism everywhere else.
• FIFO also gives exactly-once processing within a 5-minute deduplication window via MessageDeduplicationId (or content-based dedup).
• The cost is throughput: FIFO is capped far lower than standard queues (hundreds of messages/second per action, more with batching; high-throughput FIFO raises this per partition — verify current regional limits). Use FIFO only for the flows that truly need ordering, and keep the high-volume fan-out on standard queues.

Dead-letter queues — the designed off-ramp. When a worker exhausts its Retry and the error isn’t a clean business failure to compensate, the message must not vanish and must not loop forever. A dead-letter queue is the escape hatch: after maxReceiveCount failed processing attempts, SQS moves the message to a per-step DLQ, a CloudWatch alarm on ApproximateNumberOfMessages fires, the order parks in NEEDS_REVIEW, and an operator later redrives the DLQ back to the source queue once the downstream heals. As the enterprise section argued, a DLQ per step preserves the context (“which step, with what payload”) that a single global DLQ throws away. The mechanics of redrive policies, maxReceiveCount, and poison-message handling are covered in depth in the SQS/SNS FIFO, ordering, and DLQ lesson linked in the prerequisites.

Implementation guidance

Provision with Terraform; treat IAM and the state machine as the first deliverables. The state machine, the EventBridge rules, the DynamoDB tables, the SQS DLQs, and the per-Lambda least-privilege roles are all Terraform. Lay them down in dependency order — tables and queues first, worker functions and their scoped roles next, then the Step Functions definition that references them, then the EventBridge rule that starts it. A minimal DynamoDB shape communicates the intent — on-demand for spiky load, a stream for fulfillment, point-in-time recovery on:

resource "aws_dynamodb_table" "orders" {
  name             = "orders-prod"
  billing_mode     = "PAY_PER_REQUEST"          # absorbs 50x spikes, no pre-provisioning
  hash_key         = "orderId"
  stream_enabled   = true
  stream_view_type = "NEW_AND_OLD_IMAGES"        # fulfillment + audit react to changes
  attribute { name = "orderId"  type = "S" }
  point_in_time_recovery { enabled = true }
}

Pipeline and config. The infrastructure pipeline runs in GitHub Actions, authenticating to AWS via OIDC federation so there is no static access key to leak — a lesson the platform team intends never to repeat. Argo CD then handles GitOps deployment of the worker services (run on ECS/EKS where a long-lived container suits, or as Lambda where event-driven suits), continuously syncing the desired state from the repo. Terraform owns the saga and AWS primitives; Ansible handles host-level configuration and agent rollout on any EC2/ECS worker fleet. Wiz Code runs in the PR as an IaC scanner so a misconfigured S3 bucket, an over-broad IAM policy, or a public resource is caught before merge, with Wiz doing continuous CSPM on the live account as the backstop.

Identity and secrets: federate the humans, lease the keys. Customers authenticate through Okta; the internal operations console — where staff redrive DLQs and inspect stuck orders — federates through Microsoft Entra ID, and API Gateway’s authorizer validates those tokens so only authorized operators touch order state. The worker Lambdas assume scoped IAM roles via the AWS provider, but the secrets they cannot get from IAM — the payment gateway API key, the courier API credentials, webhook-signing secrets — come from HashiCorp Vault using the AWS IAM auth method with short-lived dynamic leases, so a payment credential is never baked into an environment variable or a Lambda layer.

Enterprise considerations

Failure modes, and what each one looks like. Name them before they page you.

• The double-charge on retry — EventBridge re-delivers or Lambda retries AuthorizePayment and the card is charged twice. Mitigation: idempotency keys end-to-end and an idempotent charge call to the gateway; the order’s idempotency token is the dedup anchor.
• The orphaned reservation — inventory decremented, then shipping fails, and without compensation the last hoodie is locked to a dead order. Mitigation: the ReleaseInventory compensator fires in the rollback branch; conditional writes make it safe to re-run.
• The poison message — a malformed event or a permanently failing downstream loops on retries. Mitigation: bounded Retry with backoff, then the per-step SQS DLQ parks it in NEEDS_REVIEW; an operator redrives once the dependency heals.
• The stuck saga — a downstream hangs and the execution waits indefinitely. Mitigation: per-task TimeoutSeconds and HeartbeatSeconds so a hung step fails into compensation rather than pinning an execution open.
• The non-compensatable step — a confirmation email or a physical pick-and-pack already went out. Mitigation: order compensatable steps first and irreversible steps last, and make truly irreversible actions a separate post-CONFIRMED stage.
• Region outage — see DR below.

Scalability. Each tier scales independently and natively, which is the entire reason for the event-driven shape. EventBridge and Step Functions are serverless and absorb bursts with no capacity to pre-provision; DynamoDB on-demand rides the 50× spike without a scaling lag; Lambda workers scale on concurrency — with reserved concurrency on the steps that hit fragile downstreams (the payment gateway, the courier API) so a spike does not stampede a partner into rate-limiting. The natural ceiling is the slowest external dependency, which is precisely why those calls sit behind retries, DLQs, and concurrency caps rather than in a synchronous request.

Cost optimization. The architecture is largely pay-per-use, which suits bursty commerce far better than a provisioned fleet sitting idle between drops.

The honest tradeoff: Standard workflows cost more per state transition but retain the full visual execution history finance wants for audit; Express is dramatically cheaper at flash-sale volume but keeps less history. Many teams run the saga as Standard for auditability and accept the cost, or split — Express for the hot path, Standard where a regulator might ask.

Security and Zero Trust. Every Lambda gets its own least-privilege IAM role — the payment worker cannot touch the inventory table, the inventory worker cannot call the payment gateway — so a compromised function has a tiny blast radius. Vault holds the third-party credentials with short leases; Akamai absorbs bot and DDoS pressure at the edge before it reaches API Gateway; Wiz runs continuous CSPM and Wiz Code gates the IaC so a public bucket or an over-broad policy never ships; CrowdStrike Falcon sensors on the container/EC2 worker fleet provide runtime threat detection feeding the SOC. A DLQ alarm or a security detection auto-raises a ServiceNow incident, so operations and security work a ticket, not a log line.

Observability. A saga is only as good as your ability to see it. Instrument Datadog APM so a single distributed trace spans the whole execution — intake → EventBridge → each saga step → compensation — with the order id as a tag, so “what happened to order 8842” is one search. Emit the metrics the business actually feels: saga success rate, compensation rate (a rising rate means a downstream is degrading), per-step latency and error rate, DLQ depth per step, and time-from-submit-to-confirmed. CloudWatch alarms on DLQ depth and on Step Functions ExecutionsFailed page on-call and open the ServiceNow incident; Datadog dashboards give the launch-day war room a live view of where orders are piling up.

Reliability and DR (RTO/RPO). Decide the numbers per tier. DynamoDB global tables give multi-region replication for order and inventory state with near-zero RPO and seconds RTO. The saga itself is regional, so DR means a warm standby of the Step Functions definition, EventBridge rules, and workers in a paired region (all Terraform, so it is the same code), with EventBridge archive and replay as the recovery lever — you can replay missed events into the standby region after a failover. A pragmatic target here: RTO 15 minutes, RPO near-zero for order state via global tables, with in-flight sagas at the moment of failover reconciled from the durable DynamoDB record and the EventBridge archive. Akamai health checks drive edge failover for ingress.

Governance. The Step Functions definition is the saga’s contract — keep it in version control, require review on every change (a change to compensation logic is a change to how customers get refunded), and gate edits through a ServiceNow change approval. Pin worker runtimes and dependency versions so behavior does not drift. Log every state transition and every compensation for finance’s audit trail, and retain it under the same data-retention policy that governs payment records.

Explicit tradeoffs

Accept these or do not build it. The saga buys you consistency without distributed locks, and the price is real complexity: you must design a correct compensating transaction for every forward step, reason about ordering so irreversible actions come last, and make every step idempotent because retries are guaranteed. The system is eventually consistent, not atomic — there is a window where payment has succeeded and shipping has not, and the UX must tell the shopper “order received, confirming…” rather than promising fulfillment up front. Debugging spans services and an execution history rather than a single stack trace. And the event-driven indirection that gives you elastic scale also means a flow you cannot step through in a debugger — you read it from the Step Functions history and the Datadog trace instead.

The alternatives, and when they win. If your “distributed transaction” is actually all inside one database, use a real ACID transaction — do not reach for a saga to solve a problem a single COMMIT solves. If your services are genuinely loosely coupled and you do not need a central audit of the workflow, pure EventBridge choreography is simpler and looser than an orchestrator — graduate to Step Functions when compensations and “what state is this order in” make orchestration worth it. If your steps are short and your volume is enormous, Express workflows trade execution history for cost. And if you need a true prepare/commit across systems that all support it (rare across third-party APIs), 2PC is the consistency-strict answer — at a throughput cost that a flash sale cannot pay.

The shape of the win

For the retailer, the payoff is not “a new workflow engine.” It is that on the next launch day, when the courier API times out at 2pm, order 8842 does not strand a charged customer next to a locked-up hoodie — the saga catches the failure, refunds the card, releases the inventory back to the next shopper, marks the order CANCELLED_COMPENSATED, and tells the customer honestly, all in seconds and all logged. Finance stops reconciling refunds by hand; support stops fielding “you took my money” tickets; and the platform rides a 50× spike on serverless primitives that cost almost nothing between drops. Everything upstream — the idempotent steps, the per-step DLQs, the Vault-held payment keys, the Wiz-gated IaC, the Datadog saga trace — exists to make that one outcome boringly reliable. The architecture here is the destination; start with a single happy-path flow if you must, but a commerce platform that must never leave a customer charged for something it cannot ship has to land here.

Going deeper

Everything above gets a correct saga running. This section is for the reader who will operate one at scale and needs the internals, the sharp edges, and the vocabulary that separates “it worked in the demo” from “it held up on launch day.”

The saga transaction taxonomy: compensatable, pivot, retriable. The single most useful advanced framing (from Chris Richardson’s saga work) is to classify every step into one of three kinds, because it tells you exactly how to order them:

• Compensatable transactions can be semantically undone. ReserveInventory and BurnLoyaltyPoints are compensatable — there is a ReleaseInventory, a RestoreLoyaltyPoints.
• The pivot transaction is the point of no return — the go/no-go commit. Once it succeeds, the saga is committed to running forward to completion. AuthorizePayment (specifically the capture) is a natural pivot: past it, you intend to fulfill.
• Retriable transactions come after the pivot and are guaranteed to eventually succeed — you retry them forward rather than compensating backward. BookShipment and “send confirmation email” are retriable: you never un-confirm an order because the courier API was briefly down; you retry until it books.

The design rule falls straight out of the taxonomy: order compensatable steps before the pivot, and retriable steps after it. Put every “might need undoing” step ahead of the money capture, and every “must eventually happen, cannot be undone” step behind it. This is the rigorous version of the lesson’s earlier advice to “order irreversible actions last.” Get the ordering wrong — a non-compensatable email before the pivot — and you have designed a saga that can strand a side-effect it cannot take back.

Sagas have no isolation — the missing “I” in ACID. A local ACID transaction gives you Atomicity, Consistency, Isolation, Durability. A saga is a sequence of separate transactions, so it keeps A, C, and D but loses I: other operations can see the intermediate state while a saga is mid-flight — between ReserveInventory and the final CONFIRMED, another request can read a “half done” order. That opens the classic anomalies — lost updates, dirty reads, non-repeatable reads — and beginners are genuinely surprised the first time a report double-counts a compensating order. The countermeasures (again Richardson’s) worth knowing by name: semantic lock — an in-progress flag (status = PENDING) that tells others the record isn’t settled (our PENDING/NEEDS_REVIEW states are semantic locks); commutative updates — design so order doesn’t matter (an atomic ADD commutes, a blind SET doesn’t; available - :q lands correctly either way); reread value — check a version before writing (optimistic concurrency) so you don’t clobber a concurrent change; and by value — route high-value orders through a stricter path. If you remember one thing: status is not decoration, it is your isolation mechanism — every consumer must respect it.

Payload and history limits — Claim Check and nested machines. Step Functions caps the state input/output payload at 256 KB, and a Standard execution’s history at 25,000 events (every state entry, exit, retry, and scheduling counts) — a maxed-out history fails the execution. A fat basket blows the payload cap; a large Inline Map or aggressive retries approach the event cap. Two patterns fix both: the Claim Check — write the big payload to S3 or the DynamoDB order record and pass only a pointer (orderId / S3 key) through the states, each worker fetching what it needs — which also keeps the history cheap and applies equally to the 256 KB EventBridge PutEvents limit; and Distributed Map plus nested state machines (startExecution.sync), where child executions each get their own history budget. This is precisely why Distributed Map exists.

IAM and blast radius — the least-privilege payoff. The advanced point behind “each Lambda gets its own role” is where the boundary is drawn: the payment worker can call the gateway and write only the orders item’s payment fields — it has no dynamodb:UpdateItem on the inventory table and cannot states:SendTaskSuccess a token that isn’t its own. The Step Functions execution role, separately, needs lambda:InvokeFunction scoped to exactly the worker ARNs — not *. A common real-world leak is granting that role broad lambda:InvokeFunction “to move fast,” quietly handing every future function in the account to the orchestrator. Scope it to the ARNs, and a compromised worker forges one kind of damage, not the whole saga.

Observability: reading a saga you cannot step through in a debugger. Because the flow is durable state and events, you read it from three surfaces, and should wire all three:

• Step Functions execution history — the console shows a visual graph of the exact path order 8842 took, every input/output, retry, and where it entered compensation; GetExecutionHistory exposes the same to automation.
• AWS X-Ray stitches the distributed trace across boundaries the state machine can’t see alone: enable tracing on the state machine and active tracing on the workers, and one trace spans intake → EventBridge → each task → compensation, with orderId as an annotation. The catch beginners hit — trace context does not cross a queue by magic; for SQS you must pass the AWSTraceHeader system attribute or the trace breaks there. (Service-map and segment mechanics: the X-Ray tracing lesson.)
• Business metrics — saga success rate, compensation rate (the leading indicator: a rising rate means a downstream is degrading before it fully fails), per-step latency, DLQ depth per step, submit-to-confirm time. Alarm on compensation rate, not just hard failures — it pages you an hour earlier.

Scale and cost, quantified thinking. Each tier scales independently, but the ceiling is always the slowest external dependency — the payment gateway and courier API. Protect them with reserved concurrency on those specific workers so a 50× spike queues inside AWS (where SQS and Step Functions absorb it for free) rather than stampeding a partner into rate-limiting. On cost, the sharpest lever is Standard-versus-Express per sub-flow: keep the auditable order saga on Standard, but run any later high-volume, sub-5-minute, replay-safe flow (per-item price re-checks, say) as Express to cut its per-execution cost by an order of magnitude. Also sample X-Ray at flash-sale volume — you rarely need 100% of traces — and lean on Express or Claim Check to keep Standard histories small.

Version and API caveats to check before you ship. Limits and defaults drift; confirm the current numbers for your region rather than trusting a blog (this one included): the Step Functions payload size and history-event cap, DynamoDB TransactWriteItems item count, SQS FIFO throughput (standard vs high-throughput mode), and Lambda async retry/destination behavior. Two modern defaults worth designing around: DynamoDB on-demand now absorbs sharp spikes far more gracefully than early on, and Step Functions JitterStrategy/MaxDelaySeconds let you tame retry storms declaratively instead of hand-rolling backoff in Lambda.

Practice challenges

Work these in order — they climb from “name the concept” to “redesign the flow.” Try each before opening the solution. All ARNs/IDs are placeholders; nothing here needs a live AWS account to reason through.

1. Name the compensator (beginner). The product team adds a new forward step, ApplyStoreCredit — it debits a gift-card balance to cover part of the order. Where in the sequence should it go, and what is its compensating transaction?

<details> <summary>Solution</summary>

ApplyStoreCredit is compensatable, so it belongs before the payment pivot, alongside ReserveInventory and BurnLoyaltyPoints. Its compensator is RefundStoreCredit — credit the same amount back to the gift-card balance, keyed on the order id so it is idempotent and leaves an audit trail.

Why: a compensatable step must sit ahead of the point of no return so the rollback branch can still undo it; every forward step needs a named semantic inverse. </details>

2. Order the steps for safe rollback (beginner). You have four steps: SendConfirmationEmail, ReserveInventory, AuthorizePayment (capture), BurnLoyaltyPoints. Put them in a saga-safe order and say which one must be last.

<details> <summary>Solution</summary>

ReserveInventory → BurnLoyaltyPoints → AuthorizePayment (pivot) → SendConfirmationEmail (last). The two compensatable steps come first; the payment capture is the pivot; the non-compensatable email is retriable and must come after the pivot, dead last.

Why: you cannot un-send an email, so an irreversible action must never run while an earlier step could still force a rollback — order irreversible/retriable steps after the pivot. </details>

3. Make a compensator idempotent (intermediate). ReleaseInventory runs in the rollback branch and might be retried. Write a DynamoDB UpdateItem condition so releasing the same order’s reservation twice does not credit stock twice.

<details> <summary>Solution</summary>

ddb.update_item(
    TableName="inventory-prod",
    Key={"sku": {"S": sku}},
    UpdateExpression="SET available = available + :q DELETE reservedBy :o",
    ConditionExpression="contains(reservedBy, :ov)",   # only if THIS order still holds a reservation
    ExpressionAttributeValues={
        ":q":  {"N": str(qty)},
        ":o":  {"SS": [order_id]},
        ":ov": {"S": order_id},
    },
)
# ConditionalCheckFailedException here = already released = treat as success (idempotent).

Why: gating the increment on “this order is still in the reservedBy set” means the second release finds nothing to release, fails the condition, and is safely swallowed — a retry can’t double-credit stock. </details>

4. Retry the blip, catch the business failure (intermediate). For AuthorizePayment, a CardDeclined error and a States.Timeout error need opposite treatment. Write the Retry/Catch so one is retried and one goes straight to compensation.

<details> <summary>Solution</summary>

"Retry": [
  { "ErrorEquals": ["States.Timeout", "States.TaskFailed"],
    "IntervalSeconds": 2, "MaxAttempts": 3, "BackoffRate": 2.0, "JitterStrategy": "FULL" }
],
"Catch": [
  { "ErrorEquals": ["CardDeclined"], "Next": "FailOrder" },
  { "ErrorEquals": ["States.ALL"],   "Next": "RefundPayment" }
]

States.Timeout is a transient blip → retried with backoff. CardDeclined is a definitive business failure → caught immediately to FailOrder (no charge happened, so no refund needed), before the generic States.ALL catch.

Why: retry is forward recovery for “will succeed if tried again”; compensation is backward recovery for “has definitively failed” — retrying a hard decline just wastes time before the inevitable. </details>

5. Pick the integration pattern (advanced). For three steps, choose request/response, .sync, or .waitForTaskToken: (a) ReserveInventory, a fast Lambda; (b) AuthorizePayment against a gateway that charges asynchronously and confirms via webhook minutes later; © RenderInvoicePdf, an ECS task that runs 90 seconds.

<details> <summary>Solution</summary>

(a) Request/Response (lambda:invoke) — it returns in milliseconds; just take the result and advance. (b) .waitForTaskToken — submit the charge, return the task token, and let the webhook’s Lambda call SendTaskSuccess(token) when the bank confirms; the execution sits parked (free) for minutes with a HeartbeatSeconds guard. © .sync (ecs:runTask.sync) — launch the container job and block the state until ECS reports it finished.

Why: match the pattern to the step’s duration and callback shape — fast synchronous work is request/response, external async confirmations are task-token, and long-running managed jobs are .sync. </details>

6. Kill the dual-write bug (advanced). Intake currently writes the OrderRecord to DynamoDB and then calls EventBridge PutEvents. Under load, some orders are written but their OrderSubmitted event is never published, so no saga starts. Redesign intake so an event is published if and only if the order was durably written — and explain what carries the event.

<details> <summary>Solution</summary>

Apply the outbox pattern. In one TransactWriteItems, atomically write both the order item and an OUTBOX#<orderId> item; both commit or neither does. Then a DynamoDB Streams-triggered relay Lambda reads each new outbox item and publishes it to EventBridge, marking it published.

ddb.transact_write_items(TransactItems=[
  {"Put": {"TableName": "orders-prod",
           "Item": {"orderId": {"S": oid}, "status": {"S": "PENDING"}}}},
  {"Put": {"TableName": "orders-prod",
           "Item": {"orderId": {"S": f"OUTBOX#{oid}"}, "eventType": {"S": "OrderSubmitted"}}}},
])
# DynamoDB Streams -> relay Lambda -> EventBridge PutEvents -> mark published

Why: a single atomic write removes the “write succeeded but publish failed” gap; the stream relay is at-least-once, but the already-idempotent saga tolerates the occasional duplicate event — the dual-write problem becomes a solved duplicate-delivery problem. </details>

Common beginner mistakes

These are misconceptions, not symptom-and-fix bugs — the wrong mental model that leads a beginner to build the saga incorrectly, and the right model to replace it with. (For runtime failure modes and their mitigations, see the ## Enterprise considerations “Failure modes” list above.)

“A saga is a distributed transaction, so it must be atomic.” It is not, and expecting atomicity is the root of most saga bugs. A saga trades atomicity for eventual consistency: there is a real window where payment has succeeded but shipping has not. The correct model is “a sequence of local transactions, each individually atomic, stitched together with compensations” — and the UI must reflect that with “order received, confirming…” rather than promising fulfillment up front. If you truly need atomicity and everything lives in one database, use a real ACID transaction, not a saga.

“Compensation means rolling back the database.” Compensation is a semantic inverse, not a ROLLBACK. The original transaction already committed — the card was charged, the email was sent. You cannot roll those back; you issue a new business action that neutralizes them (refund, cancellation notice). Beginners who think “rollback” try to undo committed external effects and are baffled when the payment gateway has no rewind button. Design the counter-action, and accept that some effects (a sent email) can only be neutralized, not erased.

“Exactly-once delivery will save me from double-charges.” No single AWS hop gives you end-to-end exactly-once, and even the ones advertised as exactly-once (SQS FIFO within a window, Standard Step Functions workflow execution) do not stop the Lambda underneath from running twice under a retry. Chasing exactly-once is chasing a mirage. The right model is assume at-least-once everywhere and make every step idempotent — an idempotency key on the charge, a conditional write on the reservation. Idempotency, not delivery guarantees, is what prevents the double-charge.

“I’ll just compensate everything in the rollback branch.” Compensating a step that never ran is its own bug — you refund a payment that was never captured, or release inventory that was never reserved, and now your ledger is wrong in the other direction. The right model is compensate only what actually completed, which is why each forward task stamps its completion onto the DynamoDB record and the Catch branches invoke exactly the compensators for the steps that ran. Selective unwind is the whole point.

“Choreography and orchestration are rival architectures; pick one for everything.” They are tools for different jobs, and mature systems use both. Orchestrate the transactional core where you need central control, clean compensation, and a queryable “what state is this order in?”; choreograph the loosely coupled notification fabric (analytics, email, the data lake) around it. Forcing pure choreography onto an auditable, compensating workflow gives you unobservable “event spaghetti”; forcing orchestration onto every fan-out gives you a bloated state machine that couples to consumers it shouldn’t know about.

“The state machine’s payload can carry the whole order.” Step Functions caps state input/output at 256 KB, and stuffing a fat basket, product metadata, and a fraud dossier into the execution data will eventually fail an execution and bloat the history. The right model is the Claim Check pattern: persist the big object in DynamoDB or S3 and pass only the orderId/key through the states, letting each worker fetch what it needs.

Glossary

Saga — A way to keep data consistent across services without a distributed transaction: a sequence of local transactions where each step has a compensating action that semantically undoes it if a later step fails.

Local transaction — One step of the saga that runs atomically against a single service or database (e.g. one conditional write to the inventory table). The saga is a chain of these.

Compensating transaction — The semantic inverse of a forward step — a new business action that neutralizes a committed one (refund a charge, release a reservation). Not a database ROLLBACK; the original already committed.

Orchestration — A saga style where one central coordinator (here, Step Functions) owns the sequence, calls each service, and drives compensation on failure. Central control, easy to audit.

Choreography — A saga style with no coordinator: each service reacts to events and emits its own, so the workflow emerges from the chain of reactions. Loosest coupling, hardest to observe.

Step Functions — AWS’s serverless workflow (orchestration) service. You define a state machine in JSON (Amazon States Language) and it runs it durably, with built-in retries, error catching, and execution history.

Amazon States Language (ASL) — The JSON dialect that defines a Step Functions state machine as a graph of states (Task, Choice, Parallel, Map, etc.).

Task / Choice / Parallel / Map — Core ASL state types: Task does work (invoke a Lambda or call a service), Choice branches on data, Parallel runs branches concurrently, Map runs a sub-flow once per array item.

Standard vs Express workflow — Two Step Functions execution types. Standard: up to 1 year, exactly-once execution, full retained history, priced per state transition — used for auditable sagas. Express: up to 5 minutes, at-least-once, no native history, priced per execution — used for high-volume short flows.

Retry / Catch — Per-state error handling in ASL. Retry re-runs the state on matching errors with backoff (forward recovery); Catch routes a still-failing error to another state (typically into compensation — backward recovery).

Service-integration pattern — How a Task calls a downstream: Request/Response (call and take the immediate result), Run-a-Job .sync (block until a long job finishes), or Wait-for-Callback .waitForTaskToken (pause until an external system returns a task token).

Task token — An opaque token Step Functions hands to a worker in the .waitForTaskToken pattern; the external system later calls SendTaskSuccess/SendTaskFailure with it to resume the paused execution.

Idempotency — The property that applying an operation twice has the same effect as applying it once. What makes at-least-once delivery safe (no double-charge, no double-decrement).

Idempotency key — A unique token (derived from the order/request) attached to an operation so a downstream — a payment gateway, your own DynamoDB write — can recognize and dedupe a repeat.

At-least-once / exactly-once — Delivery guarantees. At-least-once (EventBridge, SQS standard, Lambda retries) may deliver duplicates. Exactly-once is rare, per-hop, and does not compose end-to-end — so you design for at-least-once and add idempotency.

Dual-write problem — The bug where a component must update a database and publish an event with no shared transaction; a crash between them leaves the two out of sync (an order with no event, or an event with no order).

Outbox pattern — The fix for the dual-write problem: write the business record and an “outbox” event in one atomic transaction, then a separate relay (here, DynamoDB Streams → Lambda) publishes the event afterwards — guaranteeing the event fires iff the record committed.

Conditional write — A DynamoDB write with a ConditionExpression that commits only if a condition holds (e.g. available >= qty). Fails with ConditionalCheckFailedException; the backbone of idempotent, race-safe inventory.

DynamoDB Streams — An ordered change-data-capture feed of every item change in a table, consumable by Lambda — the relay mechanism behind the outbox pattern and fulfillment triggers.

SQS FIFO / message group — A queue type that preserves strict order and gives exactly-once processing within a MessageGroupId (e.g. per orderId), while different groups flow in parallel. Lower throughput than standard queues.

Dead-letter queue (DLQ) — A queue that captures messages a consumer failed to process after maxReceiveCount attempts, so poison messages neither vanish nor loop forever. Operators redrive them once the downstream recovers.

Pivot transaction — In the saga taxonomy, the go/no-go step (e.g. payment capture): before it, steps are compensatable; after it, steps are retriable and the saga runs forward to completion.

Retriable transaction — A post-pivot step guaranteed to eventually succeed; on failure you retry it forward rather than compensating (e.g. booking the courier, sending the confirmation).

Eventual consistency — The system is correct eventually, but there is a window where a saga is mid-flight and an order is “half done.” The opposite of the atomic, all-or-nothing instant of a single ACID transaction.

Claim Check pattern — Instead of passing a large payload through states/events, store it (S3/DynamoDB) and pass only a pointer, keeping executions under the 256 KB payload limit and the history cheap.

EventBridge — AWS’s serverless event bus. Rules match event patterns and route to targets; supports archive/replay and schema registry. The decoupling seam between intake and the saga, and the fan-out fabric to other consumers.

Compensation rate — The share of sagas that ended in rollback. A rising compensation rate is the leading indicator that a downstream is degrading — often before it fully fails — so it’s the metric to alarm on early.