AWS Enterprise Architecture: Data Lakehouse

In a nutshell

Imagine two ways to run a distribution business. The first is to rent a fully-staffed, climate-controlled distribution centre: the building, the shelving, and the crew that moves your goods all come as one always-on bill, whether ten trucks arrive today or none. That is a data warehouse — storage and the compute that works the data are fused, and you pay for both around the clock. The second way is to keep your goods in cheap, near-indestructible storage, hire forklift crews by the hour only when a truck actually arrives, and run a single security desk that checks every crew’s badge the same way no matter which crew shows up. That is a lakehouse: your data sits once on Amazon S3, query engines spin up only when a query runs, and one governance service (Lake Formation) decides who may read which shelf — identically for every engine.

The trick that makes the cheap storage trustworthy is an open table format (Apache Iceberg is the AWS default). Plain files on S3 are like an unlabelled storage unit — you cannot safely update them, two people writing at once corrupt each other, and you cannot ask “what did this look like last Tuesday?” Iceberg adds a thin metadata layer on top of the same files that gives you database-style guarantees — atomic writes, updates and deletes, time travel, and schema changes — without moving the data into a database. That single idea is the whole lesson: the reliability of a warehouse, on the price and openness of a lake.

Everything else here is the supporting cast. The medallion zones (bronze → silver → gold) are just three quality tiers of the same data — raw arrivals, cleaned-and-conformed, and business-ready. The catalog (Glue Data Catalog) is the card index of what tables exist; Lake Formation is the security desk on top of it. And the engines (Athena, Redshift Spectrum, EMR/SageMaker) are interchangeable crews you rent per job, all reading the same shelves.

Level: Advanced · Time: ~55 min

Prerequisites — you’ll get the most from this if you already know:

• The basics of Amazon S3 — buckets, prefixes, storage classes, and lifecycle (see S3 deep dive).
• How Apache Iceberg tables physically lay out on S3 and why compaction matters (see Deploy Apache Iceberg on S3 + Glue).
• Roughly what a data warehouse and a data lake each are, even at a hand-wave level — the next section sharpens the distinction.
• Comfort reading SQL and a little Terraform; no need to write Spark.

After this lesson you will be able to:

• Explain, to an exec or a junior engineer, exactly what problem a lakehouse solves and why it is a coupling problem, not a tooling one.
• Lay out the five tiers — ingestion, S3 storage with Iceberg, catalog + governance, processing, and consumption — and say what belongs in each.
• Choose the right query engine (Athena vs Redshift Spectrum vs EMR/SageMaker) for a given workload and defend the choice on cost.
• Wire governance so every engine enforces the same column masks and row filters through Lake Formation, and prove it in an audit.
• Name the operational jobs (compaction, snapshot expiry, catalog DR) that a lakehouse must run, and the traps that sink teams who skip them.
• Place the newer AWS direction — Amazon SageMaker Lakehouse, S3 Tables, and Zero-ETL — correctly against the established stack.

A lakehouse is not “a data lake with a SQL engine bolted on.” It is a deliberate architecture where open table formats give your S3 data the transactional guarantees of a warehouse, a single governance plane (Lake Formation) controls who sees which rows and columns, and you pick the cheapest engine that fits each workload — Athena for ad-hoc, Redshift Spectrum for BI joins against curated dimensions, EMR for heavy transforms and ML feature engineering. This article builds that architecture on AWS end to end and treats the hard parts — table format choice, governance propagation, small-file economics, and DR for a catalog that lives in three places — as first-class concerns.

The business scenario

Picture a company that has outgrown its warehouse but cannot afford a second one. This is true at 50 people and at 50,000.

The early-stage version: a Series B fintech runs a single Amazon Redshift cluster. It started as the reporting database; now it ingests clickstream, holds 18 months of transaction history, runs the dbt models, and serves the data-science notebooks. Every new use case means a bigger cluster, and the cluster is 24/7 even though batch jobs run for three hours a night. The CFO sees a line item that grows linearly with the company and asks the question every architect dreads: “Why does storing data we rarely read cost the same as the data we read constantly?”

The large-enterprise version: a retailer with 1,200 stores has a Redshift estate, a Hadoop cluster a previous team built, three Glue jobs nobody owns, and an analytics team that exports CSVs to laptops because getting access through the warehouse takes two weeks. Data is duplicated five times. Nobody can answer “who can see PII” with a straight face during the audit.

Both companies have the same underlying problem and it is not a tooling problem — it is a coupling problem:

• Storage is coupled to compute. You pay for warehouse storage at warehouse prices, and you scale the two together even though they have nothing to do with each other.
• One engine is forced to do every job. Ad-hoc exploration, scheduled BI, and 4-hour ML transforms have wildly different cost and latency profiles, but they all hit the same cluster.
• Governance is per-engine. Redshift grants, Glue IAM, and laptop CSVs each have their own access model, so the real answer to “who can read this column” is unknowable.
• Data is copied to be used. Every team that needs the data makes a copy, because the original is locked inside an engine.

The lakehouse decouples all four. One copy of data on S3 in open formats; many engines reading the same files; one governance plane (Lake Formation) that every engine honours; and compute you turn on only when a query runs. Storage drops to S3 prices (and tiers down further when cold), each workload runs on the engine sized for it, and “who can see this column” becomes a single Lake Formation policy that Athena, Redshift Spectrum, and EMR all enforce identically.

The target outcome: store once, govern once, query with whatever is cheapest for the job, and pay for compute by the query — not by the calendar.

Lake, warehouse, or lakehouse — get the framing right

Before any architecture, fix the vocabulary, because the three words are thrown around interchangeably and they are not the same thing. The clearest way to tell them apart is to ask “where does the data physically live, and what guarantees does that give me?”

• A data warehouse (Amazon Redshift) stores data in its own managed, columnar, proprietary format optimised for SQL analytics. It gives you full ACID transactions, fast joins, and mature SQL governance — but historically the storage and the compute were welded together, so you paid for a running cluster to keep data online, and to use the data anywhere else you had to copy it out.
• A data lake is just files on cheap object storage (S3) in open formats (CSV, JSON, Parquet). It is wonderfully cheap and open, and any engine can read it — but raw files have no transactions, no consistent schema, and no built-in governance. A half-finished write leaves garbage; two writers clobber each other; “who can see this column” is answered by coarse S3 bucket policies, if at all. A lake is where data goes to be cheap, not to be trusted.
• A lakehouse keeps the lake’s cheap open storage and bolts on the warehouse’s guarantees, using an open table format (Iceberg) for ACID plus a single governance plane (Lake Formation) for fine-grained, engine-agnostic access control. One copy of data; warehouse-grade trust; many engines.

The row that matters most is governance. A warehouse governs data it owns; a lake barely governs at all; a lakehouse governs one shared copy so consistently that Athena, Redshift Spectrum, and EMR all see the exact same masked columns and filtered rows. That is the property you cannot get by “bolting a SQL engine onto a lake,” and it is the reason the lakehouse is not just a rebranded lake.

A worked example — one query’s journey. An analyst runs, in Athena:

SELECT store_id, SUM(net_sales) AS sales
FROM gold.daily_store_sales
WHERE region = 'US' AND sale_date >= DATE '2026-08-01'
GROUP BY store_id;

Here is what actually happens, and why the lakehouse design makes it both cheap and safe:

1. Athena asks the Glue Data Catalog where gold.daily_store_sales lives and what its schema is. The catalog answers with the S3 location and the fact that it is an Iceberg table — Athena does not yet have permission to read the files.
2. Because Lake Formation manages that location, Athena calls Lake Formation’s credential-vending API. Lake Formation checks this analyst’s grants: they hold SELECT on the table, but there is a column mask on customer_email (not selected here, so irrelevant) and no row filter for this role. It returns scoped, temporary credentials valid only for this table’s S3 prefixes, for this query.
3. Athena reads the Iceberg metadata — the current snapshot’s manifest list — and sees the table is partitioned by days(sale_date) and physically clustered so that region and date prune cleanly. It skips every data file that cannot contain US rows from August onward.
4. It scans only the surviving Parquet files, zstd-compressed and columnar, reading just the store_id, net_sales, region, and sale_date columns — say 12 GB, not the table’s full 3 TB.
5. The bill is 12 GB ÷ 1024 × $5/TB ≈ $0.06, and it returns in seconds. The same data as raw uncompressed CSV, unpartitioned, would force a full 3 TB scan — roughly $15 and 100× slower — with no way to hide customer_email from someone who shouldn’t see it.

Every design choice in the rest of this lesson exists to make that journey cheap (partitioning, compression, compaction), safe (Lake Formation vending instead of raw S3 access), and identical no matter which engine the analyst happened to use.

Architecture overview

The data path runs left to right through five logical tiers. Imagine the diagram as a wide flow with a governance plane spanning the full width underneath and an observability plane spanning the full width on top — everything in the middle passes through both.

1. Ingestion (left edge). Three classes of source land in a raw landing bucket on S3. Batch files (vendor drops, exports) arrive via AWS DMS for relational CDC or direct S3 uploads. Streaming events (clickstream, app telemetry, IoT) flow through Amazon Kinesis Data Streams into Amazon Data Firehose, which buffers and writes partitioned objects to S3. SaaS and operational data come through AWS Glue connectors or AWS AppFlow. Nothing transforms here — raw is immutable and append-only, the system of record for replay.

2. The lake on S3 (center, the spine). S3 holds three zones following the medallion pattern — bronze (raw, as-landed), silver (cleaned, conformed, deduplicated), and gold (business-level aggregates and dimensional models). Crucially, silver and gold tables are written as open table formats — Apache Iceberg is the default on AWS in 2026 — which means each table directory contains data files plus a metadata layer that gives ACID transactions, snapshot isolation, schema evolution, and time travel directly on S3. This is the line between a “data lake” and a “lakehouse.”

3. Catalog and governance (the plane underneath). The AWS Glue Data Catalog is the single technical metadata store — every table, schema, and partition is registered once and every engine reads it. AWS Lake Formation sits on top of the catalog as the permission layer: it owns the S3 locations, and instead of granting engines raw S3 access, you grant database/table/column/row/cell-level permissions in Lake Formation. Athena, Redshift Spectrum, and EMR all call Lake Formation’s credential-vending API at query time, so a single policy (“analysts cannot see the ssn column; the EU team sees only EU rows”) is enforced identically across every engine.

4. Processing and transformation. AWS Glue (Spark serverless ETL) does the bronze→silver→gold transforms on a schedule or event trigger, writing Iceberg tables and committing atomically. For heavy, long-running, or specialized work — large reprocessing, complex Spark/ML feature pipelines, or jobs needing specific libraries — Amazon EMR (on EC2 with Spot, or EMR Serverless) reads the same S3 + Glue Catalog and writes back the same Iceberg tables. Glue and EMR are not competitors here; they are two compute shapes over one storage and one catalog, chosen per job.

5. Consumption (right edge). Three query engines serve three patterns, all reading the same gold (and silver) tables:

• Amazon Athena — serverless, pay-per-TB-scanned SQL for ad-hoc exploration, data-science queries, and anything bursty. Zero standing cost.
• Amazon Redshift Spectrum — the warehouse’s external-table feature: BI tools connect to a small Redshift cluster (or Serverless) that holds hot dimensions locally and joins them against the gold Iceberg tables on S3 via Spectrum, so you get warehouse-grade BI performance without loading the whole lake into the warehouse.
• Amazon EMR / SageMaker — read the lake directly for ML training and feature engineering.

BI tools (Amazon QuickSight, Tableau, Power BI) sit beyond the engines. The key property: the data is never copied into any engine permanently. S3 is the one copy; engines are ephemeral lenses over it, and Lake Formation governs every lens the same way.

Component breakdown

Why each is here, briefly:

S3 zones, not one bucket. Three buckets give you blast-radius isolation, distinct lifecycle policies (bronze tiers aggressively, gold rarely), and clean IAM/Lake Formation boundaries. Bronze is immutable and replayable; silver and gold are recomputable from bronze, which is the backbone of your DR story.

Iceberg is the decision that makes it a lakehouse. Plain Parquet on S3 has no transactions: a failed Spark job leaves half-written files and readers see corruption. Iceberg’s metadata layer gives atomic commits (a query sees the table before or after a write, never mid-write), time travel (FOR TIMESTAMP AS OF for audits and “oops” recovery), and schema evolution without rewriting data. On AWS in 2026, Iceberg has first-class support across Glue, Athena (v3), EMR, and Redshift, which is why it’s the default over Hudi/Delta here — one format every engine reads natively.

Glue Catalog is the contract; Lake Formation is the lock. The catalog says what exists (one source of truth for schema). Lake Formation says who may see it — and because every engine vends credentials through Lake Formation rather than reading S3 directly, you cannot accidentally have “Athena can see the SSN column but the BI tool can’t.” The policy lives in one place and travels with the data.

Glue vs. EMR is a workload decision, not a religious one. Glue: serverless, fast to start, ideal for the steady bronze→silver→gold pipeline and teams who don’t want to manage clusters. EMR: when you need Spot economics on long jobs, specific library versions, very large reprocessing, or co-locating ML feature engineering with training. Both write the same Iceberg tables to the same S3 catalog — you can move a job from one to the other without changing the data.

Three query engines because one size never fits. A data scientist running SELECT ... LIMIT 100 ten times an hour should not touch a warehouse cluster — that’s Athena (zero standing cost, pay per scan). A nightly executive dashboard joining a 10-row date dimension against a billion-row fact wants the warehouse’s local-dimension speed — that’s Redshift Spectrum (hot dims resident, facts on S3). A 4-hour model-training run wants EMR/SageMaker reading Parquet directly. Forcing all three onto one engine is exactly the coupling the lakehouse exists to break.

Implementation guidance

Bucket and zone layout. Three buckets with consistent prefixing:

s3://acme-lake-bronze-<acct>/<source>/<table>/ingest_date=YYYY-MM-DD/
s3://acme-lake-silver-<acct>/<domain>/<table>/        # Iceberg-managed layout
s3://acme-lake-gold-<acct>/<domain>/<table>/          # Iceberg-managed layout

Bronze is partitioned by ingest date for cheap pruning and lifecycle. Silver/gold use Iceberg hidden partitioning — you declare PARTITIONED BY (days(event_ts)) and queries filter on event_ts directly without knowing the physical layout, so you can change partitioning later without rewriting queries.

Terraform is the right IaC here (the team standardizes on it). Manage as code:

• S3 buckets + Block Public Access + SSE-KMS + lifecycle/Intelligent-Tiering (aws_s3_bucket, aws_s3_bucket_lifecycle_configuration).
• Glue databases and the catalog (aws_glue_catalog_database); register Iceberg tables via Glue jobs or Athena DDL rather than hand-defining columns.
• Lake Formation: aws_lakeformation_resource to register each S3 location, aws_lakeformation_lf_tag for the tag taxonomy, and aws_lakeformation_permissions / aws_lakeformation_lf_tag_policy for grants. Critically, remove the IAM-based IAMAllowedPrincipals default so Lake Formation actually governs (otherwise IAM still grants broad access and your fine-grained policy is decorative).
• Glue jobs (aws_glue_job), triggers/workflows, EMR (aws_emr_cluster or aws_emrserverless_application), Athena workgroups (aws_athena_workgroup) with per-query scan caps, and Redshift Serverless namespace/workgroup.

Iceberg table creation (via Athena/Glue DDL) — note the explicit format and compaction intent:

CREATE TABLE silver.transactions ( ... )
PARTITIONED BY (days(event_ts))
LOCATION 's3://acme-lake-silver-<acct>/payments/transactions/'
TBLPROPERTIES (
  'table_type'='ICEBERG',
  'format'='parquet',
  'write_compression'='zstd',
  'optimize_rewrite_delete_file_threshold'='10'
);

Then schedule OPTIMIZE silver.transactions REWRITE DATA USING BIN_PACK (compaction) and snapshot expiry as a Glue job — this is the small-file fix, covered below.

Incremental upserts from CDC use Iceberg MERGE INTO, which is the lakehouse’s answer to the old “you can’t update a data lake” problem:

MERGE INTO silver.customers t USING bronze_cdc.customers s
ON t.id = s.id
WHEN MATCHED AND s.op='D' THEN DELETE
WHEN MATCHED THEN UPDATE SET ...
WHEN NOT MATCHED THEN INSERT ...;

Networking. Keep all data-plane traffic on the AWS backbone via VPC Gateway Endpoint for S3 (free) and Interface (PrivateLink) endpoints for Glue, Lake Formation, Athena, KMS, and STS. EMR and Redshift run in private subnets, no public IPs; NAT only for OS/package egress. QuickSight reaches Redshift/Athena through a VPC connection. The result: queryable data never traverses the public internet.

Identity wiring. Human access flows from your IdP (the org standardizes on Entra ID) through IAM Identity Center with SAML/SCIM, mapping IdP groups to permission sets that assume data-access roles. Those roles are then granted Lake Formation permissions (not S3 permissions). So “analysts” in Entra → an Identity Center group → a permission set/role → a Lake Formation grant of SELECT on gold with the ssn column masked. Pipeline identities (Glue/EMR job roles) get their own Lake Formation grants. No human or job role gets direct s3:GetObject on the lake buckets — Lake Formation vends scoped, temporary credentials per query. This is the single most important wiring decision in the build.

Enterprise considerations

Security and Zero Trust. The model is “no implicit S3 access; every read is brokered.” Concretely: (1) Block Public Access on all three buckets, account-wide; (2) SSE-KMS with separate CMKs per zone so you can revoke gold independently and get per-zone audit in CloudTrail; (3) Lake Formation as the only path to data — pipeline and human roles hold Lake Formation grants, not bucket policies; (4) column masking on PII (ssn, card_pan) and row/cell filters for tenancy/region (EU analysts see only region='EU'); (5) all access on PrivateLink; (6) CloudTrail data events on the buckets plus Lake Formation’s own audit log give you a complete “who read which column when” trail. Tag-based access control (LF-Tags) keeps this manageable: tag tables confidentiality=pii once and grant on the tag, so new tables inherit policy automatically instead of needing per-table grants.

Cost optimization — this is where the lakehouse earns its keep, and it’s multi-layered:

• Storage tiering: Intelligent-Tiering on bronze auto-moves cold raw to cheaper tiers; lifecycle pushes ancient raw to Glacier. Gold stays in Standard (it’s read constantly). You stop paying warehouse prices for cold data entirely.
• Engine right-sizing: Athena’s pay-per-TB-scanned means ad-hoc cost tracks usage, not uptime — and Parquet + zstd + Iceberg partition pruning cut bytes scanned by 90%+ vs. raw, directly cutting the Athena bill. Workgroup per-query scan limits stop a runaway SELECT * from costing thousands.
• Compute economics: EMR Spot for task nodes saves 60–90% on reprocessing; Glue Flex discounts non-urgent jobs; Redshift runs Serverless or a small cluster holding only hot dimensions rather than the whole lake.
• The structural win: decoupling means you scale storage and compute independently and turn compute off between queries — the line item the CFO hated stops growing with the company.

Scalability. S3 scales to effectively unlimited objects and throughput; partition-prefix design avoids hotspots. Athena and Glue are serverless and absorb concurrency automatically. EMR and Redshift scale horizontally (and Redshift concurrency-scaling adds transient clusters for BI spikes). The catalog scales to millions of partitions. The real scaling discipline is file hygiene, not capacity: streaming ingestion creates millions of tiny files that wreck query performance, so scheduled Iceberg compaction (OPTIMIZE) and snapshot/orphan-file expiry are mandatory operational jobs, not nice-to-haves.

Reliability and DR (RTO/RPO).

• S3 durability is 11 nines; Cross-Region Replication on bronze (the system of record) gives geographic protection. Because silver/gold are recomputable from bronze, you replicate the irreplaceable layer and rebuild the rest.
• Catalog is the subtle DR risk — it lives in Glue/Lake Formation, not S3, so it needs its own protection: export catalog definitions and Lake Formation grants via IaC (Terraform state + scripted exports) so the metadata layer is reproducible in the DR region. A lake with no catalog is just opaque files.
• Targets: with CRR + IaC-reproducible catalog, RPO ≈ 15 min for bronze (replication lag) and RTO of a few hours in a region failure — re-point the catalog, re-run pipelines to rebuild silver/gold from replicated bronze. Iceberg time travel additionally gives near-zero-RTO recovery from logical corruption (bad pipeline run): roll a table back to the prior snapshot in seconds.

Observability. Glue/EMR push Spark metrics and logs to CloudWatch; Athena and Spectrum query history expose bytes scanned and runtime per query (the cost signal). Critically, add data-quality gates — AWS Glue Data Quality (DQDL rules) on the silver→gold boundary so freshness, null-rate, and referential checks fail the pipeline before bad data reaches dashboards. CloudTrail + Lake Formation audit logs cover access observability. Track three SLOs: pipeline freshness (gold lag), query cost (bytes scanned/day), and DQ pass rate.

Governance. Lake Formation LF-Tags are the scalable model — a taxonomy like domain, confidentiality, retention applied to databases/tables, with grants written against tags so policy is inherited, not hand-maintained per object. This is also the foundation for a data-mesh evolution: each domain owns its silver/gold databases and grants access to others via Lake Formation cross-account sharing, with a central team owning only the tag taxonomy and platform. Pair with a business catalog (e.g. Amazon DataZone) for discovery and access requests on top of the technical Glue catalog.

Reference enterprise example

NorthPeak Retail is a fictional 1,400-store home-goods chain, ~$4.2B revenue, with the classic mess: a Redshift cluster at the limit of its node count, an aging self-managed Hadoop cluster, eight unowned Glue jobs, and an analytics team that exports CSVs because warehouse access takes two weeks. Storage and compute are fused; the annual data-platform bill is ~$2.1M and rising, and the last audit flagged that nobody could prove who could read customer PII.

What they built. Over two quarters they moved to the architecture above:

• Ingestion: DMS CDC from the Oracle order system and the Postgres loyalty DB into bronze; Kinesis + Firehose for store-POS and web clickstream (~2.5B events/day); AppFlow for the Salesforce CRM. Bronze landed ~18 TB/day raw.
• Lake: ~3.5 PB total on S3. Bronze on Intelligent-Tiering (≈70% of volume, mostly cold) with Glacier lifecycle past 13 months; silver/gold as Iceberg (≈900 TB), zstd-compressed, partitioned by day.
• Processing: Glue 5.0 ran the steady bronze→silver→gold pipeline (MERGE INTO for CDC upserts); EMR Serverless handled the monthly 14-month basket-affinity reprocessing and the ML feature pipeline for demand forecasting. They retired the Hadoop cluster.
• Governance: Lake Formation with LF-Tags (domain, confidentiality, retention). customer.email and payment.card_pan column-masked for analysts; a row filter restricted the EU e-commerce subsidiary’s team to EU customers. All eight legacy Glue jobs’ direct S3 access was removed and re-granted through Lake Formation.
• Consumption: Athena for the 140-person analytics/DS org (ad-hoc, no more CSV exports); Redshift Serverless + Spectrum for the ~40 executive QuickSight dashboards (hot date/store/product dimensions resident, billion-row sales facts joined from S3 gold); SageMaker on the gold feature tables for forecasting.

Decisions worth noting. They chose Iceberg over Delta for native Redshift Spectrum + Athena v3 support across every engine. They kept Redshift (Serverless) only for hot dimensions rather than loading the lake into it — Spectrum does the heavy fact joins on S3. They set Athena workgroup per-query scan caps at 2 TB after one analyst’s SELECT * scanned 40 TB in the pilot. They made nightly OPTIMIZE + snapshot expiry a first-class pipeline after streaming ingestion produced 9M tiny files in week one and Athena latency tripled.

The outcome (12 months).

• Platform spend fell from ~$2.1M to ~$1.15M/yr (~45%) — almost entirely from decoupling cold storage from warehouse pricing and turning compute off between jobs.
• Executive dashboard latency dropped materially: Spectrum on compacted, partition-pruned Iceberg gold beat the old over-subscribed cluster.
• The PII audit finding closed: one Lake Formation report now answers “who can read card_pan” across every engine.
• Analyst time-to-data went from ~2 weeks to same-day via Identity Center + LF-Tag grants — and the CSV-on-laptop shadow IT disappeared.
• DR: bronze CRR to a second region + IaC-reproducible catalog gave a tested RPO ≈ 15 min / RTO ≈ 4 h; a corrupted forecasting run was recovered in minutes via Iceberg time travel instead of a half-day rebuild.

When to use it

Use this lakehouse when:

• You have multiple consumption patterns (ad-hoc + BI + ML) over the same data and one engine is being stretched to serve all of them.
• Storage is growing faster than query demand and you’re paying warehouse prices to keep cold data online.
• You need fine-grained, engine-agnostic governance (column/row/cell) provable in an audit — Lake Formation is the headline reason to choose this over a plain lake.
• You want open formats (Iceberg/Parquet) to avoid engine lock-in and let teams pick the right compute per job.

Trade-offs and anti-patterns:

• Operational surface is larger than a single warehouse. You now own file hygiene (compaction, snapshot/orphan expiry), catalog DR, and a multi-engine cost story. Anti-pattern: standing up the lake and skipping compaction — streaming ingestion will bury you in small files and query performance collapses. These jobs are mandatory.
• Lake Formation is easy to leave decorative. If you forget to remove IAMAllowedPrincipals, IAM still grants broad S3 access and your fine-grained policy does nothing. Anti-pattern: configuring column masks while a job role still has direct bucket access — close every S3 back door or the governance is theatre.
• Don’t put sub-second, high-concurrency operational lookups on it. Athena/Spectrum are analytical (seconds-to-minutes). Point-read, low-latency app traffic belongs on DynamoDB/Aurora; the lakehouse is for analytics, not your application’s hot path.
• Don’t force everything into Iceberg on day one. Bronze stays raw; only curate silver/gold. Converting throwaway intermediate data to managed tables adds compaction cost for no benefit.

Alternatives and how to choose:

The lakehouse on AWS is the right default when one governed copy of data on S3, queried by the cheapest engine that fits each job, beats the cost and access friction of a single all-purpose warehouse — which, for any organization running ad-hoc, BI, and ML over a growing dataset, is most of the time.

Going deeper

The sections above are the architecture an experienced team can ship. This section is the layer under it — what Iceberg actually writes to S3, how Lake Formation hands out credentials, the newer AWS building blocks that are quietly redrawing the picture in 2026, and the performance math you need when someone asks “why is this query slow / expensive.”

What Iceberg actually writes to S3 (and why atomic commits work)

An Iceberg table is not a magic file type — it is a tree of small metadata files sitting next to your Parquet data, plus one pointer that the catalog owns. Understanding the tree explains every Iceberg feature you rely on.

Glue Catalog entry  ──▶  metadata/v37.metadata.json     (current table state, schema, partition spec, snapshot list)
                              │
                              └─▶ snap-<id>.avro          (a "snapshot" = a manifest list, one per commit)
                                       │
                                       ├─▶ manifest-a.avro   (lists data files + per-file column stats: min/max, null counts)
                                       └─▶ manifest-b.avro
                                                │
                                                ├─▶ 00001-data.parquet   (your actual rows)
                                                ├─▶ 00002-data.parquet
                                                └─▶ 00003-delete.parquet  (merge-on-read delete markers, if any)

Every write — an INSERT, a MERGE INTO, a compaction — produces a new metadata.json and a new snapshot; nothing existing is mutated in place. The commit is atomic because it ends with a single compare-and-swap on the catalog pointer: “if the current metadata is still v36, make it v37.” If two jobs race, one wins the swap and the other retries against the new state. That is the whole trick behind “a reader sees the table before or after a write, never mid-write.” It is also why time travel is free: old snapshots still point at their old data files, so FOR TIMESTAMP AS OF just reads an older metadata pointer.

The column min/max stats in the manifest files are what make queries fast — the engine reads the manifest, sees “this data file’s sale_date ranges 2026-01-01 to 2026-01-31,” and skips the whole file for an August query without opening it. This is file pruning, and it is why partitioning and sorted writes matter so much: they tighten those min/max ranges.

Copy-on-write vs merge-on-read — the write-mode decision

Iceberg offers two ways to apply an UPDATE/DELETE/MERGE, set per operation via table properties (write.update.mode, write.delete.mode, write.merge.mode):

The rule of thumb the reference architecture uses: MOR on silver (CDC hammers it with upserts all day, and you compact hourly), COW on gold (read constantly by dashboards, updated in controlled batches). Getting this backwards — COW on a high-churn CDC table — is a classic way to make Glue jobs run 10× longer than they should, because every micro-batch rewrites gigabytes of untouched data.

Iceberg vs Hudi vs Delta — why Iceberg is the AWS default

All three are open table formats that add ACID to files on S3. They are more alike than different; the choice on AWS is mostly about native, first-class support across every AWS engine.

The tie-breaker is not a benchmark; it is that AWS is putting its own new managed services — S3 Tables and the SageMaker Lakehouse catalog — on Iceberg. Choosing Iceberg means one format that Athena, Redshift Spectrum, EMR, and the newer managed layers all read natively, with no conversion. Pick Hudi only if you have a genuine record-level streaming-upsert case with its indexing needs; pick Delta mainly if you are also heavily invested in Databricks.

The 2026 direction: S3 Tables, SageMaker Lakehouse, and Zero-ETL

Three newer AWS capabilities are collapsing the seams in the architecture above. They are worth knowing even if you build on the established stack today.

• Amazon S3 Tables — a purpose-built S3 bucket type (table buckets) that stores Iceberg tables as a first-class resource and runs the maintenance for you: automatic compaction, snapshot expiration, and unreferenced-file removal. It is AWS taking the mandatory-but-tedious file-hygiene jobs (the ones the “When to use it” section warns you never to skip) and making them managed. If you are starting fresh, table buckets remove a large slice of the operational surface — at the cost of less control over exactly how and when maintenance runs.
• Amazon SageMaker Lakehouse — a unifying catalog layer that presents data from both your S3 lake and your Redshift Managed Storage through a single Apache Iceberg REST catalog interface, governed by Lake Formation. The payoff: any Iceberg-compatible engine (Athena, EMR, Spark, third-party) can query lake data and warehouse data through one catalog with one permission model, without copying warehouse tables out to the lake or vice versa. It is delivered inside Amazon SageMaker Unified Studio, a single environment that folds together the old SageMaker Studio, data prep, SQL analytics, and the DataZone-based business catalog (now the SageMaker Catalog). Strategically, this is AWS answering “lake or warehouse?” with “one Iceberg-governed surface over both.”
• Zero-ETL — managed, near-real-time replication that removes the pipeline you would otherwise hand-build. Aurora (MySQL/PostgreSQL) and RDS for MySQL replicate into Redshift within seconds of a source commit, with no Glue job to own; there are also Zero-ETL integrations from DynamoDB and into the SageMaker Lakehouse / S3. It does not replace DMS + medallion curation for everything — you still want bronze as the replayable system of record and silver/gold as governed, conformed tables — but for “get this operational table queryable next to the lake, now, without building CDC plumbing,” Zero-ETL is the shortest path.

Where do these fit the reference architecture? Treat them as upgrades to specific boxes, not a rewrite: S3 Tables can back your silver/gold Iceberg storage and retire your compaction Glue jobs; SageMaker Lakehouse can become the catalog/governance plane spanning lake + Redshift; Zero-ETL can replace the DMS-to-bronze CDC path for the operational databases where seconds-fresh warehouse copies matter more than a replayable raw landing zone.

Redshift in a lakehouse: RA3, Managed Storage, and data sharing

Redshift is not the enemy of the lakehouse — in this design it is the BI-serving engine that holds hot dimensions, and three of its features are what let it play nicely with S3:

• RA3 node types (ra3.xlplus → ra3.16xlarge) separate compute from storage. Their data lives in Redshift Managed Storage (RMS), which is backed by S3 under the hood and billed separately from compute — so you size the cluster for query concurrency, not for how much data you keep. This is Redshift adopting the lakehouse’s own decoupling.
• Redshift Spectrum reads the S3 gold Iceberg tables directly as external tables (CREATE EXTERNAL SCHEMA ... FROM DATA CATALOG), so a small cluster of hot dimensions can join a billion-row fact that never leaves S3. You get warehouse-grade join performance without loading the lake.
• Redshift data sharing exposes live RMS data to other Redshift namespaces/clusters (and to compute in other accounts/Regions) without copying it — producer writes once, many consumers read. Combined with SageMaker Lakehouse, the same governed data reaches both S3-side and Redshift-side consumers.

The mental model: RA3 + RMS make Redshift storage cheap and decoupled; Spectrum makes S3 gold joinable; data sharing makes one copy reach many consumers. Redshift becomes a lens, like Athena and EMR — just the lens tuned for concurrent, low-latency BI.

Lake Formation credential vending — the mechanism, precisely

The claim “every engine enforces the same policy” rests on one API. When an integrated engine (Athena, Redshift Spectrum, EMR with the Lake Formation runtime role, or a Glue job) needs to read a governed table, it does not use its own IAM role’s S3 permissions. Instead it calls Lake Formation (GetTemporaryGlueTableCredentials / the credential-vending path), passing the table and the caller’s identity. Lake Formation:

1. Evaluates the caller’s grants (direct or via LF-Tags), including column masks and row/cell filters.
2. If the query touches a filtered column or row set, it rewrites the request plan so the engine physically cannot read excluded data — column projection drops masked columns; row filters become predicate pushdowns.
3. Returns STS credentials scoped by an IAM session policy to exactly the S3 prefixes and objects the caller may read, valid for the query’s lifetime.

Two consequences matter in production. First, this is why you must remove IAMAllowedPrincipals — while that legacy grant exists, engines fall back to their own broad S3 IAM and never call the vending path, so your fine-grained policy is bypassed entirely. Second, it is why no human or job role should carry s3:GetObject on the lake buckets: if they do, they have a back door around the security desk. The whole model is “the only key to the shelves is issued, per-query, by Lake Formation.”

Small-file economics — the math, made concrete

“Compaction is mandatory” is easy to say; here is why, quantified. Suppose a Firehose stream lands one object every 60 seconds per partition, and you have 200 active partitions. That is 200 × 1440 = 288,000 tiny files per day. Each file forces the query engine to: open it, read its footer, and issue at least one S3 GET. Query planning time and request overhead now dominate; a scan that should take 8 seconds takes 4 minutes, and your S3 request bill balloons independently of bytes scanned.

Compaction (OPTIMIZE ... REWRITE DATA USING BIN_PACK) rewrites those 288,000 files into, say, 1,150 files of ~256 MB — the size range engines are tuned for. The scan drops back to seconds, request counts fall by ~250×, and snapshot expiry (VACUUM, or S3 Tables’ automatic maintenance) then deletes the now-orphaned small files so you stop paying to store them. The design levers that prevent the problem in the first place: tune Firehose buffer hints higher (fewer, bigger objects on landing) and prefer MOR + scheduled compaction over per-micro-batch rewrites. The design lever that fixes it: never ship a streaming lakehouse without compaction and expiry as first-class, monitored pipeline jobs.

Concurrency, cost caps, and the failure modes worth pre-empting

• Athena concurrency is soft-limited per Region and account; heavy BI concurrency belongs on Redshift (with concurrency scaling for spikes), not on Athena. Use Athena workgroups with a per-query bytes-scanned cutoff (e.g. 2 TB) so one SELECT * cannot scan 40 TB and bill thousands — a real failure mode, not a hypothetical.
• Iceberg commit conflicts appear under many concurrent writers to one table (the compare-and-swap loses and retries). Reduce write fan-in per table, or isolate high-churn writers, rather than cranking retries.
• The catalog is the single point that is not on S3. Glue Data Catalog and Lake Formation grants live in the control plane; a lake with an unrecoverable catalog is opaque files. Export the catalog and grants via IaC/scripted exports so the metadata layer is reproducible in a DR Region — this is the DR gap teams discover too late.
• Iceberg time travel is your fastest logical-corruption recovery. A bad pipeline run that poisoned gold is undone in seconds by rolling the table to the prior snapshot — no restore, no rebuild — which is a categorically better RTO than replaying from bronze for logical (not physical) failures.

Practice challenges

Work these in order — they escalate from “can you read the architecture” to “can you defend a design decision under cost and security pressure.” Try each before opening the solution. Nothing here needs a live AWS account; the point is reasoning, not running.

1. (Beginner) Place the zones. A vendor drops a raw daily CSV of orders; your pipeline cleans and deduplicates it, then a nightly job builds a per-store revenue summary the dashboards read. Which medallion zone (bronze / silver / gold) does each of the three artefacts belong in, and which one must stay immutable?

<details><summary>Solution</summary>

Raw CSV → bronze (as-landed, immutable, append-only, replayable). Cleaned + deduplicated orders → silver (conformed). Per-store revenue summary → gold (business-level aggregate the dashboards read). Bronze is the one that must stay immutable — it is the system of record you replay silver and gold from.

Why: the medallion pattern is quality tiers of the same data; keeping bronze immutable is what makes silver/gold recomputable, which underpins the whole DR story. </details>

2. (Beginner→Intermediate) Pick the engine. Match each workload to Athena, Redshift Spectrum, or EMR/SageMaker: (a) a data scientist runs SELECT ... LIMIT 100 twenty times an hour while exploring; (b) 40 executive QuickSight dashboards join a 10-row date dimension against a billion-row fact every morning; © a monthly 4-hour Spark job reprocesses 14 months of data and builds ML features.

<details><summary>Solution</summary>

(a) Athena — serverless, pay-per-scan, zero standing cost for bursty ad-hoc work. (b) Redshift Spectrum — hot dimensions resident in a small cluster, billion-row facts joined from S3 gold; warehouse-grade BI latency and concurrency. © EMR / SageMaker — long, heavy Spark with Spot economics, reading Parquet directly.

Why: the lakehouse exists precisely so ad-hoc, BI, and ML each run on the engine sized for them over one copy — forcing all three onto one engine is the coupling you are breaking. </details>

3. (Intermediate) The masks are decorative. A colleague has configured Lake Formation column masks on ssn and row filters for the EU team, but an audit shows analysts can still s3:GetObject the raw Parquet and read ssn directly. Name the two things most likely wrong and how you’d fix them.

<details><summary>Solution</summary>

(1) IAMAllowedPrincipals was never removed from the databases/tables, so engines fall back to broad IAM S3 access and never call Lake Formation’s credential-vending path — remove it so Lake Formation actually governs. (2) The analyst/job roles still hold direct s3:GetObject on the lake buckets — a back door around the security desk. Strip all direct bucket access; the only path to data must be Lake Formation vending scoped, temporary credentials per query.

Why: fine-grained masks are enforced only when the engine reads through Lake Formation; any surviving direct S3 grant or the legacy IAM default makes the policy theatre. </details>

4. (Intermediate) Choose the write mode. You have a silver customers table that a CDC stream upserts thousands of times an hour, and a gold daily_sales table refreshed once nightly and read by dashboards all day. Which Iceberg write mode — copy-on-write or merge-on-read — for each, and what operational job does your choice make non-negotiable?

<details><summary>Solution</summary>

Silver customers → merge-on-read (MOR): cheap writes (delete files + new data, no full rewrite) suit the high-churn CDC stream. Gold daily_sales → copy-on-write (COW): clean files give the all-day dashboard reads the lowest read cost, and the once-nightly refresh makes the rewrite affordable. The non-negotiable job: frequent compaction (OPTIMIZE ... BIN_PACK) plus snapshot/orphan expiry on the MOR silver table, or delete files pile up and reads slow down.

Why: MOR trades read cost for write cost (right for churn), COW trades write cost for read cost (right for read-heavy) — and MOR without compaction degrades exactly where you chose it for speed. </details>

5. (Advanced) Do the cost math. An analyst’s SELECT * on a 3 TB gold table scans the whole thing at $5/TB. Your teammate proposes three fixes: (a) partition by sale_date and require a date predicate, (b) store zstd-compressed columnar Parquet and select only the 4 needed columns, © set an Athena workgroup per-query scan cap of 2 TB. Which fixes cut the bill, which is a guardrail, and roughly what does a well-partitioned, columnar version of the same query cost?

<details><summary>Solution</summary>

(a) and (b) cut the bill — partition pruning skips files outside the date range, and columnar + column projection + compression read only the needed columns’ bytes; together they typically cut bytes scanned by 90%+. © is a guardrail — it does not make any single well-written query cheaper, it caps the blast radius of a runaway SELECT * (stops it at 2 TB / $10 instead of scanning 40 TB / $200). A pruned, columnar version scanning ~12 GB costs 12 ÷ 1024 × $5 ≈ $0.06 versus 3 × $5 = $15 for the full raw scan — roughly 250× cheaper.

Why: Athena bills bytes scanned, so partitioning and columnar projection attack the bill directly; scan caps are cost insurance, not optimisation — you want both. </details>

6. (Advanced) Design the DR story. Bronze is your replayable system of record; silver and gold are recomputable from it. Sketch a DR plan that gives ~15-minute RPO and a few-hours RTO for a Region failure, and separately explain how you’d recover from a logical corruption (a bad pipeline run poisoned gold) in minutes. What is the one component that S3 durability does not protect, and how do you cover it?

<details><summary>Solution</summary>

Region failure: enable S3 Cross-Region Replication on bronze only (RPO ≈ replication lag ≈ 15 min); silver/gold are recomputable, so you don’t replicate them. In DR, re-point the catalog and re-run pipelines to rebuild silver/gold from replicated bronze (RTO ≈ a few hours). Logical corruption: use Iceberg time travel — roll the poisoned gold table back to the prior snapshot in seconds (FOR VERSION AS OF / snapshot rollback), no rebuild. The unprotected component: the catalog (Glue Data Catalog + Lake Formation grants) lives in the control plane, not S3 — export it via IaC/scripted exports so it is reproducible in the DR Region. Consider S3 Tables to make maintenance managed, but the catalog-DR gap is still yours to close.

Why: replicate only the irreplaceable layer and recompute the rest; time travel is the fast path for logical (not physical) failures; and a lake whose catalog you can’t rebuild is just opaque files. </details>

Common beginner mistakes

These are misconceptions, not typos — the wrong mental model that leads a well-meaning team into an expensive corner. Each pairs the trap with the model that avoids it.

“A lakehouse is just a data lake with Athena on top.” The seductive wrong idea: point Athena at some Parquet in S3 and call it a lakehouse. Without an open table format you have no ACID — a failed write leaves half-written files readers see as corruption — and without Lake Formation you have no consistent governance. The right model: a lakehouse is the combination of open table format (Iceberg, for transactions and time travel) and a single governance plane (Lake Formation, for engine-agnostic fine-grained access). Athena is a lens; Iceberg and Lake Formation are what make the thing a lakehouse.

“Iceberg makes storage slow/expensive, so I’ll skip it on hot data.” The misconception is that the metadata layer is overhead. It is the opposite: Iceberg’s per-file min/max stats are what let engines skip files without opening them, and hidden partitioning is what lets a query prune without knowing the physical layout. The right model: Iceberg on silver/gold usually makes queries faster and cheaper, not slower. Where you legitimately skip it is bronze — raw, throwaway, or intermediate data that no one queries as a table gains nothing from a managed format.

“Compaction is an optimisation I’ll get to later.” Beginners treat file hygiene as a nice-to-have. With streaming ingestion, “later” is week one: hundreds of thousands of tiny files per day turn an 8-second scan into minutes and inflate the S3 request bill independently of bytes scanned. The right model: compaction (OPTIMIZE) and snapshot/orphan expiry are mandatory, monitored pipeline jobs, not chores — or adopt S3 Tables to have AWS run them for you. A streaming lakehouse without compaction will collapse.

“I set up Lake Formation, so my data is governed.” The trap: configure beautiful column masks and row filters, then leave the legacy IAMAllowedPrincipals grant in place and let job roles keep direct s3:GetObject. Both are back doors that bypass every mask. The right model: Lake Formation governs only the reads that flow through its credential-vending path — so removing IAMAllowedPrincipals and stripping all direct bucket access from human and job roles is part of turning governance on, not an optional hardening step later.

“Redshift is the old way; a real lakehouse replaces it.” New teams often frame it as lake versus warehouse. In this architecture Redshift is a first-class engine — the one tuned for concurrent, low-latency BI on hot dimensions, with RA3/RMS decoupling its storage and Spectrum joining S3 gold. The right model: you don’t rip Redshift out, you right-size it to hot dimensions and let Spectrum do the heavy fact joins on S3. Pick the cheapest engine per job; sometimes that engine is Redshift.

“Zero-ETL / S3 Tables / SageMaker Lakehouse means I don’t need the medallion pipeline anymore.” The newest managed features are tempting to read as “the architecture is obsolete.” They are upgrades to specific boxes, not a replacement for curation. The right model: you still want bronze as the replayable system of record and silver/gold as governed, conformed tables. Zero-ETL shortcuts a CDC pipeline; S3 Tables shortcuts compaction; SageMaker Lakehouse unifies the catalog — none of them decide what “clean, conformed, business-ready” means for your data. That is still design work you own.

“Put everything in Iceberg on day one.” Over-eager teams convert every intermediate and throwaway dataset to managed tables. That adds compaction and metadata cost for data no one queries as a table. The right model: bronze stays raw; curate only silver and gold. Managed-table cost is justified by managed-table benefits (ACID, governance, time travel) — spend it where those benefits are actually consumed.

Glossary

• Lakehouse — an architecture that keeps data once on cheap open object storage (S3) but gives it warehouse-grade guarantees via an open table format (ACID) and a single governance plane (fine-grained access), so many engines query one governed copy.
• Data lake — raw files on object storage in open formats. Cheap and open, but with no transactions, no enforced schema, and only coarse (bucket-level) governance.
• Data warehouse — a system (Amazon Redshift) that stores data in its own managed format optimised for SQL analytics, with full ACID and mature governance; historically coupled storage and compute.
• Medallion architecture — the bronze → silver → gold quality tiers: bronze = raw as-landed (immutable, replayable), silver = cleaned/conformed/deduplicated, gold = business-level aggregates and dimensional models the dashboards read.
• Apache Iceberg — the open table format that is the AWS default: a metadata tree over Parquet files giving atomic commits, snapshot isolation, schema evolution, hidden partitioning, and time travel directly on S3.
• Open table format — a specification (Iceberg, Hudi, Delta) that adds database-style transactions and metadata to plain files on object storage, so any compatible engine reads them consistently.
• ACID — Atomicity, Consistency, Isolation, Durability: the transactional guarantees that make a table safe to write and read concurrently. Iceberg brings ACID to files on S3.
• Time travel — querying a table as it existed at a past point (FOR TIMESTAMP AS OF / FOR VERSION AS OF), used for audits and fast recovery from a bad write by rolling back to a prior snapshot.
• Snapshot — an immutable point-in-time state of an Iceberg table, created on every commit; old snapshots enable time travel until expired.
• Copy-on-write (COW) — an Iceberg write mode that rewrites whole data files on update/delete: costly writes, cheap reads. Best for read-heavy gold tables.
• Merge-on-read (MOR) — an Iceberg write mode that writes small delete files instead of rewriting: cheap writes, costlier reads until compaction. Best for high-churn silver tables.
• Compaction — rewriting many small data files into fewer large ones (OPTIMIZE ... REWRITE DATA USING BIN_PACK) to restore query performance; mandatory with streaming ingestion.
• Snapshot / orphan-file expiry — deleting old snapshots and unreferenced files (VACUUM) so expired data stops costing storage; the second half of file hygiene after compaction.
• Hidden partitioning — Iceberg’s ability to partition by a transform (days(event_ts)) that queries filter on transparently, so the physical layout can change without rewriting queries.
• File pruning — skipping data files a query can’t need, using the per-file min/max column stats in Iceberg manifests, without opening the files.
• AWS Glue Data Catalog — the shared technical metadata store: every table, schema, and partition registered once and read by every engine. The “card index” of the lakehouse.
• AWS Lake Formation — the governance plane on top of the catalog: fine-grained (database/table/column/row/cell) permissions enforced identically across engines via credential vending.
• Credential vending — Lake Formation issuing scoped, temporary STS credentials per query (instead of engines using their own S3 IAM), the mechanism that makes one policy apply to every engine.
• LF-Tags (tag-based access control) — attaching a taxonomy (domain, confidentiality, retention) to catalog objects and granting on the tags, so new tables inherit policy instead of needing per-table grants.
• IAMAllowedPrincipals — the legacy default that lets engines use broad IAM S3 access and bypass Lake Formation; must be removed or fine-grained governance is decorative.
• AWS Glue (ETL) — serverless Spark for the routine bronze → silver → gold transforms (Glue 5.0 = Spark 3.5); features include job bookmarks (incremental) and Flex (discounted non-urgent execution).
• Amazon EMR — managed Hadoop/Spark for heavy or specialised jobs, on EC2 with Spot (60–90% off) or EMR Serverless; reads the same S3 + Glue Catalog and writes the same Iceberg tables.
• Amazon Athena — serverless, pay-per-TB-scanned SQL (engine v3, Trino-based) for ad-hoc exploration; zero standing cost, governed by workgroups with per-query scan caps.
• Redshift Spectrum — Redshift’s external-table feature that joins hot local dimensions against gold Iceberg tables on S3, giving warehouse BI performance without loading the lake.
• RA3 / Redshift Managed Storage (RMS) — RA3 nodes separate compute from storage; RMS (S3-backed) is billed separately, so you size the cluster for concurrency, not data volume.
• Redshift data sharing — exposing live RMS data to other Redshift namespaces/clusters/accounts without copying it: write once, many consumers read.
• Zero-ETL — managed, near-real-time replication (Aurora/RDS → Redshift, DynamoDB → Redshift, into SageMaker Lakehouse/S3) that removes hand-built CDC pipelines for getting operational data queryable fast.
• Amazon S3 Tables — a purpose-built S3 bucket type storing Iceberg tables as a managed resource with automatic compaction, snapshot expiration, and unreferenced-file removal.
• Amazon SageMaker Lakehouse — a unifying Iceberg REST catalog presenting both S3 lake and Redshift (RMS) data through one Lake-Formation-governed interface, delivered in SageMaker Unified Studio.
• AWS DMS — Database Migration Service; here used for relational CDC (change data capture) landing into bronze, merged into silver with MERGE INTO.
• Kinesis Data Streams / Amazon Data Firehose — streaming ingestion; Streams carries events, Firehose buffers and writes partitioned (optionally Parquet) objects into bronze.
• AWS Glue Data Quality (DQDL) — rule-based data-quality gates on the silver → gold boundary that fail the pipeline before bad data reaches dashboards.
• Amazon DataZone / SageMaker Catalog — the business catalog for discovery and access requests layered on top of the technical Glue catalog (now surfaced as the SageMaker Catalog in Unified Studio).
• RPO / RTO — Recovery Point Objective (max acceptable data loss) and Recovery Time Objective (max acceptable downtime); here ~15 min / a few hours via bronze CRR + IaC-reproducible catalog.