AWS Lesson 85 of 123

AWS Landing Zone: Network Architecture — Transit Gateway, the Shared Services & Network Account, Centralized Egress/Ingress, Inspection, Direct Connect & IPAM

In a nutshell

Picture a large planned industrial park. Businesses (your workload accounts and their VPCs) move in one at a time, but nobody lets each business dig its own road to the highway, drill its own well, or invent its own street numbers — that way lies chaos. Instead the park authority lays one shared road network with a central interchange, runs shared utilities (water, power, a post office), and puts a single guarded gate with customs inspection on every road leaving the park. The AWS Landing Zone network is that park authority, and it lives in one dedicated Network account:

This foundation is built first and centrally for a brutal economic reason: the two things almost impossible to change later are the IP address plan and the packet path. Let an address overlap slip in and you may have to re-number a live network — a project measured in months. So the platform team builds this once, as code, and every new account simply inherits connectivity, DNS, egress, and on-prem reach the moment it is vended.

Level: Advanced · Time: ~59 min read

Prerequisites. This is Part 5 of the series and assumes the account structure from the earlier parts — AWS Organizations, the OU hierarchy, Control Tower with its Log Archive and Audit accounts, guardrail SCPs, and IAM Identity Center (see the Control Tower multi-account landing zone part). You should also be comfortable with VPC basics (subnets, route tables, IGW/NAT, security groups) and with CIDR notation.

After this lesson you will be able to:

Where this fits

In an AWS Landing Zone & Control Tower deployment, the first four parts gave you the account structure: AWS Organizations, the OU hierarchy, Control Tower with its mandatory Log Archive and Audit accounts, an SCP guardrail set, and IAM Identity Center for access. Control Tower deliberately stops short of building you an enterprise network — its default is one VPC per provisioned account with no transit, no inspection, and no hybrid links. Network Architecture is where the platform team fills that gap: a dedicated Network account that owns an AWS Transit Gateway hub, shared services every workload depends on, centralized egress and ingress so the internet edge is governed in one place, traffic inspection that no team can bypass, Direct Connect for the data-center backhaul, and an Amazon VPC IPAM plan so no two VPCs ever overlap. It produces the most expensive thing in the whole landing zone to change later — the IP plan and the packet path — so it is built once, centrally, and consumed by every spoke account through sharing.

AWS Landing Zone & Control Tower — animated overview

AWS Transit Gateway

What it is

AWS Transit Gateway (TGW) is a Regional, horizontally-scaled router that replaces the VPC-peering mesh with a hub-and-spoke topology. Each VPC, VPN, or Direct Connect connection becomes an attachment to the TGW; routing between attachments is governed by TGW route tables that you associate with attachments and into which routes are propagated. In a landing zone the TGW lives in the Network account and is shared to every workload account through AWS Resource Access Manager (RAM), so a spoke account creates only a VPC attachment — it never owns the hub. TGWs in different Regions are joined with inter-Region peering over the AWS global backbone (encrypted, no public internet), giving you a global mesh without managing a single tunnel.

Why it matters

VPC peering does not transit: it is strictly point-to-point and non-transitive, so n VPCs that all need to talk require n(n-1)/2 peerings, each with its own route-table entries to maintain. At 40 VPCs that is 780 peerings — operationally unmaintainable and well past several account limits. Transit Gateway collapses that to n attachments and a handful of route tables, and — critically — it is the segmentation primitive for the whole network. By giving production, non-production, and shared-services attachments different TGW route tables, you decide which environments can reach which others at the router, not with hundreds of security-group rules. It is also the single chokepoint through which you steer traffic into a firewall (covered under inspection) and out to on-premises (covered under Direct Connect).

How to do it well

Place exactly one TGW per Region in the Network account and share it org-wide with RAM (enable RAM sharing within AWS Organizations first, so principals resolve without invitations). Design multiple TGW route tables to encode segmentation, not a single default table. A proven pattern is one route table per security domain:

TGW route table Associated attachments What it can route to Effect
rt-prod Production VPC attachments Shared-services VPC, inspection/egress VPC, on-prem (DX) Prod talks to shared + leaves via inspection; no route to non-prod
rt-nonprod Dev/test/stage VPC attachments Shared-services VPC, inspection/egress VPC Isolated from prod entirely
rt-shared Shared services + inspection/egress + DX/VPN All spokes (propagated) Hub services and hybrid reachable from everywhere

Turn off route propagation where you want isolation and use static routes/blackholes to encode “these never talk.” Enable appliance mode on the inspection VPC attachment so the TGW keeps a flow pinned to the same firewall appliance for both directions (without it, asymmetric routing breaks stateful inspection across AZs). Attach the TGW to one subnet per Availability Zone in each VPC (a small dedicated /28 “TGW subnet” per AZ), and remember the attachment is only present in the AZs whose subnets you list — workloads in an un-attached AZ have no path. Manage the TGW, its route tables, associations, and RAM shares entirely as code (Terraform aws_ec2_transit_gateway* resources or CloudFormation).

Artifacts and decisions: the TGW definition and its ASN; the RAM share to the org; the set of TGW route tables encoding your segmentation matrix; the per-AZ TGW subnet standard; the appliance-mode and DNS-support settings; and an inter-Region peering design if you are multi-Region. The headline decision is your segmentation model — how many security domains (route tables) you operate, and which can reach which.

Worked example: a segmented Transit Gateway as code

The segmentation matrix above is only a diagram until it exists as routing. Here is the core of it in Terraform (illustrative; provider aws ~> 5.x, run from the Network account). One hub, three route tables, and a blackhole that makes “prod never talks to non-prod” a fact of the fabric rather than a hopeful security-group rule:

resource "aws_ec2_transit_gateway" "hub" {
  amazon_side_asn                 = 64512      # a private ASN (64512–65534)
  auto_accept_shared_attachments  = "enable"   # org-shared spokes attach with no manual accept
  default_route_table_association = "disable"   # we manage associations explicitly
  default_route_table_propagation = "disable"   # no accidental flat network
  dns_support                     = "enable"
  tags = { Name = "tgw-hub-use1" }
}

resource "aws_ec2_transit_gateway_route_table" "prod" {
  transit_gateway_id = aws_ec2_transit_gateway.hub.id
  tags               = { Name = "rt-prod" }
}
resource "aws_ec2_transit_gateway_route_table" "nonprod" {
  transit_gateway_id = aws_ec2_transit_gateway.hub.id
  tags               = { Name = "rt-nonprod" }
}
resource "aws_ec2_transit_gateway_route_table" "shared" {
  transit_gateway_id = aws_ec2_transit_gateway.hub.id
  tags               = { Name = "rt-shared" }
}

# A prod VPC attachment ASSOCIATES to rt-prod (its outbound decisions) ...
resource "aws_ec2_transit_gateway_route_table_association" "prod_app" {
  transit_gateway_attachment_id  = aws_ec2_transit_gateway_vpc_attachment.prod_app.id
  transit_gateway_route_table_id = aws_ec2_transit_gateway_route_table.prod.id
}

# ... and "prod can never reach non-prod" is a blackhole, not a comment:
resource "aws_ec2_transit_gateway_route" "prod_no_nonprod" {
  destination_cidr_block         = "10.36.0.0/14"   # the non-prod supernet
  blackhole                      = true
  transit_gateway_route_table_id = aws_ec2_transit_gateway_route_table.prod.id
}

Sharing the hub org-wide is three RAM resources — the share, the association (the TGW itself), and the principal (your Organization ARN). Turn on aws ram enable-sharing-with-aws-organization once in the management account first, or principals never resolve:

resource "aws_ram_resource_share" "tgw" {
  name                      = "tgw-hub-share"
  allow_external_principals = false
}
resource "aws_ram_resource_association" "tgw" {
  resource_arn       = aws_ec2_transit_gateway.hub.arn
  resource_share_arn = aws_ram_resource_share.tgw.arn
}
resource "aws_ram_principal_association" "org" {
  principal          = "arn:aws:organizations::123456789012:organization/o-exampleorgid"  # placeholder
  resource_share_arn = aws_ram_resource_share.tgw.arn
}

The spoke account then owns only an aws_ec2_transit_gateway_vpc_attachment pointing at the shared TGW id, plus a 0.0.0.0/0 (or summarized) route in its private subnets sending traffic to the gateway. It never sees the route tables, the blackholes, or the other spokes — exactly the blast-radius property you want. For the standalone mechanics of attachments, associations, propagation, and inter-Region peering, see the Transit Gateway multi-account architecture lesson.

Scale, quotas, and performance you should know

A landing-zone TGW is a long-lived, high-fan-in object, so its default quotas (most raiseable via Service Quotas) shape the design:

TGW dimension Default / limit Why it matters at landing-zone scale
Attachments per TGW 5,000 Comfortable for hundreds of accounts; the real ceiling is route-table complexity, not this
Route tables per TGW 20 (raiseable) You need one per security domain — plan domains, don’t sprawl
Routes per TGW route table 10,000 Summarize (IPAM pays off) so you propagate supernets, not per-VPC /20s
Peering attachments per TGW 50 Caps a full inter-Region mesh; Cloud WAN is the answer past a few Regions
Bandwidth per VPC attachment up to 100 Gbps (burst) Aggregate across many flows
Single flow (5-tuple) ~5 Gbps One large transfer can’t exceed this — parallelize or keep it off the hub
Jumbo frames 8500 bytes (VPC/Connect); 1500 (VPN) Don’t assume 9001 survives across the TGW

The single-flow ~5 Gbps ceiling surprises teams doing big backups or replication across the hub: the fix is many parallel flows or a per-VPC path, not a bigger TGW. And every byte crossing the TGW is billed for data processing (≈ $0.02/GB in us-east-1 at the time of writing — always verify current pricing) on top of the ≈ $0.05 per attachment-hour, which is the hidden cost lever the egress section returns to.

Shared services VPC

What it is

The shared services VPC is a VPC in the Network account (or a dedicated Shared Services account in larger estates) that hosts the infrastructure every workload account consumes but none should run independently: centralized DNS resolution (Route 53 Resolver inbound/outbound endpoints and Resolver rules), directory services (AWS Managed Microsoft AD), interface VPC endpoints (AWS PrivateLink) for AWS service APIs, package and patch mirrors, golden-image and CI agents, and monitoring/log-forwarding collectors. Spokes reach it over the Transit Gateway; they do not peer to it directly.

Why it matters

Without a shared services tier, every workload account re-creates the same expensive plumbing. The clearest example is interface VPC endpoints: each one carries an hourly charge per endpoint per AZ, so standing up endpoints for ssm, ssmmessages, ec2messages, ecr.api, ecr.dkr, logs, sts, kms, and a dozen more in every spoke is both costly and a configuration-drift nightmare. Centralizing them once and sharing access via TGW + centralized Route 53 Resolver turns N copies into one. The same logic applies to DNS: a single, authoritative resolution path (Resolver endpoints plus rules) means on-prem names, AWS private-hosted-zone names, and public names resolve identically from every account — the alternative is per-account DNS that answers differently depending on where you ask.

How to do it well

Make the shared services VPC the DNS hub. Create Route 53 Resolver inbound endpoints (so on-prem can resolve AWS private zones), outbound endpoints with Resolver rules (so AWS forwards on-prem domains to your data-center DNS), and share both the resolver rules and your central private hosted zones to spoke VPCs using RAM. For PrivateLink, decide the topology: either host the interface endpoints centrally and front them with Route 53 private hosted zones that you associate to spoke VPCs (so *.execute-api.<region>.amazonaws.com resolves to the central endpoint ENIs), or — increasingly common — keep latency-sensitive endpoints local and centralize only the long tail. Gateway endpoints for S3/DynamoDB are free and route-table based, so put those in each VPC directly. Run AWS Managed Microsoft AD here and share the directory to spokes. Keep this VPC non-routable to the internet directly — its egress goes through the centralized egress path like any other spoke.

Shared service AWS mechanism How spokes consume it
DNS resolution (hybrid) Route 53 Resolver inbound/outbound endpoints + rules RAM-shared Resolver rules + TGW path
Private name resolution Route 53 private hosted zones RAM-shared / associated to spoke VPCs
AWS API access (private) Interface VPC endpoints (PrivateLink) Central endpoints + shared private hosted zone, or local
Directory AWS Managed Microsoft AD RAM directory sharing
Patch/SSM, image build, log forwarders EC2/ECS in the shared VPC Reached over TGW

Artifacts and decisions: the shared services VPC CIDR and subnet plan; the Resolver endpoint and rule set; the RAM shares for Resolver rules and private hosted zones; the interface-endpoint inventory and the central-vs-local decision per service; the directory-sharing configuration. The key decision is what is genuinely shared vs. what stays local — over-centralizing latency-sensitive endpoints adds a cross-VPC hop to every API call.

Worked example: one DNS answer from every account

The hardest part of “shared services” to picture is hybrid DNS, so trace one lookup. A workload in a spoke asks for orders.db.meridian.internal (an on-prem name) and, a moment later, myqueue.sqs.us-east-1.amazonaws.com (an AWS name). You want both to resolve identically no matter which account asks:

  1. The spoke VPC uses the AmazonProvidedDNS resolver at its .2 address (the VPC CIDR base + 2).
  2. Because a Route 53 Resolver rule for meridian.internal is RAM-shared to the spoke and associated to its VPC, the .internal query is forwarded to the shared-services outbound Resolver endpoint, which relays it over the TGW/DX to on-prem DNS. The answer returns the same way.
  3. The AWS name resolves against Route 53 as usual; if you host interface endpoints centrally, a private hosted zone for sqs.us-east-1.amazonaws.com (RAM-associated to the spoke) points it at the central endpoint ENIs instead of the public API.
  4. On-prem systems that must resolve an AWS private zone hit the shared-services inbound Resolver endpoint — the mirror image of step 2.

Two Resolver endpoints (one inbound, one outbound) and a handful of rules, shared once, give every current and future account a single authoritative resolution path. The forwarding rule and its share are a few resources:

resource "aws_route53_resolver_rule" "onprem" {
  domain_name          = "meridian.internal"
  rule_type            = "FORWARD"
  resolver_endpoint_id = aws_route53_resolver_endpoint.outbound.id
  target_ip { ip = "10.34.0.10" }   # on-prem DNS, reachable over TGW/DX
  target_ip { ip = "10.34.0.11" }
  tags = { Name = "fwd-meridian-internal" }
}

# Share the rule org-wide; spokes then associate it to their VPCs.
resource "aws_ram_resource_association" "resolver_rule" {
  resource_arn       = aws_route53_resolver_rule.onprem.arn
  resource_share_arn = aws_ram_resource_share.dns.arn
}
A spoke asks for… Resolves via Because
*.meridian.internal (on-prem) Outbound Resolver endpoint → on-prem DNS RAM-shared FORWARD rule associated to the VPC
A central private-zone name Route 53 private hosted zone PHZ RAM-associated to the spoke VPC
*.amazonaws.com with central endpoints Interface endpoint ENIs in the shared VPC Private hosted zone overrides the public name
A public internet name Public Route 53 → out via central egress Default recursive resolution

Endpoint economics, made concrete. An interface endpoint bills roughly per-AZ-hour plus per-GB (≈ $0.01/hr/AZ + ≈ $0.01/GB — verify current pricing). Ten common endpoints (ssm, ssmmessages, ec2messages, ecr.api, ecr.dkr, logs, sts, kms, secretsmanager, monitoring) across 3 AZs is ~30 endpoint-AZ charges per account. In 50 accounts that is ~1,500 endpoint-AZ line items; centralizing to one shared set collapses that by well over an order of magnitude — the single biggest quiet cost win in the whole network build. The exception is latency- or throughput-sensitive endpoints (ecr.dkr in a heavy build account): measure the extra hub hop before you centralize those. Gateway endpoints for S3 and DynamoDB are free and route-table based, so keep those local in every VPC regardless.

Centralized egress and ingress

What it is

Centralized egress routes all outbound internet traffic from every spoke VPC through a single point — a NAT path and outbound firewall in the Network account — instead of giving each VPC its own NAT gateways and internet gateway. Centralized ingress does the inverse for inbound: public entry points (Application Load Balancers, AWS Global Accelerator, AWS Network Firewall or a third-party appliance) live in a dedicated ingress VPC, and traffic is forwarded inward to workloads in private spokes. Both hang off the Transit Gateway: spokes have no internet gateway of their own, and a default route (0.0.0.0/0) in the spoke points at the TGW, which carries it to the central egress VPC.

Why it matters

Decentralized egress means a NAT gateway in every VPC in every AZ — each one billed hourly and per GB processed — and, worse, an internet edge you cannot govern: outbound filtering, domain allow-listing, and egress logging would have to be re-implemented and audited in dozens of places. Centralizing puts the entire internet boundary in one account where one team owns the NAT gateways, the firewall policy, and the flow logs, and where a single SCP can deny spokes from ever attaching an internet gateway. The cost lever is real but two-sided: you consolidate NAT but you now pay for TGW data processing on every byte that crosses the hub, so the math favors centralization at scale and per-VPC NAT for a handful of chatty, high-throughput VPCs.

How to do it well

Build a central egress VPC in the Network account with public subnets (NAT gateway per AZ + internet gateway) and a TGW attachment; in the egress VPC’s TGW route table, summarize spoke CIDRs back to them, and in each spoke point 0.0.0.0/0 at the TGW. Put AWS Network Firewall (or a gateway-load-balancer-fronted appliance) inline so egress is inspected and domain-filtered, not just NAT’d. For ingress, stand up a dedicated ingress VPC: terminate public ALBs there, optionally front them with AWS Global Accelerator for anycast entry and AWS WAF on the ALB/CloudFront, and forward to workloads over the TGW — or use PrivateLink to publish a workload as an endpoint service that the ingress tier consumes. Keep appliance mode on the inspection/egress attachment so return traffic is symmetric. Always attach a DDoS posture: AWS Shield Advanced on the public ALBs/Global Accelerator and Route 53.

Edge concern Decentralized (per-VPC) Centralized (Network account)
Outbound IGW + NAT GW in every VPC/AZ One egress VPC; spokes default-route to TGW
Cost shape N×NAT hourly + per-GB 1×NAT set + TGW data processing per GB
Filtering/logging Re-implemented per VPC One Network Firewall policy + one flow-log path
Inbound Public ALB in every workload VPC Ingress VPC: ALB/WAF/Global Accelerator → TGW/PrivateLink
Governability Hard to audit SCP denies spoke IGW; single owner

Artifacts and decisions: the egress VPC and ingress VPC designs; the spoke 0.0.0.0/0-to-TGW route standard; the SCP denying AttachInternetGateway/CreateInternetGateway in workload accounts; the Network Firewall / appliance policy; the WAF, Global Accelerator, and Shield Advanced posture for ingress. The decision is centralize-everything vs. allow per-VPC NAT exceptions for a small set of high-volume VPCs where TGW processing cost would dominate.

Worked example: the packet path for centralized egress

Centralized egress lives or dies on four route entries; here is the whole path for a spoke pod reaching api.vendor.com:

  1. Spoke private subnet route table: 0.0.0.0/0 → tgw-…. The spoke has no internet gateway of its own (an SCP forbids it), so its only default path is the hub.
  2. Spoke’s TGW route table (rt-prod): 0.0.0.0/0 → egress-VPC attachment. The hub sends internet-bound traffic to the inspection/egress VPC.
  3. Egress VPC (in the Network account): the packet lands in a private subnet whose route table sends 0.0.0.0/0 to a NAT gateway, which sends it to the internet gateway. AWS Network Firewall sits inline before the NAT.
  4. Return path: the egress VPC’s TGW route table must carry routes back to every spoke CIDR (this is where a summarized IPAM supernet means one route, not fifty), and appliance mode on the egress/inspection attachment keeps the return flow pinned to the same firewall ENI so stateful inspection sees both directions.
# (1) in the spoke — the only exit is the hub
resource "aws_route" "spoke_default_to_tgw" {
  route_table_id         = aws_route_table.spoke_private.id
  destination_cidr_block = "0.0.0.0/0"
  transit_gateway_id     = aws_ec2_transit_gateway.hub.id
}

# the inspection/egress attachment MUST be appliance-mode for stateful symmetry
resource "aws_ec2_transit_gateway_vpc_attachment" "inspection" {
  transit_gateway_id     = aws_ec2_transit_gateway.hub.id
  vpc_id                 = aws_vpc.inspection.id
  subnet_ids             = [for s in aws_subnet.tgw_28 : s.id]  # one /28 per AZ
  appliance_mode_support = "enable"
  dns_support            = "enable"
  transit_gateway_default_route_table_association = false
  transit_gateway_default_route_table_propagation = false
  tags = { Name = "attach-inspection" }
}

Worked example: when centralized egress costs more

Centralization is not free — the hub taxes every byte. Compare one ordinary spoke against one 500 TB/month egress-heavy VPC (numbers are us-east-1 list, rounded, verify current pricing):

Cost component Rate Centralized (via hub) Per-VPC NAT (local)
TGW data processing ≈ $0.02/GB paid on every egress byte $0 (never crosses the hub)
NAT data processing ≈ $0.045/GB paid paid
NAT gateway hours ≈ $0.045/hr one shared set one set per VPC

For a normal VPC pushing a few TB, the TGW tax is noise and you keep it centralized for governance. For the 500 TB/month VPC, the TGW processing alone is 500,000 GB × $0.02 = ≈ $10,000/month extra, purely for routing through the hub. That is the crossover: allow a governed per-VPC NAT exception for the handful of ultra-high-throughput VPCs (still inspected locally with their own firewall), and keep everyone else centralized. The decision is a spreadsheet, not a religion.

Traffic inspection

What it is

Traffic inspection is the deep-packet / stateful firewalling and IDS/IPS layer that east-west (VPC-to-VPC), egress (VPC-to-internet), and ingress (internet-to-VPC) traffic passes through so it can be filtered, logged, and alerted on. In an AWS landing zone the inspection point is almost always an inspection VPC attached to the Transit Gateway, containing either AWS Network Firewall (a managed, Suricata-compatible stateful firewall) or a third-party NGFW (Palo Alto, Fortinet, Check Point) deployed behind a Gateway Load Balancer (GWLB) using the GENEVE protocol for transparent insertion. The TGW steers flows into the inspection VPC, the appliance inspects, and the TGW carries the flow onward.

Why it matters

Security groups and NACLs are L3/L4 stateful filters scoped to a VPC; they cannot do domain filtering, TLS-SNI inspection, intrusion detection, or centralized egress allow-listing, and they cannot inspect traffic between VPCs that ride the TGW. Compliance regimes (PCI DSS segmentation, regulated-data boundaries) routinely require an inspection layer that no application team can disable or route around. Putting inspection at the TGW makes it unavoidable: because spokes default-route through the hub and the hub’s route tables force the inspection VPC into the path, there is no spoke-side change that bypasses it.

How to do it well

Decide your inspection patterns explicitly — east-west, egress, and ingress are three different flows and you rarely inspect all three the same way. The cleanest design is a dedicated inspection VPC with the firewall, attached to the TGW, and a TGW routing scheme where the inspection VPC’s TGW route table holds the spoke CIDRs while spokes send everything (including inter-spoke) toward inspection. Enable TGW appliance mode on the inspection attachment — this is non-negotiable for stateful inspection, because it pins both directions of a flow to the same appliance across AZs and prevents the asymmetric routing that silently drops return packets. For Network Firewall, write stateful rule groups (Suricata rules and domain allow/deny lists), enable TLS inspection where policy requires decryption, and ship firewall and flow logs to the Log Archive account. For third-party NGFWs, front them with a GWLB and GWLB endpoints (GWLBe) so insertion is transparent and the appliance fleet auto-scales.

Inspection flow Path Typical control
East-west (spoke↔spoke) Spoke → TGW → inspection VPC → TGW → spoke Segmentation + IDS/IPS, often selective
Egress (spoke→internet) Spoke → TGW → egress/inspection VPC → NAT → IGW Domain allow-list, TLS-SNI, egress logging
Ingress (internet→spoke) IGW → ingress VPC (WAF/ALB) → GWLB/NFW → TGW → spoke WAF at L7 + IPS at the appliance

Artifacts and decisions: the inspection VPC design; the choice of AWS Network Firewall vs. third-party NGFW + GWLB; the appliance-mode setting; the stateful rule groups / firewall policy as code; the TLS-inspection scope; the firewall and flow-log destination in Log Archive; the explicit list of which flows (east-west / egress / ingress) are inspected. The major decision is AWS Network Firewall vs. a third-party NGFW — choose Network Firewall for an AWS-native, no-appliance-fleet operating model, and an NGFW + GWLB when you must reuse an incumbent vendor’s signatures, management plane, and SOC tooling.

Hybrid connectivity (Direct Connect)

What it is

AWS Direct Connect (DX) is a dedicated, private physical link between your data center (or colo) and AWS that does not traverse the public internet. You provision a DX connection (a 1/10/100 Gbps dedicated port at a DX location, or a sub-1G hosted connection from a partner), then carve virtual interfaces on it: a private VIF to reach VPCs, a transit VIF to reach a Direct Connect gateway (DXGW) associated with your Transit Gateway, and a public VIF to reach AWS public endpoints over private transport. In a landing zone the DX terminates in the Network account, attaches to the regional TGW via a DXGW + transit VIF, and is shared to spokes the same way every other TGW route is — so on-prem reachability is a hub property, not something each account sets up.

Why it matters

VPN over the internet is fine for a backup path but is throughput- and latency-variable and capped (a single IPsec tunnel tops out around 1.25 Gbps; ECMP across tunnels helps but the internet path remains best-effort). Enterprises with steady, high-volume hybrid traffic — data replication, large file transfer, latency-sensitive line-of-business apps, regulated data that must avoid the public internet — need DX’s dedicated bandwidth and consistent latency. The DXGW is what makes DX scale in a landing zone: one DXGW can associate to TGWs in multiple Regions and advertise on-prem prefixes to all of them (and your summarized AWS CIDR back to on-prem), so a single physical link serves a multi-Region, multi-account estate through the hub.

How to do it well

Never rely on a single DX. Provision two DX connections at two different DX locations (ideally on diverse carriers) for resiliency that meets AWS’s SLA tiers, and add a Site-to-Site VPN as a backup path over the TGW so a total DX failure fails over to the internet automatically (BGP prefers DX; VPN takes over when DX BGP drops). Terminate DX on a transit VIF → DXGW → TGW, not a private VIF to a single VPC, so every account benefits. On the DXGW, set allowed prefixes to the summarized AWS supernet you advertise to on-prem, and keep BGP advertisements summarized (this is where the IPAM plan pays off — a clean, contiguous AWS supernet means one or two prefixes to on-prem instead of dozens). Use a MACsec-capable port if data must be encrypted at the physical layer, or run a private-IP VPN over DX for IPsec. Monitor BGP session state and DX link health via CloudWatch and DX’s SLA metrics.

Hybrid option Transport Throughput Use as
Direct Connect (dedicated) Private physical port (1/10/100G) Line-rate, consistent Primary, high-volume / regulated
DX + transit VIF + DXGW Private, to TGW (multi-Region) Per DX Landing-zone-wide on-prem reach
Site-to-Site VPN (TGW) IPsec over internet ~1.25 Gbps/tunnel, ECMP Backup path / low-volume sites
DX + VPN over DX IPsec inside private DX Per DX, encrypted Encryption-required hybrid

Artifacts and decisions: the DX connection order(s) and location diversity; the transit VIF and DXGW configuration with allowed prefixes; the DXGW-to-TGW association(s); the backup VPN; the BGP/ASN and route-summarization plan; the encryption choice (MACsec or VPN-over-DX); the CloudWatch monitoring. The headline decision is resiliency tier — single DX (don’t), DX + VPN backup (common), or dual-DX at diverse locations + VPN (for SLA-bound, mission-critical hybrid).

IP address management (IPAM)

What it is

Amazon VPC IP Address Manager (IPAM) is the AWS service that plans, allocates, tracks, and audits the IP space for your whole organization. You create an IPAM (delegated to an IPAM admin account — typically the Network account) and build a pool hierarchy: a top-level pool (your private supernet, e.g. 10.0.0.0/8 or a slice of it), regional pools beneath it, and per-environment or per-OU pools beneath those. Accounts and VPCs draw CIDRs from the pools, and IPAM enforces non-overlap, tracks utilization, and flags non-compliant resources — across all accounts and Regions, integrated with AWS Organizations.

Why it matters

Overlapping CIDRs are the single most common, most painful landing-zone failure: the moment two VPCs (or a VPC and an on-prem range) overlap, Transit Gateway cannot route between them, peering breaks, and the only fix is to re-IP a live network — a migration measured in months. In a manual world (spreadsheets, tribal knowledge) overlap is inevitable as soon as account-vending hands CIDRs to dozens of teams. IPAM makes correct allocation the default: an account-vending pipeline calls IPAM, gets a guaranteed-non-overlapping CIDR from the right pool, and the supernet stays contiguous and summarizable — which is exactly what lets you advertise one or two clean prefixes over Direct Connect instead of a sprawling, unsummarizable mess.

How to do it well

Stand up IPAM before you vend the first workload account, delegated to the Network account, scoped to the organization. Design a pool hierarchy that mirrors your OU and Region structure so allocation encodes intent and stays summarizable:

IPAM (org-scoped, admin = Network account)
└─ top-level pool: 10.0.0.0/8           (entire private supernet)
   ├─ regional pool: 10.0.0.0/12  (us-east-1)
   │    ├─ prod pool:     10.0.0.0/14   → prod VPCs draw /20s
   │    ├─ nonprod pool:  10.4.0.0/14   → dev/test VPCs draw /20s
   │    └─ shared pool:   10.8.0.0/16   → shared services VPC
   └─ regional pool: 10.16.0.0/12 (eu-west-1)
        └─ ...

Set allocation rules on pools (allowed netmask lengths, required tags, locale) so teams can only draw correctly-sized, correctly-tagged CIDRs in the right Region. Wire IPAM into account vending (Account Factory for Terraform / Service Catalog) so a new account’s VPC CIDR comes from IPAM by allocation, never hand-typed. Use monitoring and the utilization dashboard to alert before a pool exhausts, and the compliance view to catch any VPC created with a CIDR outside the plan. Reserve a clean block for the TGW per-AZ /28 subnets and for future growth so summarization to on-prem never fragments.

IPAM construct Purpose Landing-zone use
IPAM (delegated admin) Org-wide IP authority Owned by Network account
Top-level / regional pools Hierarchical supernet split Mirror Regions for summarization
Environment/OU pools Per-domain allocation Prod/non-prod/shared isolation in IP space
Allocation rules Enforce netmask/tags/locale Stop wrong-size, wrong-Region CIDRs
Utilization + compliance views Track & audit Pre-exhaustion alerts; find off-plan VPCs

Artifacts and decisions: the IPAM instance and delegated-admin assignment; the full pool hierarchy with CIDRs; the per-pool allocation rules; the integration into account vending; the reserved blocks (TGW subnets, growth); the utilization alerting. The foundational decision is the supernet and its split — the contiguous private block you own and how it carves by Region and environment, because that single choice determines whether DX advertisements stay summarized for the life of the platform.

Worked example: allocating a VPC from IPAM (and the subnet math)

The pool tree above is the plan; here is the allocation an account-vending pipeline actually performs. It asks IPAM for a right-sized block from the correct pool and gets back a guaranteed-non-overlapping CIDR:

# Account Factory / Service Catalog calls this instead of hand-typing a CIDR
aws ec2 allocate-ipam-pool-cidr \
  --ipam-pool-id ipam-pool-0abcd1234EXAMPLE \
  --netmask-length 20
# → returns e.g. 10.32.16.0/20, recorded and reserved org-wide

Now size the VPC. A /20 is 4,096 addresses (2^(32−20) = 2^12). Carve it into tiers — public, private-app, private-data — and reserve a tiny /28 (16 addresses) per Availability Zone for the TGW attachment ENIs. Remember AWS reserves 5 addresses per subnet (network, VPC router, DNS, future use, broadcast), so a /28 gives 11 usable — plenty for TGW ENIs, useless for workloads. The pool arithmetic scales cleanly too: a /14 pool holds 64 × /20 VPCs (2^(20−14) = 2^6), so a per-environment /14 comfortably vends dozens of accounts before you widen it.

RFC 1918 and the strategy that keeps you summarizable

Your entire private supernet must come from RFC 1918 space, and the choice among the three ranges is not cosmetic:

RFC 1918 block Size Landing-zone use
10.0.0.0/8 16,777,216 addrs The only range big enough for a large multi-Region estate — carve everything here
172.16.0.0/12 1,048,576 addrs Fine for small estates; watch the 172.17.0.0/16 Docker-bridge default collision
192.168.0.0/16 65,536 addrs Too small for a hub; leave it for labs and home/branch offices

Three rules keep the plan summarizable for the life of the platform — which is what lets Direct Connect advertise one prefix to on-prem instead of dozens:

  1. Own a single contiguous supernet (e.g. 10.32.0.0/11) and never let an allocation land outside it. Deconflict it against every data-center range and likely acquisitions before you claim it.
  2. Carve by Region, then environment, so each Region is one aggregate (10.32.0.0/12) and each environment a sub-aggregate — the hierarchy is the summarization.
  3. Reserve growth headroom and the TGW /28 blocks up front, so widening a pool later never forces a fragmented, unsummarizable second prefix. IPAM’s allocation rules (netmask bounds, required tags, locale) enforce all of this automatically; for the deeper mechanics — pools, BYOIP, and utilization monitoring at scale — see the VPC IPAM CIDR management lesson.

Real-world enterprise scenario

Meridian Logistics is a freight and supply-chain company running 9 data centers and a target AWS estate of ~70 accounts across us-east-1 (primary) and eu-west-1 (EU data residency). They have already deployed AWS Control Tower with OUs for Workloads-Prod, Workloads-NonProd, Shared-Services, Infrastructure, and the Control Tower Security OU holding Log Archive and Audit. The platform (network) team owns a dedicated Network account under the Infrastructure OU. They are now executing the Network Architecture phase.

Outcome. Meridian onboards a new workload account end-to-end (account vended, VPC CIDR allocated from IPAM, TGW-attached to the right route table, egress + DNS + on-prem reachability inherited) in under 30 minutes, down from a multi-day networking ticket. Interface-endpoint consolidation removed ~60 duplicate endpoint-AZ charges per environment. A PCI segmentation audit passed on the strength of the unavoidable inspection path and the prod/non-prod TGW isolation. And in 18 months of growth across 70 accounts, they have logged zero CIDR-overlap incidents and zero re-IP projects — the metric that, more than any other, proves the network foundation was right.

Going deeper

Association vs. propagation — the two verbs that confuse everyone

A TGW attachment relates to a route table in exactly two independent ways, and mixing them up is the most common landing-zone routing bug:

Segmentation falls out of choosing these deliberately: prod attachments associate to rt-prod (so they route only where rt-prod allows) while shared services propagates into rt-prod (so prod can find it). Turn default association and propagation off on the TGW (as the Terraform earlier does) or every new attachment silently joins one flat table and your segmentation evaporates.

Appliance mode, precisely

Without appliance mode, the TGW load-balances flows across appliance ENIs per Availability Zone using a flow hash, and — critically — the forward and return directions of the same connection can hash to appliances in different AZs. A stateful firewall that sees only one direction of a TCP flow treats the other direction as an unsolicited packet and drops it, so connections fail intermittently (only when the hash splits them) — the worst kind of bug to reproduce. Appliance mode switches the algorithm to a symmetric 5-tuple hash so both directions always land on the same appliance ENI. It is a correctness setting for any stateful-inspection or NAT-in-path VPC, not a performance knob. Enable it on the inspection and centralized-egress attachments; you do not need it on ordinary spoke attachments.

Cloud WAN — the managed alternative to a self-managed TGW mesh

Past a handful of Regions, wiring TGW-to-TGW peering by hand (remember: 50 peerings per TGW, and each Region’s route tables maintained separately) becomes the bottleneck. AWS Cloud WAN (GA since late 2022) is AWS’s managed answer: you write a single core network policy — a versioned JSON document — that declares segments (the global equivalent of TGW route tables, e.g. prod, nonprod, shared), core network edges in each Region (AWS builds and maintains the inter-Region connectivity for you), and attachment policies that map attachments to segments by tag, automatically. A new VPC tagged segment=prod joins the prod segment in every Region with no manual route-table plumbing.

Self-managed TGW mesh AWS Cloud WAN
Inter-Region connectivity You create/maintain TGW peerings Managed by AWS between core network edges
Segmentation model One route table per domain, per Region One global segment per domain (policy-defined)
Attachment onboarding Explicit association/propagation Tag-based automatic association
Config surface Terraform/CFN per Region One central versioned policy document
Best when 1–3 Regions, granular control, cost-sensitive Many Regions/segments, want central policy + automation

Cloud WAN can coexist with TGW (you can peer a TGW into a Cloud WAN core network during migration), so the usual path is “start on TGW, adopt Cloud WAN when the Region/segment count makes the mesh painful.” Model its per-edge and processing charges before committing — the managed convenience is not free.

RAM beyond the Transit Gateway — shared subnets and managed prefix lists

RAM shares far more than the TGW. Two under-used shares tighten a landing zone:

resource "aws_ec2_managed_prefix_list" "onprem" {
  name           = "corp-onprem-supernet"
  address_family = "IPv4"
  max_entries    = 20
  entry {
    cidr        = "10.0.0.0/12"
    description = "all data-center ranges (summarized)"
  }
}

Where VPC Lattice fits (and where it doesn’t)

Teams sometimes ask whether VPC Lattice replaces the TGW. It does not — they operate at different layers. The TGW is L3: it moves packets between VPCs and on-prem. VPC Lattice is an application-layer (L7) service-to-service fabric with its own IAM-based auth and path/header routing, sitting above the network. Use the TGW to build the reachable, segmented foundation; reach for Lattice when you additionally want fine-grained, identity-aware service connectivity that doesn’t care about CIDRs. They compose; they don’t compete.

The IAM and Organizations plumbing that makes sharing work

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "DenySideDoorInternetEgress",
    "Effect": "Deny",
    "Action": [
      "ec2:CreateInternetGateway",
      "ec2:AttachInternetGateway",
      "ec2:CreateEgressOnlyInternetGateway"
    ],
    "Resource": "*"
  }]
}

Attach it to the Workload OUs, not the whole org, so the Network account can still build the one sanctioned internet gateway in the egress VPC.

Failure modes to design against

Failure mode Symptom Design defense
Asymmetric routing (no appliance mode) Intermittent drops through the firewall appliance_mode_support = "enable" on inspection/egress attachments
Blackhole too broad A whole environment loses reachability after a route change Blackhole specific supernets; review route changes as code (PR + plan)
IPAM pool exhaustion Account vending fails; teams hand-pick CIDRs “just this once” Utilization alarms well below 100%; growth headroom reserved
Overlapping CIDR from acquisition New company’s 10.x clashes; TGW can’t route both Reserve deconflicted space; bridge via NAT/PrivateLink until re-IP
Resolver endpoint saturation DNS timeouts under load (≈ 10,000 QPS per endpoint IP) Size endpoints across AZs; monitor query volume
RAM share unaccepted Spoke can’t see the TGW/rule/PHZ Enable org sharing (auto-accept); alarm on pending shares
Single DX with no backup Total hybrid outage on one link failure Dual-DX at diverse locations and/or a VPN backup over the TGW

For the deep mechanics of the egress firewall itself — stateful rule groups, domain lists, TLS inspection, and logging — see the Network Firewall centralized inspection lesson.

Deliverables & checklist

Common pitfalls

Common beginner mistakes

These are conceptual misreadings — distinct from the operational traps in Common pitfalls above. Each is a belief that quietly produces a broken or ungoverned network.

Practice challenges

Work these in order; they escalate from beginner to advanced. Each solution ends with a one-line why.

1. (Beginner) The mesh explosion. Your estate will have 40 VPCs that must all reach each other. How many VPC peerings would a full mesh need, and how many TGW attachments replaces it?

<details><summary>Solution</summary>

Full mesh = n(n−1)/2 = 40 × 39 / 2 = 780 peerings (each needing route-table entries on both sides). Hub-and-spoke needs 40 attachments (one per VPC) plus a few route tables.

Why: this n² → n collapse is the entire reason a landing zone uses a Transit Gateway instead of peering. </details>

2. (Beginner) Pool arithmetic. A per-environment IPAM pool is a /14. If every VPC is vended as a /20, how many VPCs does the pool hold? And how many usable addresses are in the /28 TGW subnet you reserve per AZ?

<details><summary>Solution</summary>

/14 → /20 is 2^(20−14) = 2^6 = 64 VPCs. A /28 has 16 addresses; AWS reserves 5 per subnet, leaving 11 usable — fine for TGW ENIs, useless for workloads.

Why: pool and subnet sizing is the arithmetic that keeps the supernet summarizable and non-overlapping. </details>

3. (Intermediate) The two route entries for centralized egress. A spoke has no internet gateway. Give the route (destination + target) that (a) the spoke’s private subnet route table needs, and (b) the spoke’s TGW route table needs, to send internet-bound traffic through the central egress VPC.

<details><summary>Solution</summary>

(a) Spoke subnet route table: 0.0.0.0/0 → tgw-… (the TGW). (b) Spoke’s TGW route table: 0.0.0.0/0 → <egress-VPC attachment>. The egress VPC then routes 0.0.0.0/0 to its NAT gateway → IGW, and its own TGW route table carries the spoke CIDRs back.

Why: centralized egress is just a default route pointed at the hub, plus return routes to each spoke — governance from four entries. </details>

4. (Intermediate) Design the segmentation. Prod must never reach non-prod; both must reach shared services; shared services must reach both. Using rt-prod, rt-nonprod, rt-shared, state each attachment’s association and propagations.

<details><summary>Solution</summary>

Why: segmentation is produced by choosing associations and propagations, not by security groups. </details>

5. (Advanced) Why the firewall flaps. A stateful firewall in the inspection VPC drops ~30% of connections at random; ping and short curls sometimes work. The attachment is not in appliance mode. Explain the mechanism and give the one-line fix.

<details><summary>Solution</summary>

Without appliance mode the TGW hashes forward and return directions independently and can pin them to firewall ENIs in different AZs; the appliance sees a half-flow and drops it, so failures are intermittent (only when the hash splits). Fix: appliance_mode_support = "enable" on the inspection attachment.

Why: appliance mode forces symmetric 5-tuple hashing — a correctness requirement for stateful inspection, not a tuning knob. </details>

6. (Advanced) The cost crossover. A single VPC egresses 500 TB/month to the internet. Estimate the extra monthly cost of sending it through centralized egress versus a local per-VPC NAT, counting only TGW data processing (≈ $0.02/GB). What do you recommend?

<details><summary>Solution</summary>

500 TB = 500,000 GB. TGW processing = 500,000 × $0.02 = ≈ $10,000/month extra that a local NAT path avoids (NAT per-GB and IGW costs are paid either way). Recommend a governed per-VPC NAT exception for this VPC — still inspected locally — and keep the rest of the estate centralized.

Why: TGW data processing taxes every hub-crossing byte, so a few ultra-high-throughput VPCs can invert the “centralize everything” default. </details>

Glossary

What’s next

Part 6 of AWS Landing Zone & Control Tower moves from the network to Security & Logging Architecture — centralized CloudTrail and Config in the Log Archive account, GuardDuty, Security Hub, and Macie delegated to the Audit account, and the detective controls that watch everything the network now carries.

AWSLanding ZoneNetwork ArchitectureEnterprise
Need this built for real?

Vinod is a Senior Cloud Architect (22+ yrs) — available for Azure / AWS / GCP architecture, landing zones, and migrations.

Work with me

Comments