Capstone: A Production Multi-Cloud GitOps Platform Across AKS, EKS & GKE

In a nutshell

This capstone takes everything the course taught — Applications, Helm and Kustomize sources, ApplicationSets, Rollouts, projects, RBAC, secrets, DR — and assembles it into one working platform that runs a single company’s apps across three clouds (Azure’s AKS, AWS’s EKS and Google’s GKE) at the same time, from one Git repository, operated by one small team. It is not a new topic; it is the proof that the topics fit together.

The mental model is an air-traffic control tower. One tower (the Argo CD hub, on its own management cluster) sequences planes landing at three airports run by three different authorities (the three clouds). The tower doesn’t fly the planes — so if the tower goes dark, every plane already in the air lands safely; you just can’t schedule new departures until it’s back. That one idea — the control plane sits off the data path — is why this platform can lose its hub and not lose its apps, and it runs through every phase below.

You’ll follow a fictional health company, Aurora, whose data-residency laws force three clouds, and build the platform in eight phases: Terraform lays the substrate, one HA hub takes over, ApplicationSets fan a platform layer across all three clouds, secrets flow from each cloud’s own store, app teams self-serve canary deploys inside guardrails, fleet SLOs and Slack alerts close the loop, and a DR runbook rebuilds the whole control plane from Git.

Level: Specialist · Time: ~61 min

Prerequisites: be comfortable with everything the earlier lessons cover — this one composes them rather than re-teaching them, and each phase links back to the lesson it draws on. After this you can: choose a multi-cloud Argo CD topology and defend it; draw the Terraform→Argo CD hand-off; fan a platform layer across clouds with one ApplicationSet; bound self-service with AppProjects, RBAC and Kyverno; and write a DR runbook that survives losing the hub.

---

Every lesson in this course built one part. You learned the pull model and what Argo CD is and is not, wrote a first Application, sourced it from Helm and Kustomize, registered a second cluster, templated dozens of apps with an ApplicationSet, gated releases with Argo Rollouts, locked the whole thing down with projects, RBAC and hardening, and learned to rebuild it all from Git after a disaster. This lesson is where every one of those parts snaps together into the single machine the course promised on day one: a production GitOps platform that runs one organization’s applications across AKS, EKS and GKE at the same time, from one Git source of truth, operated by one small platform team.

This is a capstone, so it is deliberately different from every lesson before it. We will not re-teach ExternalSecrets or the Rollout CRD field by field — you know those, and each phase links back to the lesson that owns it. Instead we synthesize: we take a concrete organization with real constraints, make the architectural decisions in the order you would actually make them, and stand the platform up in eight phases, showing the one critical manifest that anchors each phase and the reasoning that ties it to the rest. By the end you will have a mental model complete enough to walk into a design interview, sketch this platform on a whiteboard, and defend every decision — and a build runbook concrete enough to actually execute.

The honesty note this whole course runs on. There is no live three-cloud fleet attached to this lesson, so every command output below is labelled representative — it shows the shape you will see, not a transcript of a specific run. Every manifest is schema-correct for Argo CD 2.13+/3.x on Kubernetes 1.29+, with real fields, real ports and real API versions; every Terraform block uses real provider resources. Secrets are always placeholders. The one thing you must supply is three real clusters and the willingness to run the phases in order.

---

Why this matters

Most teams reach multi-cloud by accident, not by design. An acquisition brings an AWS estate into an Azure shop; a data-residency law forces one region onto a cloud you do not otherwise use; a big customer demands their workload run on their cloud. Suddenly you are operating three control planes, three ways to deploy, three on-call runbooks, and three copies of every mistake. The promise of GitOps — and the reason this course spent thirty-some lessons getting here — is that the deployment plane can be unified even when the infrastructure plane cannot. Argo CD does not care whether a target cluster is AKS, EKS or GKE; it reconciles Git into a Kubernetes API server, and all three expose the same API. The clouds differ only at the edges — identity, ingress, secrets, registries, load balancers — and this course has taught you exactly those edges, one at a time. The capstone is proving you can hold all of them in one design without the seams showing.

The stakes are concentration. A unified platform is a single pane of glass and a single point of leverage: one commit ships to three clouds, one platform team supports fifty app teams, one dashboard shows the health of the entire estate. That same concentration is a single blast radius — one bad ApplicationSet edit can break AKS, EKS and GKE in the same reconcile tick, and one compromised hub is cluster-admin on your entire fleet. Every serious decision in this lesson is really a decision about where to concentrate power and where to bound it: the topology that puts risk where you can live with it, the guardrails that keep self-service from becoming self-destruction, the DR plan that survives losing the hub itself.

The mental model to carry through all eight phases is a layer cake. Terraform owns the substrate (clusters, identities, networking) — the things that must exist before Argo CD can run. Argo CD owns everything on top (the platform layer, the apps, the policies) — the things that should be reconciled from Git, not clicked into existence. The line between them is the handoff, and getting that line in the right place is the difference between a platform you can rebuild in an afternoon and one nobody dares touch. We draw that line explicitly in Phase 1 and never cross it again.

---

The organization and its requirements

Meet Aurora Health, a fictional but plausible global digital-health SaaS. Aurora stores and processes patient data, which means the law decides where its data may live — and the law does not care about Aurora’s cloud preferences. Three regulatory regimes force three clouds:

Aurora cannot consolidate onto one cloud without breaking the law in two regions. Multi-cloud is not a preference here; it is a hard requirement, which is exactly the situation where a unified GitOps plane pays for itself. The organizational shape is the other half of the problem:

Four app teams today, a plan for twenty, and one platform team that must not become a ticket queue. That last constraint — self-service — shapes more of the design than any other. If onboarding a new app team means a platform engineer hand-writing Applications, the platform does not scale past its founders. The non-functional requirements make the constraints explicit and testable:

Hold these six requirements in mind — every architectural choice below is traceable to one of them, and the interview questions at the end are mostly “which requirement drove which decision.”

---

The architecture: hub-and-spoke across three clouds

The first and most consequential decision is topology: where does Argo CD run, and how many of it are there? This course covered the trade-off in depth in One Hub, Many Clouds; here we make the call for Aurora and move on.

Aurora runs hub-and-spoke: a small, dedicated management cluster hosts one highly-available Argo CD hub, and that hub manages three spokes — the AKS, EKS and GKE clusters that run the actual apps. We run the management cluster as an AKS cluster in a locked-down Azure management subscription, but note carefully: the hub is cloud-agnostic — it could equally be EKS or GKE. We put it on Azure because Aurora’s primary entity is there and it keeps the platform team’s blast-radius auditing in one cloud’s IAM; nothing about the design depends on that choice. The management cluster runs only platform tooling (Argo CD, its Redis HA, backups) and no Aurora application — a dedicated management cluster is the cleanest way to keep the control plane’s failure domain separate from the workloads it controls, and it is what makes the DR story in Phase 8 clean.

Here is the whole platform in one picture. Read it left to right: the platform monorepo and team repos on the left are the source of truth; the HA Argo CD hub and its ApplicationSets generate the fleet; an identity-and-policy gate decides who and what may deploy; the three cloud spokes each run the same platform stack behind their own cloud edge; and observability plus DR wrap the whole thing.

The five zones map exactly to how the platform is built and operated. Source is Git — the only place desired state lives. Mgmt Hub is the single control plane. Identity + Policy is the gate that turns “the hub can deploy anything anywhere” into “each team deploys its own thing, to its own place, only if it is signed.” Fleet Spokes are the three clouds, each running an identical platform stack behind a different edge. Operate is how you know it is working and how you get it back when it is not. The eight build phases below walk this diagram from left to right.

The phases build these layers bottom-up: substrate first (Phase 1), then the control plane and its identity (Phases 1–2), then the platform layer and its secrets (Phases 3–4), then the apps and their guardrails (Phases 5–6), then operations and survival (Phases 7–8).

---

The repository structure: platform monorepo + team repos

Before any cluster exists, decide where desired state lives — because everything else points at it. This course weighed monorepo against polyrepo at scale; Aurora uses a hybrid that gets the best of both, and it is the layout most mature platforms converge on.

platform-gitops/                     # ONE monorepo — Platform Engineering owns it
  bootstrap/
    root-app.yaml                    # the single hand-applied Application
    project-platform.yaml            # the AppProject the root app runs under
  appprojects/                       # one AppProject per app team
    team-records.yaml
    team-scheduling.yaml
    ...
  applicationsets/
    platform-ingress.yaml            # cluster generator → ingress on every spoke
    platform-observability.yaml      # Prometheus/Grafana on every spoke
    platform-eso.yaml                # External Secrets Operator on every spoke
    platform-rollouts.yaml           # Argo Rollouts controller on every spoke
    apps-records.yaml                # matrix: records' apps × its clusters
    ...
  platform/                          # the platform layer's charts/values
    ingress/{values.yaml,values-aks.yaml,values-eks.yaml,values-gke.yaml}
    observability/ ...
  policies/                          # Kyverno ClusterPolicies, NetworkPolicies

records-config/                      # ONE repo per app team — the team owns it
  base/                              # Helm chart ref or Kustomize base
  overlays/
    dev/       { values.yaml, rollout patch }
    staging/   { values.yaml }
    prod/      { values.yaml }       # prod = the correct cloud, chosen by label

The division of ownership is the whole point:

The rule that keeps this honest: the platform monorepo can reference a team’s config repo, but a team’s config repo can only ever affect that team. An app team merging a PR in records-config changes only records apps, on only the clusters its AppProject allows, because the app-delivery ApplicationSet and the team-records AppProject bound it. A platform engineer merging in platform-gitops can change the whole fleet — which is why that repo has the strictest review. Monorepo where blast radius is high and changes are rare; polyrepo where changes are frequent and must be isolated. This hybrid is not a compromise; it is the correct shape.

Never point targetRevision at a moving branch in prod. Team config repos promote by changing a pinned tag or SHA (see the promotion section); the platform monorepo’s ApplicationSets read main, but every app’s prod overlay pins an immutable image tag. A moving branch means any merge, anywhere, can roll to prod unreviewed — the opposite of auditability.

---

Phase 1 — Bootstrap: Terraform provisions the substrate

Goal: From nothing, end with four clusters (mgmt + three spokes), a per-cloud identity the hub can assume, HA Argo CD installed on the mgmt cluster, and the one root app applied. This phase is pure cluster bootstrapping and the Terraform → Argo CD handoff, done four times and then handed off exactly once.

The layer-cake line from Why this matters becomes concrete here. Terraform creates everything up to and including the Argo CD install and the root Application; the moment the root app is applied, Terraform stops and Argo CD takes over. Terraform must never manage an Argo CD Application other than the root — if it did, you would have two controllers fighting over the same resources.

The Terraform is ordinary; the two things that make it a platform bootstrap are that it (a) installs Argo CD via helm_release in HA mode and (b) applies the root app as its final act. Representative root-module shape (abbreviated — real modules are larger):

# platform-terraform/main.tf  (abbreviated, representative)
module "mgmt" {                              # the management cluster (AKS)
  source              = "./modules/aks"
  name                = "aurora-mgmt"
  location            = "westeurope"
  kubernetes_version  = "1.30"
  workload_identity   = true                 # hub authenticates to spokes via WI
}

module "spoke_aks" { source = "./modules/aks"; name = "aurora-prod-aks"; location = "westeurope" }
module "spoke_eks" { source = "./modules/eks"; name = "aurora-prod-eks"; region   = "us-east-1" }
module "spoke_gke" { source = "./modules/gke"; name = "aurora-prod-gke"; location = "asia-south1" }

# Install HA Argo CD on the mgmt cluster, then hand off.
resource "helm_release" "argocd" {
  name             = "argocd"
  namespace        = "argocd"
  create_namespace = true
  repository       = "https://argoproj.github.io/argo-helm"
  chart            = "argo-cd"
  version          = "7.7.0"                  # chart version → Argo CD 2.13.x
  values           = [file("${path.module}/values/argocd-ha.yaml")]
  depends_on       = [module.mgmt]
}

# The ONE hand-off: apply the root app-of-apps. Terraform stops here.
resource "kubectl_manifest" "root_app" {
  yaml_body  = file("${path.module}/../platform-gitops/bootstrap/root-app.yaml")
  depends_on = [helm_release.argocd]
}

The argocd-ha.yaml values file is what makes the hub survive a node loss — the installation lesson covered it; the essentials are redundant application-controller shards, ≥2 replicas of server and repo-server, and Redis in HA mode:

# values/argocd-ha.yaml  (representative essentials)
redis-ha: { enabled: true }                  # 3-node Redis, not the single-pod cache
controller: { replicas: 1 }                  # shard count; scale with fleet size
server:      { replicas: 2 }
repoServer:  { replicas: 2, resources: { limits: { memory: 2Gi } } }
applicationSet: { replicas: 2 }
configs:
  cm:
    application.instanceLabelKey: argocd.argoproj.io/instance
    timeout.reconciliation: 180s

And the root Application — the single most important manifest in the whole platform, because it is the only thing applied by hand and everything else descends from it. It points at bootstrap/, a directory of child Applications (or ApplicationSets), giving you the app-of-apps pattern as the bootstrap mechanism:

# platform-gitops/bootstrap/root-app.yaml  — the one hand-applied manifest
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: root
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io   # cascade-delete children on removal
spec:
  project: platform
  source:
    repoURL: https://github.com/aurora-health/platform-gitops.git
    targetRevision: main
    path: bootstrap                              # a dir of child apps/appsets
    directory:
      recurse: true
  destination:
    server: https://kubernetes.default.svc       # the mgmt cluster itself
    namespace: argocd
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

What just happened. Terraform built four clusters and installed an HA Argo CD on the management cluster, then applied one Application named root. That root app now syncs everything else in bootstrap/ — the AppProjects, the ApplicationSets, the platform layer — with no further human kubectl apply. From this point on, the way you change the platform is to commit to platform-gitops and let the hub reconcile. Terraform’s job is done until you need a new cluster.

The bootstrap chicken-and-egg. The root app references repos and clusters that must be reachable before the first sync. Order matters: Terraform creates the spoke identities and writes the cluster Secrets (Phase 2) into the mgmt cluster in the same apply, so that when the root app fans out the platform layer, the hub can already authenticate to all three spokes. Apply the root app before the cluster Secrets exist and every spoke Application shows Unknown — the classic first-boot failure.

---

Phase 2 — Cluster registration and identity

Goal: The hub can authenticate to all three spokes, each spoke is labelled so ApplicationSets can target it, and humans log in through SSO with their groups mapped to Argo CD roles. This is multi-cluster registration, SSO and RBAC composed into one identity story.

A spoke is registered by creating a Secret in the hub’s argocd namespace, labelled argocd.argoproj.io/secret-type: cluster. The credential inside is where the clouds diverge: the hub must call three different API servers, each with that cloud’s own auth. The clean way is exec auth — the cluster Secret’s config carries an execProviderConfig that runs the cloud’s token binary at reconcile time, so the hub holds no long-lived kubeconfig.

The EKS spoke’s cluster Secret, showing the real execProviderConfig shape (AKS and GKE differ only in the command/args):

# platform-gitops/bootstrap/clusters/prod-eks.yaml  (written by Terraform; creds via ESO/Sealed Secrets in Git)
apiVersion: v1
kind: Secret
metadata:
  name: prod-eks-us
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: cluster
    cloud: eks                      # ← the label that routes the fleet
    env: prod
    region: us-east-1
    residency: us
type: Opaque
stringData:
  name: prod-eks-us
  server: https://XXXXXXXX.gr7.us-east-1.eks.amazonaws.com
  config: |
    {
      "execProviderConfig": {
        "apiVersion": "client.authentication.k8s.io/v1beta1",
        "command": "aws",
        "args": ["--region", "us-east-1", "eks", "get-token", "--cluster-name", "aurora-prod-eks"],
        "env": { "AWS_ROLE_ARN": "arn:aws:iam::111122223333:role/argocd-hub" }
      },
      "tlsClientConfig": { "insecure": false, "caData": "<base64 cluster CA>" }
    }

The labels are not decoration — they are the organizing principle of the entire fleet. Every ApplicationSet from Phase 3 onward selects on them. Aurora’s label schema:

Get one label wrong and an app either deploys to the wrong cloud (a residency violation) or nowhere at all (an empty generator, silently). Labelling at registration is the single cheapest investment that makes fleet-wide targeting declarative.

Humans authenticate through one SSO configured in argocd-cm, federating each cloud’s identity provider so an engineer uses their existing corporate login. The SSO providers differ per cloud but land in one Argo CD:

RBAC in argocd-rbac-cm maps those groups to project-scoped roles — the platform team is admin, every app team gets sync/get on only its own project, and nobody else can do anything:

# argocd-rbac-cm  (representative)
policy.default: role:readonly
policy.csv: |
  g, aurora-health:platform-engineers, role:admin
  p, role:records-dev, applications, get,  team-records/*, allow
  p, role:records-dev, applications, sync, team-records/*, allow
  g, aurora-health:records-engineers, role:records-dev
scopes: '[groups]'

What just happened. The hub can now reach all three spokes — each with its own cloud’s auth, no static kubeconfig — every spoke is labelled with the four dimensions the fleet routes on, and every human logs in once and sees exactly the projects their SSO group allows. Crucially, app teams get sync but not create/delete on Applications: those come from ApplicationSets the platform owns, so a team can promote its own app but cannot conjure a new one outside the guardrails.

---

Phase 3 — The platform layer via ApplicationSets

Goal: Every spoke runs the same baseline platform stack — ingress controller, cert-manager, External Secrets Operator, Prometheus/Grafana, and the Argo Rollouts controller — installed and kept in lockstep by ApplicationSets, with the per-cloud differences expressed as a single goTemplate branch, not three copies.

This is the phase where the fleet stops being three clusters and becomes one platform. Each component is deployed by a platform ApplicationSet whose clusters generator loops over every spoke labelled env=prod and stamps one Application per cluster. Add a spoke Secret with the right labels and the entire platform stack appears on it automatically.

The critical manifest is the platform ingress ApplicationSet, because it shows the whole per-cloud pattern in one place: a cluster generator, a RollingSync strategy that gates the blast radius cloud-by-cloud, and a goTemplate branch that selects a per-cloud values file. This is the manifest to internalize — every other platform ApplicationSet is a variation of it.

# platform-gitops/applicationsets/platform-ingress.yaml
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: platform-ingress
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]        # a typo fails the render, not silently blanks it
  generators:
    - clusters:
        selector:
          matchLabels:
            env: prod                              # every prod spoke, any cloud
  strategy:
    type: RollingSync                              # ← gate the blast radius
    rollingSync:
      steps:
        - matchExpressions: [{ key: cloud, operator: In, values: [aks] }]   # AKS first
        - matchExpressions: [{ key: cloud, operator: In, values: [eks] }]   # then EKS
        - matchExpressions: [{ key: cloud, operator: In, values: [gke] }]   # then GKE
  template:
    metadata:
      name: 'ingress-{{.name}}'
      labels:
        cloud: '{{index .metadata.labels "cloud"}}'   # RollingSync steps match on THIS label
    spec:
      project: platform
      source:
        repoURL: https://github.com/aurora-health/platform-gitops.git
        targetRevision: main
        path: platform/ingress
        helm:
          valueFiles:
            - values.yaml                                        # shared defaults
            - 'values-{{index .metadata.labels "cloud"}}.yaml'   # ← the per-cloud branch
      destination:
        server: '{{.server}}'
        namespace: ingress
      syncPolicy:
        automated: { prune: true, selfHeal: true }
        syncOptions: [CreateNamespace=true]

The per-cloud branch resolves to one of three values files, each setting the ingress class its cloud understands — one app definition, three correct edges:

# platform/ingress/values-eks.yaml  (the whole per-cloud delta — a few lines, not a copy)
ingress:
  className: alb
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip

The RollingSync strategy is the single most important safety feature in this phase and deserves a hard look. Without it, editing the platform ingress chart reconciles to AKS, EKS and GKE on the same tick — a mistake breaks all three clouds at once. With it, the change rolls to AKS first; only when those Applications are Healthy does EKS proceed, then GKE. A bad change is caught after the first cloud, not the last. The steps match on the cloud label the template stamps onto each generated Application — that wiring (template label ↔ rollingSync matchExpressions) is easy to get wrong and produces a rollout that never progresses if the labels do not line up.

What just happened. One ApplicationSet per platform component now maintains that component on every spoke, forever. Register cluster number four and the full stack lands on it with no new manifests. The per-cloud differences live in tiny values files keyed by the cloud label, and every fleet-wide platform change rolls out cloud-by-cloud instead of all at once.

The blast-radius rule for the whole platform layer. Every platform ApplicationSet gets RollingSync and missingkey=error, and every one sets applicationsSync: create-update (never create-delete) plus preserveResourcesOnDeletion: true on the destructive ones, so that narrowing a selector can never prune a running platform component off the fleet. The convenience of “one object drives everything” is exactly why one object must be handled with this much care.

---

Phase 4 — Secrets: External Secrets Operator per cloud

Goal: No plaintext secret ever touches Git or a manifest. Each spoke’s ESO (installed in Phase 3) pulls secrets from its own cloud’s secret store, using that cloud’s workload identity. This composes the secrets lesson with the three per-cloud identity models.

The pattern is uniform — a SecretStore per namespace declares the provider and how to authenticate; an ExternalSecret declares which remote keys to materialize into a native K8s Secret — but the provider block is the cloud edge. The identity is always the cloud’s workload identity, never a static credential, so nothing long-lived lives in the cluster.

The three SecretStore manifests, side by side — this is the one place worth showing all three, because the provider block is precisely the multi-cloud seam:

# AKS — Azure Key Vault
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata: { name: cloud-secrets, namespace: records }
spec:
  provider:
    azurekv:
      authType: WorkloadIdentity
      vaultUrl: "https://aurora-eu-kv.vault.azure.net"
      serviceAccountRef: { name: eso-records }
---
# EKS — AWS Secrets Manager
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata: { name: cloud-secrets, namespace: records }
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef: { name: eso-records }   # IRSA-annotated SA
---
# GKE — Google Secret Manager
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata: { name: cloud-secrets, namespace: records }
spec:
  provider:
    gcpsm:
      projectID: aurora-prod-apac
      auth:
        workloadIdentity:
          clusterLocation: asia-south1
          clusterName: aurora-prod-gke
          serviceAccountRef: { name: eso-records }

Because the SecretStore is named identically (cloud-secrets) in every cluster, the ExternalSecret that consumes it is cloud-neutral — the app team writes it once and it works on all three clouds, since only the store’s provider block differs:

# records-config/base/external-secret.yaml  — identical on AKS, EKS and GKE
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata: { name: records-db, namespace: records }
spec:
  refreshInterval: 1h
  secretStoreRef: { name: cloud-secrets, kind: SecretStore }
  target: { name: records-db, creationPolicy: Owner }
  data:
    - secretKey: password
      remoteRef: { key: records-db-password }

This is the payoff of naming the store consistently: the platform absorbs the per-cloud difference (in the SecretStore the platform ApplicationSet deploys), so the app team writes one portable ExternalSecret. The seam is hidden exactly where it should be.

What just happened. Every app on every cloud now gets its secrets injected from that cloud’s native store, authenticated by workload identity, with zero plaintext in Git. The app team’s manifest is identical across clouds; the platform hid the seam in the SecretStore. And because the cluster Secrets themselves (Phase 2) are the only credentials not derivable from a cloud store, they are the one thing the DR plan in Phase 8 must back up.

---

Phase 5 — Application delivery and progressive rollout

Goal: App teams ship to the fleet by merging a PR in their own config repo, and every production change goes out as a canary gated by Prometheus analysis — no platform ticket, no big-bang deploy. This is Argo Rollouts plus metric analysis and auto-rollback, delivered through a self-service ApplicationSet.

Each app team gets one app-delivery ApplicationSet in the platform monorepo, a matrix of the team’s app directories crossed with the clusters that team’s residency allows. The team never touches this ApplicationSet — they merely commit to their config repo, and apps appear:

# platform-gitops/applicationsets/apps-records.yaml
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata: { name: apps-records, namespace: argocd }
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - matrix:
        generators:
          - git:                                   # one entry per app dir the team commits
              repoURL: https://github.com/aurora-health/records-config.git
              revision: main
              directories: [{ path: apps/* }]
          - clusters:
              selector:
                matchLabels: { env: prod, residency: eu }   # records = EU data → AKS only
  template:
    metadata:
      name: 'records-{{.path.basename}}-{{.name}}'
    spec:
      project: team-records                         # ← the guardrail from Phase 6
      source:
        repoURL: https://github.com/aurora-health/records-config.git
        targetRevision: main
        path: '{{.path.path}}/overlays/prod'
      destination:
        server: '{{.server}}'
        namespace: records
      syncPolicy:
        automated: { prune: true, selfHeal: true }

Note the residency selector: records handles EU patient data, so its clusters filter is residency: eu — the ApplicationSet cannot place a records app on the US or India cloud even if someone tried. Data residency is enforced by the generator, then again by the AppProject in Phase 6. Defense in depth, applied to a compliance requirement.

The app itself deploys as a Rollout, not a Deployment, so its production change is a canary. The steps shift traffic in stages with a metric gate between them:

# records-config/apps/api/base/rollout.yaml  (canary with a Prometheus gate)
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata: { name: records-api, namespace: records }
spec:
  replicas: 6
  selector: { matchLabels: { app: records-api } }
  strategy:
    canary:
      canaryService: records-api-canary
      stableService: records-api-stable
      trafficRouting:
        nginx: { stableIngress: records-api }
      steps:
        - setWeight: 10
        - pause: { duration: 5m }
        - analysis: { templates: [{ templateName: success-rate }] }
        - setWeight: 50
        - pause: { duration: 10m }
        - setWeight: 100
  template:
    metadata: { labels: { app: records-api } }
    spec:
      containers:
        - name: api
          image: aurora.azurecr.io/records-api:v2.4.1   # pinned tag — promotion bumps THIS
          ports: [{ containerPort: 8080 }]

The AnalysisTemplate is what makes the canary decide instead of just wait — it queries Prometheus and aborts (rolling back to stable in seconds) if the success rate drops below the SLO:

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata: { name: success-rate, namespace: records }
spec:
  metrics:
    - name: success-rate
      interval: 1m
      successCondition: "result[0] >= 0.99"          # ≥99% non-5xx
      failureLimit: 3                                  # 3 bad samples → abort
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            sum(rate(http_requests_total{app="records-api",code!~"5.."}[2m]))
            /
            sum(rate(http_requests_total{app="records-api"}[2m]))

The traffic router differs per cloud (NGINX here, but ALB or GCLB or a service mesh elsewhere) — that per-cloud wiring is the subject of the traffic-management lesson. The strategy is identical everywhere, which is why the app team writes it once.

What just happened. An app team ships by merging a PR: their app-delivery ApplicationSet notices the new/changed directory, Argo CD syncs the Rollout, and the new version rolls out as a Prometheus-gated canary that auto-aborts on a regression — all on the correct cloud for that team’s data residency, all without a platform engineer in the loop. That is the self-service NFR, delivered.

---

Phase 6 — Multi-tenancy and security

Goal: Bound what every actor can do, so self-service cannot become self-destruction. An AppProject per team is the tenancy wall; hardening — signed commits, image verification, default-deny networking — is the supply-chain and blast-radius wall.

The AppProject is the load-bearing guardrail. It pins the only repos, clusters/namespaces and resource kinds a team’s Applications may touch, and — critically for auditability — it can require signed commits via signatureKeys and forbid deploys outside a change window via syncWindows:

# platform-gitops/appprojects/team-records.yaml
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata: { name: team-records, namespace: argocd }
spec:
  description: "Records (EHR) — EU data only"
  sourceRepos:
    - https://github.com/aurora-health/records-config.git    # ONLY this repo
  destinations:
    - server: https://prod-aks-eu.example              # ONLY the EU (AKS) cluster
      namespace: records                                # ONLY this namespace
  clusterResourceWhitelist: []                          # deny ALL cluster-scoped resources
  namespaceResourceWhitelist:
    - { group: "*", kind: "*" }
  roles:
    - name: developer
      policies:
        - p, proj:team-records:developer, applications, sync, team-records/*, allow
      groups: [ aurora-health:records-engineers ]
  signatureKeys:
    - keyID: ABCDEF1234567890                           # commits MUST be GPG-signed by this key
  syncWindows:
    - kind: deny
      schedule: "0 22 * * 5"                            # no Friday-night prod deploys
      duration: 60h
      applications: ["*"]

Read what that project forbids: a records Application can only pull from the records repo, can only deploy to the EU AKS cluster’s records namespace, can create no cluster-scoped resource (no ClusterRole, no CRD, no escaping its namespace), must be built from a GPG-signed commit, and cannot sync during the Friday-night freeze. A fat-fingered or compromised app repo is contained by the project before anything is applied. That containment table across all four teams:

Above the projects sit two supply-chain controls, deployed to every spoke by a platform ApplicationSet. Kyverno admission verifies that every image is signed by Aurora’s cosign key and comes from an Aurora registry — a backstop that catches anything the project let through:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata: { name: verify-aurora-images }
spec:
  validationFailureAction: Enforce
  webhookTimeoutSeconds: 30
  rules:
    - name: check-signature
      match: { any: [{ resources: { kinds: [Pod] } }] }
      verifyImages:
        - imageReferences:
            - "aurora.azurecr.io/*"
            - "111122223333.dkr.ecr.us-east-1.amazonaws.com/aurora/*"
            - "asia-south1-docker.pkg.dev/aurora/*"
          attestors:
            - entries:
                - keys:
                    publicKeys: |-
                      -----BEGIN PUBLIC KEY-----
                      <cosign public key placeholder>
                      -----END PUBLIC KEY-----

And a default-deny NetworkPolicy in every app namespace, so a compromised pod cannot talk to another team’s workloads — you then add back only the flows the app needs:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: default-deny, namespace: records }
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]

What just happened. Every app team now operates inside a box it cannot see out of: its own repo, its own cluster and namespace, signed commits only, signed images only, no lateral network reach, and no ability to create Applications the platform did not template. Self-service is safe because the guardrails are declarative and enforced in depth — at the Argo CD gate, at admission, and at the network.

---

Phase 7 — Observability and notifications

Goal: Know the health of the whole fleet at a glance, and get paged when something drifts — without logging into three clouds. This is Argo CD observability and notifications applied at fleet scale.

Two things must be observable: Argo CD itself (is the platform reconciling?) and the fleet’s apps (are they healthy?). The hub’s application-controller exports Prometheus metrics — sync counts, reconciliation latency, app health — that a single Grafana dashboard turns into one view across AKS, EKS and GKE. The per-spoke Prometheus (from Phase 3) federates its app metrics up to the mgmt cluster’s Prometheus, so one query spans the fleet.

Notifications turn a breached SLO into a Slack page. Argo CD’s notifications controller subscribes to app events and routes them; Aurora pages the owning team’s channel on OutOfSync or Degraded and the platform channel on anything cluster-level:

# argocd-notifications-cm  (representative)
trigger.on-degraded: |
  - when: app.status.health.status == 'Degraded'
    send: [app-degraded]
template.app-degraded: |
  message: "🔴 {{.app.metadata.name}} is Degraded on {{.app.spec.destination.server}}"
subscriptions: |
  - recipients: [slack:aurora-platform]
    triggers: [on-degraded, on-sync-failed]

What just happened. One Grafana shows every app on every cloud; one set of SLOs defines “healthy fleet”; and a drift, a failed sync, or an unreachable spoke pages the right humans in Slack without anyone watching three consoles. Observability is what makes a one-team-fifty-app platform actually operable.

---

Phase 8 — Disaster recovery: the hub is gone

Goal: Survive the loss of the entire management cluster and rebuild the control plane from Git, with the fleet’s apps never going down. This is the HA & DR lesson as the platform’s final proof.

The reason DR here is survivable is the fact this whole course leaned on: Argo CD’s control plane is not in the data path. If the management cluster burns down, the records-api pods on the EKS spoke keep serving US patients — they do not route through the hub. You lose reconciliation (drift correction, self-heal, the ability to ship changes), not availability. DR is therefore rebuild-from-Git, not restore-a-database, and it is fast.

The recovery hinges on one distinction: almost everything is already in Git, and a tiny set of state is not.

The runbook, in order — the order matters because the hub must be able to reach the spokes before the root app fans out:

Step 6 is the safety check that separates a clean DR from a fleet-wide outage. When the rebuilt hub applies the root app, it compares Git to the already-running workloads on the spokes; because they match, it marks them Synced with nothing to change — it re-adopts them. If instead the diff shows everything wanting to redeploy, your restored state or a targetRevision is wrong; you stop and fix it rather than let recovery cause the outage you were recovering from.

What just happened. You proved the platform’s most important property: the control plane is rebuildable and the fleet outlives it. A dead hub is a degraded state (no new deploys, no drift correction) measured in minutes to recover, not an outage of Aurora’s applications. That is the difference between a platform and a house of cards.

---

Environment promotion across the fleet

A change to records-api should flow dev → staging → prod the same way on all three clouds, and every step must be a reviewed, attributable Git change — the auditability NFR. Aurora promotes by pinning an immutable image tag per environment overlay and moving that pin with a pull request; nothing auto-flows to prod. This is sync policies and the promotion patterns applied fleet-wide.

The promotion PR is tiny and auditable — one line:

# records-config/apps/api/overlays/prod/kustomization.yaml
images:
  - name: aurora.azurecr.io/records-api
    newTag: v2.4.1        # was v2.4.0 — THIS one-line PR is the prod promotion

Because prod overlays pin an exact tag (never a moving branch), the diff a reviewer approves is the thing that ships, and a rollback is a git revert of that one-line PR. An image updater can open the dev bump PR automatically, but the promotion to prod stays a human decision. The residency selector in each team’s ApplicationSet guarantees the promoted image lands only on the legally-correct cloud — promotion and residency are enforced by the same label machinery.

---

Cost and operations

Multi-cloud tempts a 3× bill; a disciplined platform avoids it. The unifying moves: one hub (not three), one observability stack, spot/preemptible nodes for platform components, and right-sized spokes.

Operationally, the platform’s leverage is that the number of things a human touches does not grow with the fleet. Adding cluster four is a labelled cluster Secret plus a Terraform module; the platform stack fans onto it automatically. Onboarding app team five is one AppProject and one app-delivery ApplicationSet in a PR. The platform team’s job becomes owning the guardrails and the templates, not the individual deployments — which is the only way five engineers support fifty teams.

The synthesis in one table — every phase, the lesson it draws on, and the one manifest that anchors it:

---

Going deeper

The eight phases stand the platform up; this section is what a staff engineer asks after it runs — the questions that separate “it demoed” from “it is on-call at 3 a.m. across three clouds.” Nothing here re-teaches a phase; it is the depth the phases assume.

The production-readiness scorecard

Before Aurora declares go-live, someone reviews the platform against a bar. Here is the scorecard that review uses — each row is a decision already made above, restated as the thing a reviewer actually checks and the phase that has to show it.

A platform that passes every row is not “finished,” but a platform that fails any row is not ready for patient data, and each failure maps to a phase you can point at. Grade the compliance and policy story against this bar specifically — it is the row auditors ask about first.

Where the single hub actually strains

“One hub for the whole fleet” is the right call for Aurora, but concentration has an engineering cost that only shows at scale, and the install’s controller: { replicas: 1 } is a starting point, not a resting place. Four pressure points decide when you must scale the hub:

The exec-auth point is the one people miss: unlike a static kubeconfig, execProviderConfig mints a token by calling the cloud’s IAM at reconcile time, so a large fleet is also a steady call rate against three clouds’ token endpoints — a throttle there looks exactly like a flapping spoke. Redis-HA is disposable but not free to lose: dropping the cache forces a full re-list and a reconcile storm, so treat a Redis outage as a thundering-herd event, not a no-op. The scaling and sharding lesson owns the sizing math; the capstone point is only that the hub is the one shared dependency, so you instrument and scale it before it becomes the fleet’s ceiling.

Three failure modes, traced end to end

The symptom→fix table later is for single seams. These three failures test whether the composition holds — walked through so you can watch each control do its job.

A cloud’s control-plane API goes dark (regional outage). Say EKS’s API server is unreachable for an hour. The hub marks every prod-eks Application Unknown and argocd_cluster_info drops that spoke — the platform channel pages. Crucially, the US patients keep being served: the running records/billing pods on EKS are not in the hub’s data path, so they route traffic exactly as before. AKS and GKE reconcile untouched — the failure is isolated to one cloud label. Because selfHeal cannot reach the spoke, it does not thrash; when the API returns, the hub re-adopts the unchanged workloads with an empty diff. The only thing you actually lost for that hour was the ability to deploy to EKS — a degraded-reconciliation window, not an outage.

A bad image passes CI and is promoted to prod. A regression ships in records-api:v2.4.2; the promotion PR bumps the tag and merges. The app-delivery ApplicationSet syncs it — but as a Rollout, not a Deployment — so it enters the canary at setWeight: 10. Only 10% of EU traffic hits the bad build. At step 3 the success-rate AnalysisTemplate queries Prometheus, sees the 5xx ratio breach 0.99, and after failureLimit: 3 bad samples the Rollout auto-aborts to stable in seconds. The owning team is paged; the fix is a one-line git revert of the tag bump. No platform engineer touched it, the blast radius was 10% of one cloud, and residency held — the bad build never had a path to the US or India clouds.

The hub dies mid-rollout. The management cluster is lost while a platform-ingress RollingSync is halfway through — AKS done, EKS in progress, GKE not started. Everything already synced stays; the in-flight step simply stops; every spoke keeps serving. You run the Phase 8 runbook: Terraform a new mgmt cluster, reinstall HA Argo CD, restore the cluster Secrets, apply the root app. The rebuilt hub reads the same Git desired state and resumes the RollingSync from where reality diverges from Git — AKS shows an empty diff (re-adopt), EKS/GKE converge to the intended chart. Idempotent desired state is what turns “the hub died during a change” from a corruption event into a resume.

Version and API caveats (Argo CD 3.0 GA)

Every manifest above is schema-correct on Argo CD 3.0 GA, but 3.0 changed a few defaults worth pinning down in a platform this size:

• Fine-grained RBAC now covers logs and applicationsets as first-class resources. An app team that could tail pod logs on 2.x may lose that view unless you grant logs, get explicitly in argocd-rbac-cm — audit the Phase 2 policy against 3.0 before cutover.
• Prefer annotation-based resource tracking (argocd.argoproj.io/tracking-id) over the instanceLabelKey label the install sets. At fleet scale the label can collide when instance names overlap; the annotation is collision-proof and is the recommended default.
• Use server-side apply for the heavy CRDs. kube-prometheus-stack and Kyverno ship large CRDs that can exceed the client-side last-applied-annotation limit; set ServerSideApply=true as a sync option on those platform apps.
• ESO is moving to a stable API. The external-secrets.io/v1beta1 shown here is superseded by GA external-secrets.io/v1 in ESO ≥ 0.10 — plan the apiVersion bump as a fleet-wide, RollingSync’d platform change, exactly the kind of edit Phase 3’s safety rails exist for.
• Keep the exec plugin binaries patched. kubelogin, the AWS CLI, and gke-gcloud-auth-plugin live in the hub image; a stale one breaks auth to just its cloud. Pin and patch them like any other dependency.

Where to take it next

Once the eight phases are solid, the natural extensions each harden one axis: a staging residency zone per cloud so promotion has a real pre-prod on the correct legal cloud; policy-report dashboards surfacing Kyverno results next to sync health; Gateway API traffic routing so the canary strategy is identical across clouds instead of per-router; a second warm-standby hub in another region to cut DR RTO toward zero; and per-team cost attribution off the same cloud/residency/team labels the fleet already carries. Each is a small, additive change precisely because the platform’s shape — labels, ApplicationSets, projects — was designed to absorb it.

---

Hands-on lab

This lab is the capstone build runbook at design/config level. There is no live three-cloud fleet here, so you execute it against your own AKS/EKS/GKE clusters (or substitute three kind clusters to rehearse the shape for free — the ApplicationSets, projects and rollouts are cloud-neutral; only ingress/secrets/registry edges need the real clouds). Each phase gives the anchor manifest, the verify command with representative output, and a one-line “what just happened.” ⚠️ Real clouds bill: managed control planes, load balancers, NAT, and private endpoints all cost money — tear everything down at the end.

Phase 0 — repos. Create platform-gitops (structure from the repo section) and one *-config per team. Commit the bootstrap/, appprojects/, applicationsets/, platform/ and policies/ trees.

Phase 1 — bootstrap.

cd platform-terraform && terraform init && terraform apply    # 4 clusters + HA Argo CD + root app
argocd login <mgmt-argocd-host> --sso                          # first login via SSO
argocd app get root                                            # representative:
#   Name:            argocd/root
#   Project:         platform
#   Sync Status:     Synced to main (a1b2c3d)
#   Health Status:   Healthy

What just happened: Terraform built the substrate and handed off; the root app is now the source of the whole platform.

Phase 2 — register + identify. Apply the three cluster Secrets (via the bootstrap/clusters/ path the root app syncs), then:

argocd cluster list        # representative:
# SERVER                                          NAME          VERSION  STATUS   LABELS
# https://...eks.amazonaws.com                    prod-eks-us   1.30     Successful  cloud=eks,env=prod,residency=us
# https://prod-aks-eu.example                     prod-aks-eu   1.30     Successful  cloud=aks,env=prod,residency=eu
# https://...container.googleapis.com             prod-gke-in   1.30     Successful  cloud=gke,env=prod,residency=india

What just happened: the hub authenticates to all three clouds and every spoke carries its routing labels.

Phase 3 — platform layer. The root app has already synced the platform ApplicationSets. Verify the fan-out:

kubectl -n argocd get applicationset
kubectl -n argocd get applications -l app.kubernetes.io/part-of=platform -o wide
# representative: ingress-prod-aks-eu / -eks-us / -gke-in all Synced/Healthy

What just happened: one ApplicationSet produced one ingress Application per cloud, each with the correct class.

Phase 4 — secrets. Apply a team SecretStore + ExternalSecret, then confirm the native Secret materialized:

kubectl -n records get externalsecret records-db
# NAME         STORE          REFRESH  STATUS         READY
# records-db   cloud-secrets  1h       SecretSynced   True

What just happened: ESO pulled the secret from the cloud store via workload identity; no plaintext in Git.

Phase 5 — an app with a canary. Merge a tag bump in records-config, then watch the rollout:

kubectl argo rollouts get rollout records-api -n records --watch
# representative:  Status: ॥ Paused (setWeight 10)  →  Analysis: success-rate ✔  →  Healthy

What just happened: the app shipped as a Prometheus-gated canary, self-service, on the residency-correct cloud.

Phase 6 — guardrails. Apply the AppProjects and Kyverno policy, then prove the box holds:

# Try to sync an app to a namespace the project forbids — it must be REJECTED:
argocd app set records-api --dest-namespace billing
# representative: FATA[] application ... is not permitted in project team-records

What just happened: the AppProject rejected an out-of-bounds deploy before anything hit a cluster.

Phase 7 — observe. Import the fleet Grafana dashboard and confirm one view spans three clouds; send a test Slack alert. What just happened: one pane, three clouds, and drift now pages a human.

Phase 8 — DR game day. In a test environment, delete the mgmt cluster and run the runbook. Confirm the spokes kept serving throughout, and that argocd app diff is empty after re-adoption. What just happened: you proved the fleet survives losing the hub.

Verify the fleet is healthy (the one command that summarizes everything):

argocd app list -o wide | awk 'NR==1 || $0 !~ /Synced.*Healthy/'   # show only NOT-healthy apps
# representative healthy fleet: only the header prints — every app Synced/Healthy across all 3 clouds
kubectl --context mgmt -n argocd get applications \
  -o custom-columns=NAME:.metadata.name,SYNC:.status.sync.status,HEALTH:.status.health.status

Teardown (do this — real clouds bill):

argocd app delete root --cascade        # removes every generated app across the fleet
cd platform-terraform && terraform destroy   # removes all 4 clusters + identities + LBs
# Manually confirm no orphaned cloud LBs/NAT/private-endpoints remain in each cloud console.

---

Practice challenges

Six graded exercises, all framed as extend or harden the Aurora platform — the way you would actually grow it. Each is design/config level (no live three-cloud fleet needed; three kind clusters rehearse the shape). Try each before opening the solution.

1. (Beginner) Add a fourth spoke with zero new ApplicationSets. Aurora needs an EU dev cluster. Bring it into the fleet so the full platform stack lands on it automatically.

<details> <summary>Solution</summary>

Create one labelled cluster Secret in the hub’s argocd namespace — cloud: aks, env: dev, residency: eu, plus its execProviderConfig — exactly as in Phase 2. Apply nothing else. The platform ApplicationSets’ clusters generators match on labels, so ingress, cert-manager, ESO, Prometheus and Rollouts fan onto it on the next reconcile.

kubectl -n argocd get applications -l app.kubernetes.io/part-of=platform | grep dev-aks
# representative: ingress-/observability-/eso-…-dev-aks all Synced/Healthy

Why: it proves the fan-out is truly label-driven — new capacity is a registration, not a rewrite. </details>

2. (Beginner→Intermediate) Onboard a fifth app team, self-service. A pharmacy team handling EU data must ship without a platform ticket.

<details> <summary>Solution</summary>

Two files in one platform-gitops PR: a team-pharmacy AppProject pinning pharmacy-config → the EU (AKS) cluster’s pharmacy namespace (deny cluster-scoped resources), and an apps-pharmacy app-delivery ApplicationSet with a matrix of apps/* × {env: prod, residency: eu}. Map the team’s SSO group to the project’s sync role. The team then owns pharmacy-config; merges appear as apps.

Why: onboarding is two declarative files — which is what lets five platform engineers support fifty teams. </details>

3. (Intermediate) Make a fleet-wide platform edit safe. The platform-observability ApplicationSet has no RollingSync. Harden it so a bad chart bump can’t break all three clouds at once.

<details> <summary>Solution</summary>

Add a RollingSync strategy with per-cloud steps and make the template stamp the label the steps match:

spec:
  strategy:
    type: RollingSync
    rollingSync:
      steps:
        - matchExpressions: [{ key: cloud, operator: In, values: [aks] }]
        - matchExpressions: [{ key: cloud, operator: In, values: [eks] }]
        - matchExpressions: [{ key: cloud, operator: In, values: [gke] }]
  template:
    metadata:
      labels: { cloud: '{{index .metadata.labels "cloud"}}' }

Also set goTemplateOptions: ["missingkey=error"] and syncPolicy.applicationsSync: create-update.

Why: the steps match the label the template stamps — get that wiring wrong and the rollout stalls at step one; get it right and a mistake is caught after the first cloud. </details>

4. (Intermediate→Advanced) Enforce a change-freeze and signed images for billing. Regulators require billing to deploy only in-window and only signed images from Aurora’s registry.

<details> <summary>Solution</summary>

On the team-billing AppProject add syncWindows (a deny window over the freeze, an allow window for the approved change window) and signatureKeys so only GPG-signed commits sync. Image signing is already enforced fleet-wide by the Kyverno verifyImages policy scoped to Aurora registries — confirm the EKS ECR path is in its imageReferences. See environment promotion for the PR flow the window gates.

Why: compliance is a per-project declarative control, enforced at the Argo CD gate and again at admission — defense in depth. </details>

5. (Advanced) Scale the hub for 40 clusters. Reconciliation p95 has crossed 30s as the fleet grew. Keep one hub; make it keep up.

<details> <summary>Solution</summary>

Shard the application-controller (controller.replicas: 3 with round-robin or consistent-hashing sharding, or enable dynamic cluster distribution), add repo-server replicas with a raised timeout and manifest caching, and run the applicationset-controller with replicas: 2. Watch argocd_app_reconcile p95 and workqueue_depth fall, and confirm exec-auth token calls aren’t hitting a cloud IAM quota. The scaling lesson has the sizing math.

Why: the hub is the one shared dependency — you scale it deliberately before it becomes the fleet’s ceiling, not after an incident. </details>

6. (Advanced→Expert) Remove the hub as a DR single point. Get DR RTO toward zero without splitting the fleet into three Git sources.

<details> <summary>Solution</summary>

Stand up a warm-standby hub in a second region that watches the same platform-gitops, with auto-sync off (observe-only) so it never fights the primary. On hub loss, restore the cluster Secrets to the standby and flip auto-sync on — the runbook drops from “rebuild” to “promote.” The trade-off: a second hub is a second concentrated credential to guard, so it stays observe-only until promoted. (Sharding by cloud is the alternative — smaller blast radius, but three DR plans again.)

Why: it attacks the platform’s deliberate concentration — one hub is one blast radius — while keeping the single Git source of truth that makes the fleet coherent. </details>

---

Common beginner mistakes

These are conceptual traps — the wrong mental model, not a symptom to grep for (those are in the troubleshooting table below). Each is the misunderstanding, then the model that replaces it.

• “Multi-cloud means one Argo CD per cloud.” It feels safer to give each cloud its own control plane. But Argo CD reconciles any Kubernetes API server, so one hub runs AKS, EKS and GKE from a single Git source — you unify the deployment plane even though the infrastructure plane can’t consolidate. Reach for per-cloud hubs only when regulatory isolation of the control plane itself demands it, not by default.

• “If the hub goes down, the apps go down.” The hub is not in the data path. Losing it costs you reconciliation — new deploys, drift correction, self-heal — not availability; the pods on every spoke keep serving. Internalizing this is what makes the DR plan a calm rebuild-from-Git instead of a panic, and it is the single most-tested idea in a capstone interview.

• “Terraform and Argo CD both manage clusters, so let Terraform manage the Applications too.” Two controllers reconciling the same object fight forever. There is exactly one hand-off: Terraform owns the substrate up to installing Argo CD and applying the one root Application; Argo CD owns everything above it. Blur that line and DR becomes archaeology.

• “An ApplicationSet is just a way to type fewer YAML files.” It is a fleet-wide actuator: one edit fans to every matching cluster on the next tick. That power is exactly why it needs RollingSync, create-update (never create-delete), and missingkey=error — treat it as a loaded control, not a templating convenience.

• “Data residency is a policy you write down and audit.” On this platform it is enforced by construction: cluster residency labels plus ApplicationSet selectors decide where an app can even be generated, and the AppProject destinations reject anything out of zone at the sync gate. A residency violation should be impossible, not merely caught in review.

• “Giving a team sync gives them control of the platform.” Teams get sync, not create. The Applications themselves come from platform-owned ApplicationSets, so a team can promote its own app but cannot conjure a new one outside the guardrails. Self-service and least privilege are not in tension here — the ApplicationSet is what reconciles them.

---

Common mistakes and troubleshooting

The cross-cutting failures — the ones that only appear when the pieces are combined — are the ones that bite a multi-cloud platform. Single-lesson gotchas are covered in their own lessons; these are the seams.

The three nastiest deserve prose. Blast radius through ApplicationSets is the defining multi-cloud hazard: the very object that makes the fleet manageable is a loaded gun, because one edit fans to every matching cluster instantly. RollingSync is not optional at fleet scale — and its steps must match the labels your template stamps, a wiring people get subtly wrong so the rollout stalls at step one. Per-cloud identity on the hub is the failure that looks like an Argo CD bug but is a cloud IAM problem: the hub authenticates three different ways, and a single missing exec binary or ungranted role takes out exactly one cloud while the other two look perfect — always check whether the failure is isolated to one cloud label before you suspect Argo CD. DR that redeploys instead of re-adopts is the scariest, because it turns a recovery into the outage: the empty-diff check before enabling auto-sync is the one step you never skip.

---

Cheat-sheet

The platform, in one breath: Terraform builds a mgmt cluster + AKS/EKS/GKE spokes and hands off to one HA Argo CD hub; the root app-of-apps fans platform ApplicationSets (ingress, cert-manager, ESO, Prometheus, Rollouts) across every spoke with a per-cloud branch; app teams self-serve canary deploys through app-delivery ApplicationSets bounded by AppProjects + Kyverno; fleet SLOs + Slack watch it; a rebuild-from-Git runbook survives losing the hub.

Phased build runbook:

Key-manifest index:

Commands that summarize fleet state:

---

Interview and exam questions

Q: Design a GitOps platform for a company whose data-residency rules force workloads onto AKS, EKS and GKE simultaneously. What topology do you choose, and why? A: Hub-and-spoke: one HA Argo CD hub on a dedicated management cluster managing all three clouds as spokes. It gives one control plane, one ApplicationSet-driven fan-out, and one DR plan for a small platform team — the alternatives (one Argo per cluster, one hub per cloud) multiply operational surface without benefit at this scale. I bound the hub’s concentrated blast radius with HA, RBAC, and per-team AppProjects, and I keep the management cluster workload-free so the control plane’s failure domain is separate from the apps.

Q: One cloud’s apps all show Unknown while the other two are healthy. Walk me through the diagnosis. A: The failure is isolated to one cloud label, which almost always means the hub cannot authenticate to that one spoke — an Argo CD-shaped symptom with a cloud-IAM cause. Check three things in order: is that cloud’s exec plugin binary present in the hub image; is the hub’s identity actually granted on that spoke (access entry / RBAC / GSA binding); and is the cluster Secret’s execProviderConfig correct. It is not a general Argo CD outage, because the other two clouds work.

Q: Why is a single ApplicationSet edit the most dangerous operation on this platform, and how do you make it safe? A: A cluster-generator ApplicationSet reconciles a change to every matching spoke on the next tick, so a bad template breaks AKS, EKS and GKE at once — the fleet is one object. RollingSync makes it safe by rolling the change cloud-by-cloud (steps matching the cloud label the template stamps), so a mistake is caught after the first cluster. I also set create-update (never create-delete) and missingkey=error so a narrowed selector can’t prune the fleet and a typo fails the render instead of silently blanking a field.

Q: How does an app team ship to production without a platform engineer, and what stops them from doing something dangerous? A: They merge a one-line tag bump in their own config repo; their app-delivery ApplicationSet syncs it as a Prometheus-gated canary on the residency-correct cloud. What stops harm is layered: RBAC gives them sync not create; their AppProject pins the only repo/cluster/namespace they can touch, denies cluster-scoped resources, and requires signed commits; Kyverno rejects unsigned images at admission; and a default-deny NetworkPolicy prevents lateral movement. Self-service is safe because the guardrails are declarative and enforced in depth.

Q: The management cluster is destroyed. What happens to the running apps, and what is your recovery? A: Nothing happens to the running apps — the hub is not in the data path, so the spokes keep serving. I lose reconciliation, not availability. Recovery is rebuild-from-Git: Terraform a fresh mgmt cluster, install HA Argo CD, restore the cluster Secrets (the one thing not in app Git), re-apply config, apply the root app, and confirm argocd app diff is empty so the hub re-adopts rather than redeploys. RTO is under an hour and it is a degraded-reconciliation window, not an app outage.

Q: Where do you draw the line between Terraform and Argo CD, and why does it matter? A: Terraform owns the substrate — clusters, node pools, per-cloud identities, networking — up to and including installing Argo CD and applying the one root Application. Argo CD owns everything above: the platform layer, apps, projects, policies. The line matters because if Terraform managed Argo CD Applications too, two controllers would fight over the same resources; and because a clean handoff is exactly what makes DR a fast rebuild rather than an archaeology project.

Q: How do you enforce data residency technically, not just by policy? A: With cluster labels and two independent enforcement points. Each spoke is labelled residency: eu|us|india; a team’s app-delivery ApplicationSet selects only its residency (records → residency: eu), so it cannot generate an app on the wrong cloud. Then the team’s AppProject destinations pins the exact cluster and namespace, rejecting any out-of-zone deploy at the Argo CD gate. Two layers, both declarative, so a residency violation is impossible-by-construction rather than caught-in-review.

Q: Your monthly cloud bill jumped after onboarding a fourth cluster. What’s your first hypothesis and fix? A: The new spoke likely brought on-demand nodes and a public load balancer per app with no autoscaler — the classic multi-cloud cost multiplier. Fix: spot/preemptible pools for stateless platform pods, a shared ingress rather than an LB per service, cluster autoscaler with scale-to-zero for dev, and a check for orphaned cloud LBs/NAT. The platform’s cost discipline is one hub, one observability stack, and keeping data (and thus egress) in-cloud.

Q: Why name the ESO SecretStore identically in every cluster? A: So the app team’s ExternalSecret is cloud-neutral. The per-cloud difference (azurekv vs aws vs gcpsm, and the workload-identity binding) lives in the SecretStore the platform ApplicationSet deploys; because the store’s name is the same everywhere, the ExternalSecret referencing it is written once and works on all three clouds. The platform absorbs the seam so the app doesn’t have to.

Q: How do you promote a change across the fleet while keeping it auditable, and how do you roll back? A: Pin an immutable image tag per environment overlay and move the pin with a reviewed PR — dev auto-syncs on merge, staging and prod are tag-bump PRs with review gates, and prod also passes canary analysis. Because prod overlays never point at a moving branch, the diff a reviewer approves is exactly what ships, and rollback is a git revert of that one-line PR. The residency selector guarantees the promoted image lands only on the legal cloud.

---

Glossary

• Hub-and-spoke — one Argo CD control plane (the hub) that manages many target clusters (the spokes). Here, one hub on a management cluster drives AKS, EKS and GKE.
• Management cluster — a small, workload-free cluster that runs only the platform’s control plane (Argo CD, Redis, backups), kept separate so the control plane’s failure domain doesn’t overlap the apps.
• Spoke — a cluster that runs actual workloads and is managed by the hub. Aurora has three: the AKS (EU), EKS (US) and GKE (APAC) prod clusters.
• App-of-apps / root app — the single, hand-applied Application that points at a directory of other Applications/ApplicationSets, so everything else is bootstrapped from Git by one object.
• ApplicationSet — a controller that generates many Applications from a template plus a generator (e.g. one per cluster, or a matrix of app-dirs × clusters). The fleet’s fan-out mechanism.
• Generator — the part of an ApplicationSet that produces the list to template over: clusters (every matching spoke), git (every matching directory), matrix (a cross-product of the two).
• RollingSync (progressive sync) — an ApplicationSet strategy that rolls a change out in ordered steps (here, cloud-by-cloud) instead of to every cluster at once, so a bad change is caught early.
• AppProject — the tenancy wall: it pins the repos, clusters/namespaces and resource kinds a team’s Applications may touch, and can require signed commits and enforce sync windows.
• Exec auth / execProviderConfig — a cluster credential that runs a cloud token binary at reconcile time (kubelogin / aws eks get-token / gke-gcloud-auth-plugin) so the hub holds no long-lived kubeconfig.
• Workload Identity — a cloud’s way of giving a Kubernetes ServiceAccount a cloud IAM identity with no static key (Entra Workload ID on Azure, IRSA/Pod Identity on AWS, GKE Workload Identity on GCP).
• ESO / SecretStore / ExternalSecret — the External Secrets Operator pulls secrets from a cloud store into native K8s Secrets. The SecretStore declares the provider (the per-cloud seam); the ExternalSecret declares which keys to sync (cloud-neutral).
• Argo Rollouts / Rollout — a Deployment replacement that ships new versions progressively (canary or blue-green) instead of all at once.
• AnalysisTemplate — the metric query (here, a Prometheus success-rate) that lets a canary decide: promote if healthy, auto-abort if the SLO breaks.
• Canary — a release that shifts a small slice of traffic to the new version first, soaks, checks metrics, then widens — so a regression is contained to that slice.
• Kyverno verifyImages — an admission policy that rejects any Pod whose image isn’t signed by the org’s key and from an allowed registry — a backstop beneath the AppProject.
• Data residency — the legal requirement that data stays in a jurisdiction; enforced here by residency labels + ApplicationSet selectors + AppProject destinations.
• Blast radius — how much breaks from one action. The capstone’s recurring theme: one hub concentrates it (the risk), and HA + RollingSync + projects bound it (the mitigation).
• Re-adoption — after a DR rebuild, the hub compares Git to the already-running workloads, finds them equal, and marks them Synced with nothing to change — recovery without redeploying.
• RTO / RPO — Recovery Time Objective (how fast you’re reconciling again — < 60 min here) and Recovery Point Objective (how much desired state you lose — ~0, because Git is the source).
• Sync window — an AppProject schedule that allows or denies syncs in a time range (e.g. a Friday-night prod freeze).

---

Key takeaways

• The deployment plane unifies even when infrastructure can’t. Argo CD reconciles Git into any Kubernetes API server, so one hub runs AKS, EKS and GKE from one source of truth; the clouds differ only at the edges — identity, ingress, secrets, registries — which is exactly what the rest of this course taught you to handle.
• A capstone is synthesis, not a topic list. The platform is eight phases building five layers bottom-up: substrate (Terraform) → control plane + identity → platform layer + secrets → apps + guardrails → operations + DR. Each phase is one prior lesson, composed — not re-taught.
• Concentration is the core trade-off. One hub is one pane of glass and one blast radius. Every safety decision — HA, RollingSync, AppProjects, Kyverno, default-deny networking, the empty-diff DR check — is about bounding a risk you deliberately concentrated for the operational win.
• Labels are the fleet’s organizing principle. cloud, env, region, residency on each cluster Secret turn fleet-wide targeting, per-cloud branching, and data-residency enforcement into declarative selectors instead of hand-written per-cluster manifests.
• Self-service is safe only because the guardrails are declarative. App teams ship by merging a PR because AppProjects, RBAC, Kyverno and NetworkPolicy bound what a merge can do — enforced at the Argo CD gate, at admission, and at the network. That is what lets five platform engineers support fifty app teams.
• The control plane is rebuildable and off the data path. Losing the hub is a degraded-reconciliation window measured in minutes, not an app outage — because Git is the backup and the spokes keep serving. Prove it with a DR game day; the diff-before-sync re-adoption check is the step you never skip.
• You can now design and defend a real multi-cloud GitOps platform end to end — the topology, the handoff, the fan-out, the guardrails, the promotion flow, and the recovery — which is the whole thing this course set out to teach.