Environment Promotion: Dev → Staging → Prod the GitOps Way (PR-Based, No Drift)

In a nutshell

Environment promotion is how a change travels from where you build it to where customers use it: dev → staging → prod. This lesson’s one big idea is that, done the GitOps way, promotion is not a command you run against a cluster and not a button you click — it is a small, reviewed edit to a file in Git. You change what the next environment should run, and Argo CD makes the cluster match. That’s it.

Picture a sealed shipping container. You pack it once (build the image once), and from then on you never repack it — you just move that exact container from a test dock to a staging dock to the customer dock, and at each dock a signed manifest entry (a pull request) says “this exact container is cleared for the next stop.” The checks get stricter as you near the customer: the test dock waves things through, the customer dock needs named sign-offs and only opens at set hours. The thing that moves is a reference to the sealed container; the container’s contents never change, and each dock’s local rules (how many forklifts, which gate) stay put.

Get that mental model and everything else falls into place: build once and promote a pinned reference (never rebuild per environment), keep one shared base so environments can’t silently drift apart, and let the gate tighten as you go right — dev auto-syncs, prod needs review, a manual sync, and a change window.

Level: Expert · Time: ~63 min

Prerequisites: you’re comfortable with a basic Argo CD Application (source/destination/syncPolicy), and with Kustomize bases and overlays. If not, skim Your first Application and Kustomize integration first. After this you can: structure a repo so a promotion is a one-line diff; kill config drift with a shared base and per-env overlays; gate prod with CODEOWNERS, manual sync, and sync windows; fan a promotion out across regions safely; and reason about the rendered-manifests pattern, Image Updater, and Kargo.

---

Every GitOps team eventually stands in front of the same whiteboard and asks the question that decides whether their platform is trustworthy or terrifying: a change is good in dev — now how does it get to staging, and then to production, without anyone breaking anything? Get the answer right and promotion becomes a boring, auditable, one-line pull request that anybody on the team can review over coffee. Get it wrong and promotion becomes a folklore ritual — someone SSHes to a jump box, runs a helm upgrade from their laptop, clicks Sync in the Argo CD UI, and prays that prod resembles the staging they tested.

This lesson is the whole discipline in one place. We treat promotion as what it actually is under GitOps: a controlled change to an environment’s desired state in Git. Not a command you run against a cluster. Not a button you press. A commit — almost always a pull request — that moves one number forward. We will compare the three repo-structuring patterns honestly (the good, the fallen-out-of-favour, and the “it depends”), destroy config drift with a shared base, wire up a real PR-based pipeline with reviewers and sync windows, fan a promotion out across regions, and look at the purpose-built tools (Argo CD Image Updater’s write-back, and Kargo) that automate the paperwork.

Everything here targets Argo CD 2.13+/3.x on Kubernetes 1.29+, and the mechanics are cloud-neutral — a tag bump is a tag bump on AKS, EKS or GKE alike. The one genuinely cloud-specific edge, per-environment secrets, gets all three clouds covered where it appears. Let’s build the pipeline every reviewer will thank you for.

---

Why this matters

Promotion is where GitOps either pays off or falls apart, because it is the exact seam between “a change is safe here” and “a change is live for customers”. A platform can have flawless manifests, gorgeous ApplicationSets and a self-healing controller, and still cause a Sev-1 — because the human process for moving a change forward was a Slack message and a hand-run command. The manifests were declarative; the promotion was not.

There are only two ways promotion goes wrong, and almost every production incident born of a deploy is one of them:

The mental model that dissolves both is a single sentence: an environment is a folder in Git that declares desired state, and promotion is a reviewed edit to that folder. Dev, staging and prod are not three clusters you push to; they are three files (or three directories) you change, and Argo CD’s job is to make each cluster match its file. Promotion never touches a cluster directly. It touches Git, a human (or a policy) approves the change, and the pull-based controller does the rest.

Hold onto three consequences of that model, because the rest of the lesson is their detail:

• The artifact is immutable; only the reference moves. You do not promote “the code” — you promote a pointer (an image tag, ideally pinned to a digest) from one environment’s file to the next. The bytes never change between dev and prod.
• Every environment difference must be explicit and reviewed, or it becomes drift. If staging and prod diverge in ways nobody can see in a diff, you have a time bomb. A shared base makes each difference a line someone approved.
• The gate lives in Git and in the sync policy, not in a person’s discipline. Branch protection, CODEOWNERS, required reviewers and Argo CD sync windows are the controls. “We’re careful” is not a control.

If you have run kubectl apply and nothing else, this lesson is the leap from “I can deploy” to “I can run a promotion pipeline a bank would sign off on.” Let’s start with the principle everything hangs from.

---

The core principle: promotion is a Git change, not a kubectl

Under a push-based world (plain CI running kubectl/helm against clusters), promotion is an action: a pipeline stage that reaches into staging, then another that reaches into prod. The environments are destinations you push to, and the record of what’s running lives in the head of whoever ran the last pipeline.

Under GitOps, promotion inverts. The environment’s desired state is a file in Git; a controller (Argo CD) continuously makes the cluster match that file. So to change what’s running, you change the file — and because it’s Git, that change is a commit, which means it can be a pull request, which means it can be reviewed, approved, recorded and reverted. The promotion is the PR.

The declarative column is the entire reason GitOps exists, and promotion is where you feel it most. Because the desired state is a reviewed artifact in Git, the question “what is running in prod, and who approved it?” has a precise answer: the file at HEAD of the prod path, and the approver on the PR that last changed it. No archaeology.

Two anti-patterns are worth naming immediately, because they quietly re-introduce the push model into a GitOps shop:

• Clicking Sync in the UI to promote. The UI sync button reconciles the current Git state onto the cluster. It does not change desired state, so it is not a promotion — and if you find yourself clicking it to “push a change to prod”, the change came from somewhere other than a reviewed commit. That is drift with extra steps.
• kubectl edit/kubectl set image on the prod deployment. This changes the live state, not the desired state. Argo CD will faithfully report OutOfSync and, if self-heal is on, revert your change. You didn’t promote; you started a fight with the controller.

The full set of pretenders, and their honest replacement:

Promotion is a diff in Git. Everything else in this lesson is about making that diff small, legible, correct, and hard to get wrong.

---

What actually moves — and what stays put

If promotion is “edit the file”, the sharpest question is: edit which line? The answer is the crux of doing this safely. You promote a reference to an immutable artifact. You do not promote the artifact, and you do not promote environment-specific configuration.

An image built by CI is content — a set of layers, addressable by a cryptographic digest (sha256:…). That content is immutable: registry.example.com/web@sha256:9f2c… means exactly one set of bytes, forever. A tag (:2.3.1) is a human-friendly label pointing at a digest. Promotion moves that pointer from environment to environment; the bytes it points at never change. This is why “the same artifact runs everywhere” is achievable at all.

Pinning is what makes “same artifact everywhere” a guarantee rather than a hope. Each artifact type has a way to pin it so a promotion is deterministic:

The rows that trip teams up are the env-specific config and secrets rows — the things that stay put. Beginners imagine promotion as “copy staging’s whole config to prod.” It is the opposite: only the artifact reference crosses the boundary; everything env-specific is a permanent, reviewed difference that lives in that environment and never travels. Prod’s replica count, prod’s ingress hostname, prod’s HPA ceiling, prod’s secret-store name — these are supposed to differ, forever. Promotion must move the tag without dragging staging’s replica count into prod.

That is exactly what a shared base plus per-env overlays gives you, and why the structure you choose (next section) is not a cosmetic preference — it decides whether “move only the tag” is a one-line diff or a merge minefield.

The immutability rule, stated once so you never forget it: the artifact you deploy to prod must be bit-for-bit the artifact you validated in staging, which is the one you first ran in dev. The only correct way to guarantee that is to promote a reference (tag pinned to a digest), and to never rebuild the image for a different environment. A rebuilt image is a different artifact wearing a familiar name.

---

Repo & branch structuring patterns, compared honestly

Where you put per-environment config decides everything downstream: how a promotion looks as a diff, how badly environments drift, and how big the blast radius of a mistake is. There are three patterns in the wild. Two are good for different reasons; one is a trap that many teams learn about the hard way.

Pattern 1 — Directory-per-environment (the recommended default)

One branch (main), one directory per environment, each pointing at a shared base:

gitops-repo/
  base/                          # ONE definition of the app — shared by all envs
    kustomization.yaml
    deployment.yaml
    service.yaml
  envs/
    dev/
      kustomization.yaml         # base + dev overlay (replicas:1, image tag)
    staging/
      kustomization.yaml         # base + staging overlay (replicas:3, image tag)
    prod/
      kustomization.yaml         # base + prod overlay (replicas:12, image tag, HPA)

Promotion is copying the tag from one env’s file to the next, via a PR against main. The tag that dev proved gets written into envs/staging/kustomization.yaml; later, into envs/prod/kustomization.yaml. Each environment is a directory Argo CD watches with a separate Application. This is the pattern most teams converge on because the promotion diff is a single, obvious line and every environment lives on one branch you can grep in one place.

Pattern 2 — Branch-per-environment (fallen out of favour)

A long-lived branch per environment — dev, staging, prod — and each Argo CD Application tracks its own branch (targetRevision: prod). Promotion is a merge or cherry-pick from staging into prod.

It sounds elegant (“prod is just the prod branch”) and it is how many teams started, but it has three structural problems that compound at scale:

• Silent drift is the default. Environment-specific differences (replica counts, hostnames) live as commits on each branch. Over months, staging and prod accumulate divergent history, and nobody can tell at a glance what’s actually different. The branches quietly stop being comparable.
• Merge hell. Promotion-by-merge drags every difference between branches, not just the tag you meant to promote. You either get merge conflicts on the env-specific bits, or — worse — you accidentally promote a staging-only config change into prod along with the tag. Cherry-picking one commit avoids that but turns promotion into a manual archaeology exercise.
• The diff lies. A merge from staging to prod shows dozens of unrelated changes, so reviewers rubber-stamp it. The review — the whole point of PR-based promotion — becomes noise.

Branch-per-env is not wrong for a tiny single-config app, but for anything with real per-environment differences it turns the reviewed-diff superpower into a liability. This is why the community broadly moved to directory-per-env.

Pattern 3 — Overlay-per-environment (the mechanism, not a rival)

Strictly, “overlay-per-env” is how you express the per-env differences inside Pattern 1 (or even Pattern 2): Kustomize bases + overlays, or Helm with a shared chart and per-environment values files. It is not really a third repo layout so much as the templating engine that keeps directory-per-env DRY. We lean on it heavily in the drift section, and the mechanics live in the dedicated lessons: Kustomize integration and, for the fan-out, ApplicationSets & generators.

Here is the honest comparison — the table to screenshot when your team argues about this:

The recommendation the rest of this lesson assumes: directory-per-env on a single branch, with a Kustomize (or Helm) shared base, and per-environment overlays that express only the delta. It gives you the small, legible promotion diff that makes review meaningful, keeps every environment comparable, and is exactly what the promotion tooling expects.

If you are still weighing repo topology at the org level — one monorepo versus many — that is its own decision with its own trade-offs, covered in Monorepo vs Polyrepo GitOps. This lesson is orthogonal to it: directory-per-env works inside a monorepo or a per-app config repo.

---

Avoiding config drift between environments (the #1 promotion problem)

Ask any platform engineer for their worst deploy story and a shocking fraction reduce to four words: staging and prod diverged. A value someone changed in prod months ago during an incident and never wrote down. A feature flag flipped in staging for a demo. A resource limit bumped in prod but not staging. Individually trivial; together they mean “passed in staging” stops predicting “works in prod”. This silent divergence is config drift between environments, and it is the single biggest reason promotions fail even when the artifact is correct.

The fix is structural, not disciplinary. You cannot solve drift by asking people to be careful; you solve it by making every environment difference explicit, minimal, and reviewed. Concretely: one shared base, and per-environment overlays that contain only the delta.

Drift almost always enters through the same handful of doors — know them so you can shut them:

Here is the shared base — one definition of the app, identical for every environment:

# base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml
  - service.yaml
commonLabels:
  app.kubernetes.io/name: web

# base/deployment.yaml — the SHARED shape; env-specific numbers are NOT here
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 1                     # a safe default; overlays override per env
  selector:
    matchLabels:
      app.kubernetes.io/name: web
  template:
    metadata:
      labels:
        app.kubernetes.io/name: web
    spec:
      containers:
        - name: web
          image: registry.example.com/web   # tag set by the overlay, not here
          ports:
            - containerPort: 8080
          resources:
            requests:
              cpu: 100m
              memory: 128Mi

Now each environment overlay expresses only what genuinely differs. Dev:

# envs/dev/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: web-dev
resources:
  - ../../base
images:
  - name: registry.example.com/web
    newTag: "2.3.1"               # the promoted reference — dev has it first
# dev keeps base defaults: replicas 1, small resources

Prod:

# envs/prod/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: web-prod
resources:
  - ../../base
images:
  - name: registry.example.com/web
    newTag: "2.3.0"               # prod still on the previous version — until promoted
replicas:
  - name: web
    count: 12                     # the ONLY structural prod difference, made explicit
patches:
  - path: hpa.yaml                # prod-only HorizontalPodAutoscaler, visible in Git

The magic is what you cannot do here: you cannot let prod drift, because there is nowhere for an unreviewed difference to hide. If prod needs 12 replicas, that is a line in envs/prod/kustomization.yaml that someone approved. If someone kubectl edits it to 20 during an incident, Argo CD reports OutOfSync and (with self-heal) reverts it — the drift is loud, not silent. The base guarantees the two environments share a shape; the overlay makes every deviation a reviewed artifact.

Use this rule to decide where any given setting belongs:

The anti-drift test: can you run diff <(kustomize build envs/staging) <(kustomize build envs/prod) and explain every line of the output by pointing at an overlay? If yes, you have no drift — only intended, reviewed differences. If a line surprises you, you just found your next incident early. Wire that diff into CI and drift becomes a failing check, not a postmortem.

If an overlay starts to look like a near-complete copy of the base, your base is under-parameterised — pull the shared parts up. The override file is the diff a reviewer approves; keep it small enough to read in ten seconds.

---

PR-based promotion: the actual flow

Now we assemble the pieces into the flow a real team runs. The shape is always the same, and CI does the building while Git + Argo CD do the promoting — the handoff is the whole art.

Read the pipeline left → right. The image is built once and is immutable; from there, promotion is a sequence of pull requests moving that one reference through gated environments, with Argo CD reconciling each environment’s file onto its cluster.

The six badges mark where teams win or lose: promote the immutable artifact rather than rebuild it (1); a shared base so environments never drift silently (2); dev auto-syncs and is disposable (3); staging is the review gate where the canary runs (4); prod demands approval and a sync window with manual sync (5); and the fan-out to regions goes one at a time, not all at once (6). Let’s walk each stage.

The flow, step by step

Notice the asymmetry that makes this safe: the gate tightens as you move right. Dev auto-syncs (speed). Staging needs a review (a human confirms the tag). Prod needs a named approver and a maintenance window and a human on the sync button (defence in depth). The same artifact flows through all three; only the ceremony changes.

The promotion PR — what a reviewer actually sees

This is the payoff of directory-per-env plus a shared base. Promoting 2.3.1 from staging to prod is this diff, and nothing else:

# PR: "Promote web 2.3.1 to prod"  (edits envs/prod/kustomization.yaml)
   images:
     - name: registry.example.com/web
-      newTag: "2.3.0"
+      newTag: "2.3.1"

A reviewer approves that in seconds and knows exactly what changes: the prod reference moves from the version prod runs today to the version staging just proved. No replica counts, no hostnames, no surprises — because those live in the overlay and don’t move. Harden it further by pinning the digest, so the tag can never be re-pointed under you:

# envs/prod/kustomization.yaml — digest-pinned promotion (belt and braces)
images:
  - name: registry.example.com/web
    newName: registry.example.com/web
    digest: sha256:9f2c4b8e1d3a6c7f0b2e5d8a1c4f7b0e3d6a9c2f5b8e1d4a7c0f3b6e9d2a5c8f1

The gates, per environment

The controls are not vibes; they are concrete settings you configure once. Here is the matrix:

Branch protection + CODEOWNERS are your Git-side gate. A CODEOWNERS file makes the right people mandatory reviewers on the right paths:

# .github/CODEOWNERS — path-scoped promotion approval
/base/                 @acme/platform-team
/envs/dev/             @acme/web-developers
/envs/staging/         @acme/web-developers @acme/qa
/envs/prod/            @acme/platform-team  @acme/sre-oncall

Now a PR that edits envs/prod/ cannot merge without a platform-team and an SRE approval — enforced by the forge, not by etiquette. Combine with “require pull request before merging” and “dismiss stale approvals” on the protected branch and the Git gate is airtight.

The branch-protection settings that make CODEOWNERS actually bite (configure once on the protected branch / prod path):

Manual sync for prod is simply the absence of an automated block on the prod Application. A merged prod PR then sits as OutOfSync until a human runs argocd app sync web-prod. That deliberate human step is a feature: the merge is reviewed and the apply is intentional.

Sync windows add when. Configured on the AppProject, they restrict the times a sync may happen at all — even a manual one:

# AppProject with a prod change window
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: prod
  namespace: argocd
spec:
  sourceRepos:
    - https://github.com/acme/gitops-repo.git
  destinations:
    - server: https://prod-eu.example.com
      namespace: 'web-prod'
    - server: https://prod-us.example.com
      namespace: 'web-prod'
  syncWindows:
    - kind: allow
      schedule: '0 14 * * 2,4'      # Tue & Thu, 14:00
      duration: 2h                   # a 2-hour change window
      applications:
        - 'web-prod'
      manualSync: true               # humans may sync inside the window
      timeZone: 'Europe/London'
    - kind: deny
      schedule: '0 0 * * *'          # a standing deny...
      duration: 24h                  # ...covering all other times
      applications:
        - 'web-prod'
      manualSync: false              # ...that even blocks manual sync

With that project, an attempt to sync web-prod outside Tuesday/Thursday afternoons is rejected — the change waits for the window. Inside the window, the on-call engineer runs the sync deliberately.

The syncWindows fields, for reference:

Automating the paperwork — Image Updater and Kargo

Opening promotion PRs by hand is fine at low volume, but three tools automate the boilerplate. Compare them honestly:

Argo CD Image Updater is the right tool for the first hop: point it at dev, and every time CI pushes a new tag matching your constraint, it commits that tag into envs/dev (write-back-method git), and Argo CD syncs it. Dev advances with zero human paperwork. It is not a promotion engine, though — it keys off the registry, not off “staging is healthy”, so it does not model dev → staging → prod on its own. Use it to feed the pipeline, not to run it.

Kargo is the emerging, purpose-built answer to exactly the problem this lesson describes. Instead of you scripting PRs, Kargo models the pipeline as first-class objects:

A Kargo Warehouse subscribing to your registry looks roughly like this (schema evolves across Kargo versions — treat as illustrative of the model, and check the docs for your version):

# Illustrative Kargo Warehouse — subscribes to the image, emits Freight on new versions
apiVersion: kargo.akuity.io/v1alpha1
kind: Warehouse
metadata:
  name: web
  namespace: web
spec:
  subscriptions:
    - image:
        repoURL: registry.example.com/web
        semverConstraint: ^2.0.0        # only 2.x tags become Freight
        discoveryLimit: 20

Kargo then promotes that Freight dev → staging → prod, running your verification (a Rollouts AnalysisRun) at each stage and writing the tag into the right env path via Git — the pipeline this lesson builds by hand, expressed declaratively. The honest trade-off: it’s another controller and a new mental model to operate, and the API is still maturing, so adopt it when hand-rolled PR automation becomes the bottleneck, not before.

---

Progressive promotion: canary in staging, and fan-out across regions

Two refinements turn a competent pipeline into a resilient one: catch bad versions with a canary before prod, and never hit all of prod at once.

Canary in staging before prod

Promotion answers “which artifact”; progressive delivery answers “how carefully do we switch to it”. They compose: promote the tag into staging, then let Argo Rollouts shift traffic to it gradually while watching metrics, so a bad version aborts in staging and never earns its prod PR. The full mechanics are in Argo Rollouts canary & blue-green; here is the shape that matters for promotion:

# envs/staging uses a Rollout instead of a bare Deployment: canary with an analysis gate
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: web
spec:
  strategy:
    canary:
      steps:
        - setWeight: 20
        - pause: {duration: 5m}
        - analysis:                       # metric gate: abort if bad
            templates:
              - templateName: success-rate
        - setWeight: 50
        - pause: {duration: 5m}
        - setWeight: 100

The promotion-level point: staging is where the canary earns its keep. A version that degrades error rate or latency aborts here — automatically, via the analysis step — and the prod promotion PR is never opened. By the time a change reaches prod, it has been proven both functionally (it works) and progressively (it can be rolled forward without a spike). Prod can then run its own canary or blue-green as a final safety net.

Fan-out across clusters and regions

Production is rarely one cluster. You promote to prod-eu, watch it, then prod-us — never both on the same reconcile tick, or a bad change takes out every region simultaneously. An ApplicationSet with a cluster generator renders one prod Application per region from the same prod overlay (so regions can’t drift from each other), and a RollingSync strategy sequences them:

# One ApplicationSet fans the prod overlay across regions — gated, one at a time
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: web-prod
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  strategy:
    type: RollingSync
    rollingSync:
      steps:
        - matchExpressions:
            - key: region
              operator: In
              values: [eu]          # eu goes first, must be Healthy...
        - matchExpressions:
            - key: region
              operator: In
              values: [us]          # ...before us starts
  generators:
    - clusters:
        selector:
          matchLabels:
            env: prod               # every registered prod cluster
  template:
    metadata:
      name: 'web-prod-{{index .metadata.labels "region"}}'
    spec:
      project: prod
      source:
        repoURL: https://github.com/acme/gitops-repo.git
        targetRevision: main
        path: envs/prod             # the SAME overlay for every region — no drift
      destination:
        server: '{{.server}}'
        namespace: web-prod
      syncPolicy: {}                # no automated block → manual sync for prod

Two promotion-critical details: every region reads path: envs/prod, so prod-eu and prod-us are structurally identical by construction — you cannot drift one region from another. And RollingSync means a bad promotion is contained to eu while you watch, before us is touched. This is the fan-out failure mode from the troubleshooting table, defused. (The generator mechanics live in the ApplicationSets lesson.)

The regions promote in a strict order, each gating the next — this is the whole safety property of RollingSync:

A subtlety worth internalising: a naïve ApplicationSet list generator applies the same template to every element, so if you generate dev/staging/prod from one list, they’d all get the same sync policy — and prod would auto-sync. Either keep prod in a separate Application/ApplicationSet (what we do above — this set is prod-only), or use spec.templatePatch to give prod a different syncPolicy. Fanning one template across tiers without differentiating the gate is exactly how “ApplicationSet hit prod too early” incidents happen.

---

Secrets & config that differ per environment

This is the one genuinely cloud-specific edge in promotion, so all three clouds get covered. Everything else in this lesson is cloud-neutral — a tag bump is identical on AKS, EKS and GKE. Secrets are not, because the store lives in the cloud.

The rule from the “what moves” section stands: you never promote a secret value, and secret values never live in Git. What differs per environment is which secret store an environment points at — dev reads dev secrets, prod reads prod secrets — and that pointer is a reviewed line in the overlay. The External Secrets Operator (ESO) is the clean way to express it: an ExternalSecret names a per-env store; the store lives in each cloud’s secret manager. (Depth in Secrets: Sealed Secrets, ESO, SOPS & Vault.)

# envs/prod overlay adds an ExternalSecret pointing at the PROD store (name differs per env)
apiVersion: external-secrets.io/v1beta1        # v1 is now GA; v1beta1 remains widely deployed
kind: ExternalSecret
metadata:
  name: web-secrets
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: prod-store                            # dev overlay says dev-store; that's the ONLY delta
    kind: ClusterSecretStore
  target:
    name: web-secrets
  data:
    - secretKey: db-password
      remoteRef:
        key: prod/web/db                        # per-env path in the cloud store
        property: password

The per-env difference is secretStoreRef.name (and the remoteRef.key path) — one reviewed line, never a value. The store itself is backed by the cloud, and that is where the three clouds diverge:

Per-env ClusterSecretStore, one per cloud (paired blocks — the fields genuinely differ):

# AKS — prod store backed by Azure Key Vault (Workload Identity)
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: prod-store
spec:
  provider:
    azurekv:
      authType: WorkloadIdentity
      vaultUrl: "https://web-prod-kv.vault.azure.net"
      serviceAccountRef:
        name: eso-prod
        namespace: external-secrets

# EKS — prod store backed by AWS Secrets Manager (IRSA / Pod Identity)
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: prod-store
spec:
  provider:
    aws:
      service: SecretsManager
      region: eu-west-1
      auth:
        jwt:
          serviceAccountRef:
            name: eso-prod
            namespace: external-secrets

# GKE — prod store backed by Google Secret Manager (Workload Identity)
apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: prod-store
spec:
  provider:
    gcpsm:
      projectID: "acme-web-prod"
      auth:
        workloadIdentity:
          clusterLocation: europe-west1
          clusterName: prod-eu
          serviceAccountRef:
            name: eso-prod
            namespace: external-secrets

The promotion consequence: because each environment’s overlay names its own store, a promoted tag lands in prod and the prod pods resolve prod secrets automatically — no secret ever travelled, no value ever touched Git, and dev’s credentials can never leak into prod because dev’s overlay never referenced the prod store. The image reference you promoted is cloud-neutral; the secret store it lands next to is cloud-specific and per-env. That separation is the whole trick.

---

Hands-on lab

You will build a complete dev → staging → prod promotion at the config level — a directory-per-env repo, a shared base, three Applications with prod on manual-sync inside a sync window, and then walk an actual promotion as a series of PRs, watching the diff shrink to one line. It runs on a free local kind cluster; nothing here bills. We simulate the three “environments” as three namespaces on one cluster so anyone can follow — the manifests are identical to a real three-cluster setup except the destination.server values.

No real cluster is required to read this and learn the structure. If you do run it, the outputs shown are representative — exact strings vary by version. We do not promote by rebuilding anything; the “artifact” is a stock public image whose tag we move.

Step 0 — Prerequisites

# kind, kubectl, and the Argo CD CLI installed; then a throwaway cluster:
kind create cluster --name promo
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
kubectl -n argocd rollout status deploy/argocd-server   # wait until Available

What just happened: a local cluster with Argo CD installed. This is the control plane that will reconcile our three “environments”.

Step 1 — Lay out the directory-per-env repo

Create this structure and push it to a Git repo Argo CD can read (any public repo works for the lab; substitute your URL below):

gitops-lab/
  base/
    kustomization.yaml
    deployment.yaml
  envs/
    dev/kustomization.yaml
    staging/kustomization.yaml
    prod/kustomization.yaml

# base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml

# base/deployment.yaml — a stock image so the lab needs no CI
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 1
  selector:
    matchLabels: {app: web}
  template:
    metadata:
      labels: {app: web}
    spec:
      containers:
        - name: web
          image: hashicorp/http-echo      # tag is set per-env by the overlay
          args: ["-text=hello"]
          ports: [{containerPort: 5678}]

# envs/dev/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: web-dev
resources: [../../base]
images:
  - name: hashicorp/http-echo
    newTag: "0.2.3"          # dev gets the new version first

# envs/staging/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: web-staging
resources: [../../base]
images:
  - name: hashicorp/http-echo
    newTag: "0.2.1"          # staging still on the old version
replicas:
  - {name: web, count: 3}    # explicit, reviewed per-env difference

# envs/prod/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: web-prod
resources: [../../base]
images:
  - name: hashicorp/http-echo
    newTag: "0.2.1"          # prod still on the old version too
replicas:
  - {name: web, count: 5}

kubectl create namespace web-dev
kubectl create namespace web-staging
kubectl create namespace web-prod

What just happened: one base, three overlays. Dev is ahead (0.2.3); staging and prod trail (0.2.1). The per-env deltas (replica counts) are explicit lines. This is the no-drift structure in miniature.

Step 2 — The AppProject with a prod sync window

# project-prod.yaml
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: prod
  namespace: argocd
spec:
  sourceRepos: ['*']
  destinations:
    - {server: 'https://kubernetes.default.svc', namespace: 'web-prod'}
  clusterResourceWhitelist: []            # deny cluster-scoped resources (least privilege)
  namespaceResourceWhitelist:
    - {group: '*', kind: '*'}
  syncWindows:
    - kind: allow
      schedule: '0 14 * * 2,4'            # Tue & Thu 14:00, 2h window
      duration: 2h
      applications: ['web-prod']
      manualSync: true

kubectl apply -f project-prod.yaml

What just happened: prod lives in its own AppProject with a sync window — outside Tue/Thu afternoons, even a manual sync is blocked (we’ll see the block, then override it for the lab). (Project boundaries in depth: AppProjects & multi-tenancy.)

Step 3 — Three Applications, three sync policies

The heart of the lab: dev auto-syncs, staging auto-syncs, prod is manual.

# apps.yaml — one file, three Applications
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata: {name: web-dev, namespace: argocd}
spec:
  project: default
  source:
    repoURL: https://github.com/YOU/gitops-lab.git
    targetRevision: main
    path: envs/dev
  destination: {server: https://kubernetes.default.svc, namespace: web-dev}
  syncPolicy:
    automated: {prune: true, selfHeal: true}    # dev: fully automated
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata: {name: web-staging, namespace: argocd}
spec:
  project: default
  source:
    repoURL: https://github.com/YOU/gitops-lab.git
    targetRevision: main
    path: envs/staging
  destination: {server: https://kubernetes.default.svc, namespace: web-staging}
  syncPolicy:
    automated: {prune: true, selfHeal: true}    # staging: automated (add a canary in real life)
---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata: {name: web-prod, namespace: argocd}
spec:
  project: prod                                 # the project WITH the sync window
  source:
    repoURL: https://github.com/YOU/gitops-lab.git
    targetRevision: main
    path: envs/prod
  destination: {server: https://kubernetes.default.svc, namespace: web-prod}
  syncPolicy: {}                                # NO automated block → prod is manual-sync

kubectl apply -f apps.yaml
argocd app list           # representative:
# NAME         SYNC STATUS   HEALTH    PROJECT
# web-dev      Synced        Healthy   default
# web-staging  Synced        Healthy   default
# web-prod     OutOfSync     Missing   prod        <- manual: nothing applied yet

What just happened: dev and staging reconciled themselves. Prod shows OutOfSync/Missing because it has no automated block — it will not apply until a human syncs it. That is the manual-sync gate, working.

argocd app sync web-prod        # first prod sync, inside the lab
# (If outside the window you'd see a block — see Step 6.)
argocd app get web-prod         # now Synced/Healthy on 0.2.1

The three Applications differ only in path, project, and sync policy — the gate made concrete:

Step 4 — Promote 0.2.3: dev → staging (the review PR)

In a real repo this is a PR; here we make the same edit. Change staging’s tag to match dev:

# PR #1: "Promote web 0.2.3 to staging"  (envs/staging/kustomization.yaml)
   images:
     - name: hashicorp/http-echo
-      newTag: "0.2.1"
+      newTag: "0.2.3"

git commit -am "Promote web 0.2.3 to staging" && git push
# staging auto-syncs (or: argocd app sync web-staging)
argocd app get web-staging      # representative: Synced, image now 0.2.3
kubectl -n web-staging get deploy web -o jsonpath='{..image}{"\n"}'
# hashicorp/http-echo:0.2.3

What just happened: the promotion to staging was a one-line diff copying the exact tag dev proved. In production this PR would require a CODEOWNERS review before merge. Staging now runs 0.2.3; prod still runs 0.2.1.

Step 5 — Promote 0.2.3: staging → prod (the approval PR)

# PR #2: "Promote web 0.2.3 to prod"  (envs/prod/kustomization.yaml)
   images:
     - name: hashicorp/http-echo
-      newTag: "0.2.1"
+      newTag: "0.2.3"

git commit -am "Promote web 0.2.3 to prod" && git push
argocd app get web-prod
# representative: SYNC STATUS: OutOfSync   (merged, but prod is manual — not applied)

What just happened: the prod desired state now says 0.2.3, but the cluster still runs 0.2.1. Prod is OutOfSync and stays that way until a human syncs — the merge is done, the apply is a separate, deliberate act. In real life PR #2 required named reviewers (platform + SRE) to merge at all.

Step 6 — Sync prod, and meet the window

argocd app sync web-prod
# If run OUTSIDE the Tue/Thu 14:00–16:00 window, representative:
# FATA[0000] rpc error: code = FailedPrecondition desc = Cannot sync web-prod:
#            an active sync window blocks this sync

The sync window is doing its job: even a human, with an approved merge, cannot apply to prod outside the change window. To complete the lab regardless of the day, widen the window to “now” (edit the schedule/duration in project-prod.yaml to cover the current time, or temporarily add - {kind: allow, schedule: '* * * * *', duration: 1h, applications: ['web-prod'], manualSync: true}), re-apply the project, then:

argocd app sync web-prod        # inside the window now
argocd app get web-prod         # Synced/Healthy, image 0.2.3
kubectl -n web-prod get deploy web -o jsonpath='{..image}{"\n"}'
# hashicorp/http-echo:0.2.3

What just happened: the same artifact (0.2.3) that started in dev is now in prod, having passed a review gate (staging PR), an approval gate (prod PR), and a timing+manual gate (sync window + human sync). It was never rebuilt and never applied by hand — every hop was a reviewed Git change reconciled by Argo CD.

Step 7 — Prove there’s no drift

# Every env difference is explainable from the overlays — nothing hidden:
diff <(kubectl kustomize envs/staging) <(kubectl kustomize envs/prod)
# representative: only namespace + replica count differ (5 vs 3) — both are reviewed overlay lines

What just happened: the only differences between staging and prod are the ones you wrote in the overlays. There is no silent drift, because there is nowhere for it to hide.

Teardown

kind delete cluster --name promo      # removes everything; nothing was billable

You have built and run a promotion pipeline: directory-per-env, shared base, per-env sync policies, a prod sync window, and a real dev → staging → prod walk where the promotion diff was one line and the artifact never changed.

---

Going deeper

You have the working pipeline. This section fills the four gaps teams hit after the basics work — none of it re-teaches the flow above: the promotion mechanism (distinct from repo layout), the rendered-manifests pattern that makes prod diffs exact, what Kargo does step-by-step, and the audit trail you get for free.

Promotion mechanisms, compared (a different axis than repo layout)

The repo-structuring section answered where per-env config lives. This answers a different question: what actually performs the reference-move, and where is the gate? Three mechanisms show up in the wild, and they compose rather than compete.

These are layers, not alternatives: the production-grade setup runs all three at once — a PR-based review gate whose merged commit is a direct Git write into a directory-per-env repo, with an ApplicationSet generating the per-env Applications. A Git-directory generator that turns “add an environment” into “add a directory” looks like this:

# ApplicationSet that creates one Application per envs/<name> directory
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: web-envs
  namespace: argocd
spec:
  goTemplate: true
  generators:
    - git:
        repoURL: https://github.com/acme/gitops-repo.git
        revision: main
        directories:
          - path: envs/*            # dev, staging, prod each become an Application
  template:
    metadata:
      name: 'web-{{.path.basename}}'   # web-dev, web-staging, web-prod
    spec:
      project: default
      source:
        repoURL: https://github.com/acme/gitops-repo.git
        targetRevision: main
        path: '{{.path.path}}'
      destination:
        server: https://kubernetes.default.svc
        namespace: 'web-{{.path.basename}}'
      # syncPolicy omitted here → apply a templatePatch to make prod manual

That single object keeps App definitions DRY — but the caveat from the fan-out section holds: without a templatePatch, prod inherits dev’s syncPolicy and would auto-sync. Generator convenience never removes the need to gate prod differently.

The rendered-manifests (hydrated-branch) pattern — exact prod diffs

A subtlety bites teams months in. With Kustomize overlays or Helm values, what a reviewer sees in a promotion PR is the overlay delta — newTag: "2.3.1". But what Argo CD ultimately applies is the rendered output, and that depends on the base, the chart version, and even the Kustomize/Helm version at sync time. So a change to base/deployment.yaml, or a chart bump, can alter prod’s rendered manifests without any edit to the prod overlay — an invisible promotion. argocd app diff shows the rendered difference, but the PR does not, and the PR is where your review and audit live.

The fix is the rendered-manifests pattern (a.k.a. the hydrated-manifests pattern): CI — or Argo CD itself — renders each environment’s fully-hydrated, plain YAML and commits it to a separate branch or path. Argo CD then points at that hydrated output, so what it syncs is exactly what was rendered and reviewed, with no client-side templating at apply time.

The repo grows a second, machine-written branch:

# Two branches, two audiences:
main                         # the DRY branch — humans edit base/ + overlays here (PRs land here)
environments/dev             # HYDRATED — CI-written plain YAML, no kustomize/helm needed to read it
environments/staging         # HYDRATED
environments/prod            # HYDRATED — Argo CD's Application tracks THIS

A promotion PR still lands on main (bump newTag), CI runs kustomize build envs/prod > out.yaml and pushes the result to environments/prod, and the hydrated diff is exact: every changed line of final YAML is visible. A base or chart change that touches prod now shows up as a prod hydrated-diff even though the overlay line never moved — the invisible promotion becomes visible, in plain YAML an auditor can read without any rendering engine. (See also Diffing, drift & ignoreDifferences for the reconcile-time diff.)

Argo CD ships a first-class version as the Source Hydrator (alpha from 2.14, evolving through 3.x — treat the schema as still-moving). Instead of a separate CI job, Argo CD reads a dry source, hydrates it, writes a hydrated branch, and syncs from there:

# Source Hydrator (ALPHA — schema may change): Argo CD hydrates dry → branch → sync
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: web-prod
  namespace: argocd
spec:
  project: prod
  sourceHydrator:
    drySource:                       # where humans edit (base + overlay)
      repoURL: https://github.com/acme/gitops-repo.git
      targetRevision: main
      path: envs/prod
    syncSource:                      # where Argo CD SYNCS from (hydrated plain YAML)
      targetBranch: environments/prod
      path: .
    hydrateTo:                       # where hydrated output is written
      targetBranch: environments/prod
  destination:
    server: https://kubernetes.example.com
    namespace: web-prod
  # no automated block → prod stays manual-sync

Trade-offs, stated honestly: two branches to reason about, CI (or the hydrator) must be trusted and its runs reproducible, and the repo carries generated YAML. For anything where “what exactly runs in prod” must be answerable to the byte — regulated shops, large fleets, teams burned by a silent base change — the exactness is worth it. For a two-service startup it may be more machinery than the problem needs; the pattern scales down poorly and up extremely well.

What Kargo actually does, step by step

The lesson already introduced Kargo’s model (Warehouse → Freight → Stage, with verification). The piece worth adding is the mechanics: a Kargo Stage promotes Freight by running an ordered list of promotion steps, which is just the PR-writing you did by hand, expressed declaratively:

# Illustrative Kargo promotionTemplate steps (schema evolves across versions)
spec:
  promotionTemplate:
    spec:
      steps:
        - uses: git-clone
          config: {repoURL: https://github.com/acme/gitops-repo.git, checkout: [{branch: main, path: ./repo}]}
        - uses: kustomize-set-image      # the tag bump — the one-line diff, automated
          config: {path: ./repo/envs/staging, images: [{image: registry.example.com/web}]}
        - uses: git-commit
          config: {path: ./repo, message: "promote web to staging"}
        - uses: git-push
          config: {path: ./repo}
        - uses: argocd-update             # nudge Argo CD to sync the target app
          config: {apps: [{name: web-staging}]}

Two nuances that matter for gating: Kargo verifies Freight by running an Argo Rollouts AnalysisRun (see Rollouts analysis & auto-rollback) before that Freight is eligible for the next Stage, and auto-promotion is off by default — a human (or policy) must approve a piece of Freight into a Stage, which is your prod approval gate modelled as a first-class object rather than a branch-protection rule. The cost is another controller and a maturing API — adopt it when hand-rolled automation is the bottleneck, not before.

The audit trail: Git history is the promotion record

A quietly enormous benefit falls out of “promotion is a commit”: you never build a separate deployment log, because Git already is one. The exact production history of an app is one command:

git log --oneline -- envs/prod/kustomization.yaml
# 9f2c4b8 Promote web 2.3.1 to prod   (approved by @sre-oncall, @platform-lead)
# 3a7d1e0 Promote web 2.3.0 to prod
# c81b2f5 Promote web 2.2.9 to prod
git blame envs/prod/kustomization.yaml   # who moved prod to its current tag, and when

Correlate three records and you have complete change-management evidence, generated for free: the PR (what changed and who approved it), the merge commit (when it entered desired state), and argocd app history web-prod (when the controller actually applied it). For SOC 2 / PCI / ISO change controls, that triad is the audit — no CMDB, no spreadsheet, no “who deployed this?” archaeology. git revert of a promotion PR is therefore an auditable rollback: the reversal is itself a reviewed commit with its own approver.

---

Common mistakes and troubleshooting

Promotion failures cluster into a predictable set. Keep this table close; each row is a real incident pattern.

You will watch the Application status move through a small set of states as a promotion travels — read them fluently:

Three gotchas deserve extra words, because they cause the most damage:

1. Promoting a rebuilt image (the silent artifact swap). The most dangerous mistake because it looks fine: CI has a “build and deploy to prod” job that rebuilds from source. Even from the same commit, a rebuild can pull newer base-image layers, different dependency patch versions, or a differently-resolved lockfile — a different binary wearing tag 2.3.1. Everything you validated in staging is now unverified in prod. The only defence is architectural: build the image exactly once, in one job, and let every environment reference that one artifact by digest. If your prod deploy job contains docker build, you have this bug.

2. Silent config drift. Prod and staging start identical and diverge one incident at a time — a kubectl edit here, a hotfix there, none written back to Git. Six months later “staging passed” predicts nothing. The structural fix is the shared base (so differences must be explicit) plus self-heal on (so an out-of-band kubectl edit is reverted and reported as OutOfSync, making drift loud). The diff-in-CI check turns drift from a postmortem finding into a failed pipeline.

3. Prod auto-syncing an untested change. Convenience is the enemy here. If prod has automated: {selfHeal: true} and tracks main, then any merge that touches the prod path rolls out with no human in the loop — including a mistaken one, at 2am, with no one watching. Prod earns two locks the other environments don’t: manual sync (no automated block) so a human triggers the apply, and a sync window so even that human can only do it in an agreed slot. Yes, it’s more friction. Prod is where you want friction.

---

Common beginner mistakes

The troubleshooting table above is symptom → cause → fix. This one is different: it is the set of mental-model errors that make promotion feel confusing before it clicks. Each row is a belief a beginner arrives with, why it is wrong, and the model that replaces it. Fix the model and the symptoms above mostly never happen.

If you internalise one line from this table: promotion moves a pointer to an immutable artifact through gated Git edits; everything else stays where it is. Most beginner pain is some variation of trying to move the wrong thing, in the wrong way, at the wrong gate.

---

Practice challenges

Six exercises escalating beginner → advanced. Try each before opening the solution. They build on the lab’s directory-per-env repo; all are safe on a kind cluster or on paper.

1 (Beginner) — Find the single line a promotion changes. Given this prod overlay, which one line does a “promote 2.3.1 to prod” PR edit — and which lines must it leave alone?

# envs/prod/kustomization.yaml
namespace: web-prod
resources: [../../base]
images:
  - name: registry.example.com/web
    newTag: "2.3.0"
replicas:
  - {name: web, count: 12}

<details> <summary>Solution</summary>

Only newTag: "2.3.0" → newTag: "2.3.1". The namespace, resources, and replicas: 12 are permanent, reviewed per-env differences — they do not move during promotion.

Why: promotion moves the artifact reference and nothing else; env-specific config stays put. </details>

2 (Beginner) — Spot the real promotion. Which of these actually promotes a change to staging? (a) kubectl set image deploy/web web=registry.example.com/web:2.3.1 -n web-staging; (b) clicking Sync on web-staging in the UI; © a PR editing envs/staging/kustomization.yaml; (d) helm upgrade from your laptop.

<details> <summary>Solution</summary>

Only ©. (a) and (d) mutate live state with personal creds and leave no reviewed record — Argo CD will report OutOfSync and self-heal may revert them. (b) re-applies the current Git state; it changes no desired state, so it promotes nothing.

Why: under GitOps a promotion is a reviewed change to desired state in Git — never a kubectl, a laptop helm, or a UI sync click. </details>

3 (Intermediate) — Write the drift gate. Add a CI check that fails the build if staging and prod differ in any way you did not intend. What command, and what makes it trustworthy?

<details> <summary>Solution</summary>

# In CI: render both overlays and diff them; a surprising line fails the job
diff <(kubectl kustomize envs/staging) <(kubectl kustomize envs/prod)

Trustworthy only if you can explain every line of the output by pointing at an overlay (e.g. namespace, replica count, image tag). Wire it so any unexplained difference is a red build. Combine with selfHeal: true so out-of-band kubectl edits are reverted and surfaced as OutOfSync.

Why: it turns silent, disciplinary drift into a loud, structural failing check — a pipeline red, not a postmortem. </details>

4 (Intermediate) — Give prod two locks. A teammate’s prod Application has syncPolicy: {automated: {prune: true, selfHeal: true}} and tracks main. Name the two changes that make prod refuse to deploy an unreviewed change, and the Git-side control that backs them.

<details> <summary>Solution</summary>

1. Remove the automated block → syncPolicy: {}, making prod manual-sync (a human must run argocd app sync web-prod).
2. Add a sync window on the prod AppProject (spec.syncWindows) so even that manual sync only lands in an agreed slot.

Backed on the Git side by branch protection + CODEOWNERS on envs/prod/, so the merge also requires named approvers. Merge gate + apply gate + timing gate. (See Sync policies: automated, selfHeal, prune.)

Why: prod needs defence in depth — the reviewed merge and the deliberate apply are two separate, intentional acts. </details>

5 (Advanced) — Make the promotion tamper-proof. Your registry does not enforce tag immutability, so :2.3.1 could be re-pushed under you. Convert a tag-only promotion to one that cannot be silently changed, and say what it prevents.

<details> <summary>Solution</summary>

Pin the digest instead of (or alongside) the tag:

images:
  - name: registry.example.com/web
    digest: sha256:9f2c4b8e1d3a6c7f0b2e5d8a1c4f7b0e3d6a9c2f5b8e1d4a7c0f3b6e9d2a5c8f1

A digest is content-addressed: sha256:… names exactly one set of bytes, forever. Even if someone re-pushes :2.3.1 pointing at different content, prod keeps running the reviewed bytes.

Why: immutability by content-addressing turns “same artifact everywhere” from a hope into a guarantee — a re-pushed mutable tag can no longer change what prod runs. </details>

6 (Advanced) — Design an exact-diff, region-safe prod promotion. Sketch a setup where (a) prod PR diffs show the true rendered change even when the base moves, and (b) a bad promotion can only take out one region before you stop it. Name the mechanisms.

<details> <summary>Solution</summary>

• (a) Exact diffs: adopt the rendered-manifests pattern — CI (or Argo CD’s Source Hydrator) renders envs/prod to fully-hydrated plain YAML on an environments/prod branch; the prod Application syncs from that branch. A base or chart change now surfaces as a concrete hydrated-diff, not an invisible one.
• (b) Region safety: render every region’s prod Application from the same overlay via an ApplicationSet cluster generator (no cross-region drift), and add a strategy.type: RollingSync that steps prod-eu (must be Healthy) before prod-us begins. (See Rollouts analysis & auto-rollback for the health gate.)

Why: the hydrated branch makes the change reviewable to the byte, and RollingSync contains blast radius to one region while you watch — exactness plus containment, the two properties a fleet-scale prod promotion needs. </details>

---

Cheat-sheet

The promotion flow, the structuring decision, and the tooling — dense reference.

The promotion flow (what happens, in order):

Key commands:

The env-structuring decision:

Gates, per environment:

Automation note:

---

Glossary

Every term this lesson uses, in plain language.

• Promotion — moving a change from one environment to the next (dev → staging → prod). Under GitOps it is a reviewed edit to the next environment’s desired state in Git, never a command run against a cluster.
• Desired state — the way things should be, declared in Git. Argo CD continuously makes the cluster (the live state) match it.
• Immutable artifact — a built thing (a container image, a pinned chart) whose bytes never change. You promote a reference to it, not the thing itself.
• Image digest (@sha256:…) — a content address: a hash that names exactly one set of image bytes, forever. The gold standard for pinning a promotion.
• Image tag (:2.3.1) — a human-friendly, mutable label pointing at a digest. Safe to promote only if the registry enforces tag immutability, or you also pin the digest.
• Reference / pointer — the tag or digest an environment’s file names. Promotion moves this pointer; the artifact it points at stays put.
• Config drift — silent divergence between environments (a value changed in one env and never written to Git) that makes “passed in staging” stop predicting “works in prod”.
• Shared base — one definition of the app (Kustomize base/ or a Helm chart) that every environment shares, so environments can’t diverge in shape.
• Overlay — the per-environment file that expresses only the delta from the base (the tag, replica count, hostname). The diff a reviewer approves.
• Kustomize — a template-free tool that layers overlays on a base. images:/replicas: fields express per-env changes without duplicating YAML. (Depth: Kustomize integration.)
• Directory-per-env — one branch, one directory per environment, all sharing a base. The recommended layout: promotion is a one-line diff.
• Branch-per-env — a long-lived branch per environment, promoted by merge/cherry-pick. Fallen out of favour: branches drift and merges drag unrelated changes.
• PR-based promotion — the promotion arrives as a pull request whose approval is the gate. The default for staging and prod.
• CODEOWNERS — a file mapping repo paths to required reviewers, so a PR touching envs/prod/ cannot merge without the platform/SRE teams.
• Branch protection — repo rules (require PR, require reviewers, dismiss stale approvals, required status checks) that make the Git gate enforced by the forge, not by etiquette.
• Manual sync — a prod Application with no automated block: a merged PR sits OutOfSync until a human runs argocd app sync. The deliberate apply step.
• Sync window — AppProject.spec.syncWindows: cron-scheduled allow/deny windows restricting when a sync may happen, even a manual one. deny beats allow.
• Self-heal — syncPolicy.automated.selfHeal: true: Argo CD reverts out-of-band kubectl edits back to Git, making drift loud instead of silent.
• Prune — syncPolicy.automated.prune: true: Argo CD deletes live resources no longer present in Git, so removals promote too.
• ApplicationSet — a controller that generates many Applications from one template plus a generator (git-directory, list, cluster). Removes the toil of hand-authoring per-env or per-region Apps. (Depth: ApplicationSets & generators.)
• RollingSync — an ApplicationSet strategy that syncs generated Apps in ordered steps (e.g. prod-eu must be Healthy before prod-us starts), containing blast radius across regions.
• Canary / AnalysisRun (verification) — a progressive rollout (Argo Rollouts) that shifts traffic gradually while a metric analysis aborts a bad version — the automated quality gate before a version earns its prod PR.
• Rendered-manifests (hydrated-branch) pattern — CI or Argo CD renders each env’s fully-hydrated plain YAML to a separate branch/path, so prod diffs are exact and a base/chart change can’t slip through invisibly.
• Source Hydrator — Argo CD’s built-in (alpha, 2.14+) implementation of the rendered-manifests pattern: it reads a drySource, hydrates it, writes a hydrated branch, and syncs from there.
• Kargo — Akuity’s purpose-built promotion engine. Warehouse subscribes to artifacts and emits immutable Freight; a Stage requests and promotes Freight via ordered steps, with verification and (by default) human approval.
• Image Updater — an Argo CD add-on that watches a registry and writes a new matching tag back to Git. Ideal for auto-advancing dev; not a multi-stage promotion engine. (Depth: Image Updater.)
• External Secrets Operator (ESO) — reads secret values from a cloud store at runtime via an ExternalSecret + a per-env ClusterSecretStore, so values never live in Git or travel between envs. (Depth: Secrets: Sealed Secrets, ESO, SOPS & Vault.)
• git revert — the GitOps rollback: revert the promotion PR to restore the prior desired-state tag, and let Argo CD reconcile the cluster back — auditably, never with kubectl.

---

Interview and exam questions

Q: In one sentence, what is a promotion under GitOps? A: A controlled, reviewed change to an environment’s desired state in Git — almost always a pull request that moves an immutable artifact’s reference (an image tag/digest) into the next environment’s file — after which the pull-based controller (Argo CD) reconciles the cluster to match. It is never a kubectl command or a UI Sync click.

Q: What actually gets promoted, and what must not move? A: You promote a reference to an immutable artifact — an image tag (ideally pinned to a digest) or a pinned chart version. You do not promote the artifact itself (never rebuild per env) and you do not promote environment-specific config (replicas, hostnames, resource limits, secret-store names) — those are permanent, reviewed differences that live in each environment’s overlay and stay put.

Q: Compare directory-per-env and branch-per-env promotion. Why has branch-per-env fallen out of favour? A: Directory-per-env keeps every environment on one branch with a shared base, so promotion is a one-line newTag diff and environments stay comparable. Branch-per-env uses a long-lived branch per environment and promotes by merge/cherry-pick, which (a) lets branches drift silently, (b) drags unrelated env-specific changes on merge (“merge hell”), and © produces noisy diffs that defeat review. Directory-per-env preserves the reviewed-diff superpower; branch-per-env erodes it.

Q: What is config drift between environments, and how do you prevent it structurally? A: Drift is silent divergence between environments (e.g., a value changed in prod during an incident and never written to Git) so that “passed in staging” stops predicting “works in prod”. You prevent it structurally with a shared base plus per-env overlays that contain only the delta — every difference becomes an explicit, reviewed line — reinforced by self-heal (reverts out-of-band kubectl edits) and a CI diff check between rendered environments.

Q: Why is rebuilding the image per environment an anti-pattern? A: Because a rebuild — even from the same commit — can produce a different binary (newer base layers, different resolved dependencies) under the same tag, so what you validated in staging is not what runs in prod. The immutability guarantee (“same artifact everywhere”) only holds if you build once and promote a digest-pinned reference. A prod deploy job containing docker build has this bug.

Q: How do you make prod refuse to auto-deploy an unreviewed change? A: Two locks. Remove the automated block from the prod Application so it is manual-sync (a human must run argocd app sync), and define sync windows on the prod AppProject so syncs are only permitted in an agreed window (with manualSync controlling break-glass). Add branch protection + CODEOWNERS on envs/prod/ so the merge also requires named approvers. The Git merge is gated and the apply is gated.

Q: What are sync windows and how do they interact with manual sync? A: AppProject.spec.syncWindows are cron-scheduled allow/deny windows restricting when syncs may occur. deny overrides allow; if only allow windows exist and none is active, syncs are blocked. Each window’s manualSync: true|false decides whether manual syncs are permitted when the window would otherwise block — so you can allow break-glass manual syncs while blocking automated ones, or lock both.

Q: How does Argo CD Image Updater fit a promotion pipeline, and what are its limits? A: Image Updater watches a registry and, on a new matching tag, writes that tag back to Git (write-back-method git), which Argo CD then syncs. It’s ideal for auto-advancing dev on every build with no manual PR. Its limit: it keys off the registry, not off “staging is healthy”, so it does not model dev→staging→prod on its own. Use it to feed the pipeline’s first hop, not to run the whole promotion.

Q: What is Kargo, and when would you reach for it over hand-rolled PR automation? A: Kargo (by Akuity) is a purpose-built promotion engine that models the pipeline as first-class objects — Warehouses (subscribe to artifacts, emit immutable Freight), Stages (environments that request and promote Freight, with verification via Rollouts AnalysisRuns), and Promotions. Reach for it when multi-stage promotion with verification outgrows scripted PRs; the trade-off is another controller and an evolving API to operate.

Q: How do you promote safely across multiple prod regions? A: Render every region’s prod Application from the same prod overlay via an ApplicationSet cluster generator (so regions can’t drift from each other), and add a RollingSync strategy that sequences them — prod-eu must be Healthy before prod-us begins — so a bad promotion is contained to one region while you watch, instead of hitting the whole fleet on one reconcile tick.

Q: A prod promotion caused an incident. Walk through the rollback. A: git revert the prod promotion PR — that restores the prior desired-state tag in envs/prod, and the reverted change goes through the same review gate. Then sync prod (inside the window, or break-glass) so the cluster returns to the previous artifact. Because desired state is in Git, rollback is a normal reviewed change, not a scramble; the exact prior state is recoverable byte-for-byte. Avoid kubectl/UI rollbacks, which create drift the controller will fight.

Q: How do per-environment secrets work without promoting secret values? A: You never put values in Git or promote them. Each environment’s overlay references its own secret store (e.g., ESO secretStoreRef: dev-store vs prod-store) — a single reviewed line — and the store is backed by the cloud’s secret manager (Azure Key Vault on AKS, AWS Secrets Manager on EKS, Google Secret Manager on GKE). A promoted image lands next to the prod store and resolves prod secrets at runtime; dev’s overlay never references the prod store, so credentials can’t leak across environments.

---

Key takeaways

• Promotion is a reviewed Git change, not a command. Dev, staging and prod are files that declare desired state; you promote by editing the next environment’s file via a PR, and Argo CD reconciles the cluster. Never kubectl, never a UI Sync click.
• Promote the reference to an immutable artifact — and nothing else. Move an image tag (pinned to a digest) or a pinned chart version. Build the image once; never rebuild per env. Env-specific config (replicas, hostnames, secret-store names) stays put as permanent, reviewed differences.
• Directory-per-env with a shared base is the default. It makes promotion a one-line diff and keeps environments comparable. Branch-per-env drags unrelated changes on merge and drifts silently — avoid it for anything with real per-env deltas.
• Config drift is the #1 promotion failure, and the fix is structural. One shared base + overlays that contain only the delta make every difference explicit and reviewed; self-heal + a rendered-diff CI check make any drift loud instead of silent.
• The gate tightens left to right. Dev auto-syncs (speed); staging needs a CODEOWNERS review and a canary; prod needs required reviewers, manual sync, and a sync window. That defence-in-depth is deliberate friction where it counts.
• Progressive delivery and promotion compose. A canary with metric analysis in staging aborts a bad version before it earns a prod PR; a RollingSync ApplicationSet fans prod out region-by-region from one overlay, so no single change hits the whole fleet at once.
• Automate the paperwork carefully. Image Updater auto-advances dev; Kargo is the purpose-built engine (Warehouse → Freight → Stage, with verification) for multi-stage promotion at scale — adopt it when scripted PRs become the bottleneck.
• Secrets are the one cloud-specific edge. Never promote a value; per-env overlays name a per-env store backed by Key Vault (AKS), Secrets Manager (EKS) or Secret Manager (GKE). The artifact you promote is cloud-neutral; the store it lands beside is per-env and per-cloud.