Argo CD HA & Disaster Recovery: Backup, Restore & Rebuilding the Control Plane

In a nutshell

Argo CD disaster recovery sounds terrifying and turns out to be almost boring — and this is the ninety-second version of why. Argo CD does not own your applications; it keeps them matching a description that lives in Git. So if Argo CD, or the entire cluster it runs on, disappears, your running apps keep serving — you have lost the manager, not the workforce. Recovery means standing up a new Argo CD and pointing it back at Git, not restoring a precious database.

Think of Argo CD as the building superintendent. The super keeps every apartment matching the blueprints, fixes anything that drifts, and lets the contractors in. If the super quits overnight, the tenants do not even notice — the lights stay on, the water runs, the building is the building. You have simply lost the person who corrects problems and runs renovations, until you hire a replacement. And hiring one is quick, because the blueprints (your Git repos) are safe in the filing cabinet. The only things you must never lose are the super’s ring of keys and the tenant directory — how to get into each building and who is allowed in. Those are not written on the blueprints, and they are the whole of what you back up: the cluster credentials, the repo credentials, and Argo CD’s own identity config.

Everything else you might instinctively protect is either already in Git (the manifests, and your Application definitions if you are declarative) or genuinely disposable (Redis is a cache Argo CD rebuilds on its own). Get those two facts straight — rebuild from Git; back up only the keys — and the HA topology, the backup command, and the rebuild runbook in the rest of this lesson all fall into place.

Level: Expert · Time: ~62 min

Prerequisites: comfort with kubectl and the argocd CLI, the architecture of Argo CD’s components, the app-of-apps pattern, and how cluster registration stores a cluster Secret. After this you can name exactly which Argo CD state is irreplaceable, run and verify a scheduled argocd admin export, stand up an HA control plane that survives a zone loss, and rebuild the whole control plane on a fresh cluster and re-adopt every running app without a single redeploy.

Most disaster-recovery lessons start by scaring you. This one starts by calming you down, because the single most important fact about recovering Argo CD is that it is far less scary than recovering a database — and understanding why changes every decision you make about HA and backups.

Here is the fact. Argo CD is, for practical purposes, stateless and rebuildable from Git. Its whole job is to reconcile a desired state that lives in your Git repositories into your clusters. That desired state — the manifests, the Helm values, the Kustomize overlays — is not inside Argo CD. It is in Git, versioned and replicated, already backed up by every clone and every mirror. And Argo CD’s control plane is not in the data path: it watches Git and nudges clusters toward it, but no user request to your running application ever passes through the application-controller. So if Argo CD falls over — or the entire cluster hosting it burns down — the Deployments it manages keep serving traffic. You lose reconciliation, drift correction, and the ability to ship new changes, but you do not lose your apps.

That does not make DR free. There is a small, specific set of state that is not in your app Git repos, and if you lose it your rebuilt Argo CD comes up blind — unable to reach its clusters, unable to authenticate a single engineer, unable to prove that the app it sees is the app that is running. This lesson is about exactly that: what that state is, how to keep Argo CD highly available so you rarely need to rebuild, how to back up the irreplaceable bits, and how to rebuild the entire control plane on a brand-new cluster and have it re-adopt every running app without redeploying a thing. If you have worked through cluster bootstrapping — where Terraform creates the cluster and installs Argo CD, then hands off — you already know the rebuild machinery; DR is that machinery pointed at a fresh cluster in anger.

Why this matters

Teams over-engineer Argo CD DR because they reason about it like a stateful application. A database is authoritative — lose it and its backups and the data is gone, because the database is the source of truth — so people carry that fear to Argo CD and imagine they must snapshot etcd, replicate Redis, and quiesce the controller before every backup. Almost none of that is true, and the mismatch wastes both money and confidence.

Argo CD inverts the database model. Git is authoritative; Argo CD is a cache and a reconciler on top of it. The desired state is in Git, the live state is in your clusters’ own etcd, and Argo CD’s Redis is a cache it regenerates by re-cloning the repos and re-listing the clusters. The only genuinely unique state it holds is a short list of connection and identity facts: how to reach each managed cluster, how to authenticate to each Git repo, who your users are, and how they map to permissions. Everything else it can rebuild.

The second load-bearing idea is the control plane / data path split, and it is the reason your on-call rotation can sleep. Argo CD’s components — the application-controller, repo-server, API server, ApplicationSet controller, whose roles you met in Argo CD Architecture: Components, the Repo-Server & the Controller — form a control plane that decides what should exist and makes it so. But a running Pod’s traffic flows through the cluster’s own kube-proxy, ingress, and mesh — never through Argo CD. Pull the entire Argo CD namespace and your checkout service still takes orders. That is categorically unlike an API gateway or a mesh sidecar, which is in the data path and whose failure is an immediate outage.

Hold that table in your head for the rest of the lesson. It tells you where to spend effort (HA so you rarely rebuild; a small, well-encrypted backup of the connection/identity state) and where not to (heroic etcd snapshots and Redis replication for data that is disposable). DR done right for Argo CD is cheap, fast, and boring — which is exactly what you want at 3 a.m.

The insight that shapes everything: Argo CD is (mostly) stateless

Let us make “mostly stateless” precise, because the “mostly” is where the whole lesson lives. Walk through everything Argo CD knows and ask one question of each — if I deleted the entire Argo CD namespace right now, could I get this back without a backup? — and the answers sort into three piles that come back for free, plus one short list that does not:

Every item in that last row is a credential or an access-control fact — the plumbing that lets a fresh Argo CD reach your repos and clusters and let the right humans in. That short list is your entire backup scope, and the next section enumerates it precisely.

Hence a best practice to adopt before you need DR: define your Application, ApplicationSet, and AppProject objects declaratively and commit them to Git. Apps created by clicking around the UI or one-off argocd app create commands exist only in the cluster’s etcd — lose the cluster, lose them. Living in a platform-gitops repo as an app-of-apps, the CRs are in Git too, and recovering them is just re-applying one root object. The declarative discipline you adopt for review and reproducibility also becomes your control-plane backup — not a coincidence, just the GitOps model doing its job.

Now the failure view, the one your incident commander cares about:

The bottom row is the disaster this lesson’s runbook targets, and the right-hand column is why it is survivable. When the hub is gone, your fleet is un-managed but up — you have minutes to an hour to rebuild before the lack of reconciliation bites (a drift no one corrects, a deploy no one can ship). A data-path component’s “gone” means “outage now”; Argo CD hands you a recovery window measured in coffee, not lost revenue per second.

The one way to squander this forgiveness is to put Argo CD in the data path by accident — say, having it host the only copy of a config a running app reads live, or coupling app health to the controller. Keep the boundary clean: Argo CD reconciles; it never serves. If a workload depends on Argo CD being up, you have designed a data-path dependency into a control plane and thrown away its single best property.

What state actually matters: the backup scope

If the backup scope is “credentials and access-control facts,” let us name every object precisely, because a DR plan that misses one of these is a DR plan that fails at the worst moment. Everything below lives in the argocd namespace as either a ConfigMap or a Secret, or as a custom resource.

The cluster Secrets — the critical, sensitive bit

When you register a spoke cluster (cluster registration has its own lesson), Argo CD stores the connection as a Secret labeled argocd.argoproj.io/secret-type: cluster — the only thing that lets the hub reach that spoke. It is a credential, so it is not in your app Git repos; lose it and the rebuilt hub sees the Application objects but cannot connect to the cluster they target, parking every one in Unknown/Failed with a connection error.

apiVersion: v1
kind: Secret
metadata:
  name: prod-eks-cluster
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: cluster   # this label is what makes it a "cluster"
type: Opaque
stringData:
  name: prod-eks
  server: https://ABCD1234.gr7.us-east-1.eks.amazonaws.com
  config: |
    {
      "execProviderConfig": {
        "apiVersion": "client.authentication.k8s.io/v1beta1",
        "command": "aws",
        "args": ["eks", "get-token", "--cluster-name", "prod-eks"]
      },
      "tlsClientConfig": { "caData": "<base64-CA>" }
    }

The config blob carries the auth method — an exec plugin, a bearer token, or client certs. That is genuine secret material, and it is per cloud: AKS spokes use kubelogin, EKS use aws eks get-token, GKE use gke-gcloud-auth-plugin. Back up these Secrets, and make sure the rebuilt hub image actually contains the exec-plugin binaries, or the restored Secret will still fail to authenticate.

The full backup scope, in one table

This is the centerpiece of the lesson — the “what to back up vs what’s already in Git vs what’s disposable” table. Print it; it is your DR checklist.

Three columns, one lesson: the middle column is what Git already protects, the right column is what comes back for free, and the rows where “In your app Git?” is No and “Back it up?” is Yes are your entire DR responsibility — a handful of objects, all in the argocd namespace, all captured by one command. Each maps to a preferred restore source:

Why Redis is genuinely disposable

The last two rows deserve a word, because “disposable” makes people nervous. Redis stores only computed data — rendered manifests, the resource tree, cached diffs, some operation state — every byte of it derivable. Delete Redis and repo-server recomputes manifests, the controller re-lists clusters, and the cache refills. The official docs are blunt: Redis is “a disposable cache and can be safely rebuilt without service disruption.” So you never snapshot it, never restore it, and never panic when it is empty after a rebuild — losing it costs a brief reconcile churn, not one byte of data.

The nuance that trips people: argocd admin export does include the sensitive Secrets — the cluster creds, repo creds, and argocd-secret — in plaintext. That is what makes it a complete backup, and it is also what makes the export file itself a secret. A backup that is one kubectl apply away from restoring your whole control plane is also one leaked S3 object away from handing an attacker every spoke credential you own. Treat the export like the crown-jewel secret it is: encrypt it in transit and at rest, restrict who can read it, and never, ever commit it raw to Git.

HA topology: engineer it so you rarely rebuild

The best disaster recovery is the disaster that never escalates. HA keeps Argo CD serving through node failures, zone outages, and rolling upgrades, so a single dead Pod is a non-event instead of the start of a runbook — the same install from Installing Argo CD: Helm, Manifests, HA & First Login, wired for redundancy. The HA manifests live at manifests/ha/install.yaml (cluster-scoped) and manifests/ha/namespace-install.yaml (namespace-scoped), and the Helm chart’s redis-ha.enabled=true produces the same topology. The difference from the default install is redundancy on every tier plus a real, quorum-based Redis.

# Install HA Argo CD from the upstream manifests (pin to a release, not stable, in prod)
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/v2.13.3/manifests/ha/install.yaml
# NOTE: the HA install schedules 3 redis + 3 haproxy + multi-replica components with
# pod anti-affinity, so it needs at least 3 schedulable nodes across zones.

What “HA” actually changes, component by component

The application-controller is the interesting one: it is a StatefulSet that shards, not a Deployment that merely replicates. Each replica owns a subset of your managed clusters, so scaling it both spreads load and removes the single-controller point of failure. Raise the replica count and set the matching env var so each shard knows the total:

# argocd-application-controller StatefulSet (excerpt) — HA + sharding
spec:
  replicas: 3
  template:
    spec:
      containers:
        - name: argocd-application-controller
          env:
            - name: ARGOCD_CONTROLLER_REPLICAS   # MUST match spec.replicas
              value: "3"
            # sharding strategy: legacy | round-robin | consistent-hashing
            - name: ARGOCD_CONTROLLER_SHARDING_ALGORITHM
              value: "round-robin"

The deeper treatment of why and how to shard — monorepo pressure, repo-server parallelism, the reconcile budget — is its own topic in Scaling Argo CD: Sharding, the Repo-Server & Monorepo Performance. For DR you need only the HA-relevant fact: a single controller replica is a single point of failure for reconciliation (not for your apps, which keep running), so production runs at least two shards.

Redis HA: the one stateful-ish tier, and what losing it means

Redis is the only component with anything resembling a data plane, so HA treats it specially. The redis-ha topology runs three Redis servers (one master, two replicas) each paired with a Sentinel for automatic failover, fronted by three HAProxy Pods that always route clients to the current master. Argo CD’s components connect to HAProxy, not to Redis directly, so a master failover is transparent — Sentinel promotes a replica, HAProxy re-points, and the controller never notices.

The crucial DR framing: losing Redis — even the whole redis-ha set — is not data loss, only cache loss. Worst case, the cold cache produces a short reconcile burst and slightly slower UI until it warms; apps do not restart and no backup is consulted. The only Redis failure that hurts is a quorum loss — if two of three Sentinels are unreachable (a bad two-node drain), failover cannot elect a master and reconciliation stalls until quorum returns. That is a HA availability bug, not a DR data-loss event; the fix is the quorum-aware disruption budget, next.

PodDisruptionBudgets and anti-affinity: surviving maintenance

HA replicas are worthless if a routine node drain schedules them all onto one node and then evicts that node. Two mechanisms prevent it. Pod anti-affinity (baked into the HA manifests) spreads replicas across nodes and zones. PodDisruptionBudgets cap how many replicas a voluntary disruption (a drain, a cluster upgrade) may take down at once. Without PDBs, kubectl drain will happily evict every argocd-server replica simultaneously and take your control plane down during a maintenance window you scheduled yourself.

# Representative PDBs — values illustrate intent; the HA manifests ship equivalents.
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: argocd-server
  namespace: argocd
spec:
  minAvailable: 1                      # keep the API/UI reachable during a drain
  selector:
    matchLabels:
      app.kubernetes.io/name: argocd-server
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: argocd-redis-ha-server
  namespace: argocd
spec:
  minAvailable: 2                      # PRESERVE Redis Sentinel quorum (2 of 3)
  selector:
    matchLabels:
      app.kubernetes.io/name: argocd-redis-ha

The redis-ha-server PDB with minAvailable: 2 matters most and is the one people get wrong: set it to 1 (or omit it) and a two-node drain can take two Redis Pods, break quorum, and stall reconciliation cluster-wide — a self-inflicted incident during planned maintenance. It is the direct fix for the “HA Redis quorum lost” symptom below.

Multi-AZ and the cloud edges

HA within one zone survives a node failure; HA across zones survives a zone failure — the outage that actually shows up on a cloud status page. Spreading the control plane across zones is where the three clouds differ.

The rule is identical everywhere — at least three nodes across two, ideally three, zones so anti-affinity places one replica per zone — only the knob differs: an AZ-aware node pool on AKS/EKS, a regional cluster on GKE. Put the hub on a regional/multi-AZ cluster and a zone outage becomes a rescheduling event, not a rebuild.

Backup: argocd admin export, the sensitive Secrets, and Git

You now know the scope. Backing it up is, satisfyingly, one command: argocd admin export walks the argocd namespace and writes a single YAML stream of the config ConfigMaps (argocd-cm, argocd-rbac-cm, known-hosts, TLS certs), the Argo CD Secrets (argocd-secret plus your cluster and repo creds), and every Application, ApplicationSet, and AppProject. One file, the entire control-plane state.

# Export ALL Argo CD state to a file. Talks to the Kubernetes API via your kubeconfig,
# NOT to argocd-server — so it works even when the API/UI is down.
argocd admin export -n argocd -o argocd-backup.yaml
# equivalently, the default out ("-") streams to stdout:
argocd admin export -n argocd > argocd-backup.yaml

# See what you captured — the kinds present prove the scope:
grep '^kind:' argocd-backup.yaml | sort | uniq -c
#   14 kind: Application
#    3 kind: AppProject
#    2 kind: ApplicationSet
#    6 kind: ConfigMap
#    9 kind: Secret          <-- cluster creds, repo creds, argocd-secret (PLAINTEXT)

Two properties make this command the backbone of Argo CD DR, and both are easy to miss:

The real flags you will use (current for 2.13+/3.x):

The nastiest export gotcha, straight from the docs: argocd admin export will not fail if you run it against the wrong namespace. Point it at default instead of argocd and it cheerfully writes a nearly empty file and exits 0. Your backup “succeeds” every night and contains nothing. Always pass -n argocd explicitly, and — this is the important part — verify the output, e.g. assert a minimum kind: count, so an empty backup fails loudly instead of silently.

A backup CronJob

A backup you run by hand is a backup you forget. Schedule the export as an in-cluster CronJob that streams to encrypted object storage. Running inside the cluster, argocd admin export uses the ServiceAccount token as its kubeconfig automatically — you just need RBAC to read the objects.

# 1. A ServiceAccount + Role: read the ConfigMaps, Secrets, and Argo CD CRs.
apiVersion: v1
kind: ServiceAccount
metadata:
  name: argocd-backup
  namespace: argocd
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: argocd-backup
  namespace: argocd
rules:
  - apiGroups: [""]
    resources: ["configmaps", "secrets"]
    verbs: ["get", "list"]
  - apiGroups: ["argoproj.io"]
    resources: ["applications", "applicationsets", "appprojects"]
    verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: argocd-backup
  namespace: argocd
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: argocd-backup
subjects:
  - kind: ServiceAccount
    name: argocd-backup
    namespace: argocd

# 2. The CronJob: export → gzip → encrypted object store, every 6 hours.
apiVersion: batch/v1
kind: CronJob
metadata:
  name: argocd-backup
  namespace: argocd
spec:
  schedule: "0 */6 * * *"          # every 6h → 6h worst-case RPO for the small state
  concurrencyPolicy: Forbid
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      backoffLimit: 2
      template:
        spec:
          serviceAccountName: argocd-backup   # IRSA/WI-annotated for the object store
          restartPolicy: OnFailure
          containers:
            - name: backup
              image: quay.io/argoproj/argocd:v2.13.3   # MATCH your Argo CD version
              command: ["/bin/sh", "-c"]
              args:
                - |
                  set -euo pipefail
                  ts=$(date +%Y-%m-%dT%H%M%SZ)
                  argocd admin export -n argocd \
                    | gzip \
                    | aws s3 cp - "s3://acme-argocd-backups/${ts}.yaml.gz" \
                        --sse aws:kms --sse-kms-key-id "$KMS_KEY_ID"

The --sse aws:kms is not optional decoration. The export contains plaintext cluster and repo credentials, so it must be encrypted at rest (KMS/CMK) as well as in transit. The CronJob’s ServiceAccount uses workload identity — never a static key — to reach the store, exactly as you set up when bootstrapping. Per cloud:

Store the backup in a different region (or at least a different blast radius) from the Argo CD cluster — a backup that lives only in the hub’s region is one a regional outage takes with it. Cross-region replication is cheap for a few kilobytes of nightly YAML and removes the “the DR backup was in the datacenter that burned” failure mode.

Git is the primary backup; the export is the belt

Keep the hierarchy straight. Git is the primary backup of your app desired state and — if you are declarative — of the control-plane CRs. The argocd admin export is the belt-and-braces snapshot of the small sensitive state Git deliberately does not hold (cluster creds, argocd-secret). If you follow GitOps discipline fully, the only thing the export adds over Git is the credentials — which is why some teams skip full exports and instead back up just the sensitive Secrets (Sealed Secrets in Git, or the cloud secret store via ESO) and rebuild the rest from Git. Both are valid; pick by how much you value a one-command identical restore versus a fully-declarative rebuild.

Restore: argocd admin import versus rebuilding from Git

There are two ways to bring the state back, and the choice is a philosophy, not a mechanic: import a snapshot, or rebuild from Git and re-adopt. Knowing both — and when each is right — is the core skill of Argo CD DR.

Here is the whole model in one picture. Read it left to right: Git holds the desired state (the primary backup) and a small backed-up state holds the sensitive credentials the app repos deliberately omit; a disaster destroys the hub while the workloads keep serving off the data path; you rebuild Argo CD on a fresh cluster and restore that small state; then you apply the one root Application and Argo CD re-adopts every running app from Git — the argocd app diff coming back empty is the proof that recovery caused no redeploy.

The badges mark the decisions that make or break a recovery: Git is the real backup (1); the cluster Secrets are the sensitive bit that is not in your app repos (2); Redis is a disposable cache you never back up (3); the control plane is off the data path, which is why your apps keep serving through the disaster (4); you rebuild first and restore the small state before applying the root app (5); and you diff before you sync so recovery re-adopts instead of redeploying (6).

argocd admin import

Import is the mirror of export: it reads the YAML stream and creates/updates the objects in a target namespace. Like export, it talks to the Kubernetes API, not to argocd-server, so you can restore into a cluster whose Argo CD is freshly installed and not yet fully up.

# Restore everything from the encrypted backup into a fresh Argo CD.
# The "-" means "read the manifest stream from stdin".
gunzip -c argocd-backup.yaml.gz | argocd admin import -n argocd -

# What it does: upserts the ConfigMaps, Secrets, and CRs from the file. After this,
# the fresh Argo CD has your projects, apps, cluster creds, and RBAC — as of the backup.

Import restores an exact point-in-time snapshot: whatever apps, projects, and config existed when the export ran. That is its strength (a fast, identical rebuild) and its weakness (it can resurrect an app you deleted after the backup, and it is only as fresh as the last export — its RPO is your backup interval).

Rebuild-from-Git: the cleaner, GitOps-native path

The alternative is to not import the CRs at all. You install Argo CD, restore only the sensitive state Git does not hold (cluster Secrets, argocd-secret, repo creds), and apply your root app-of-apps — the same bootstrap seed Terraform plants when it hands a fresh cluster off. Argo CD reads the platform-gitops repo, recreates every child Application/ApplicationSet from Git, and reconciles. The control-plane CRs come back from Git — current, reviewed, identical to what you run day to day — not from a snapshot that may be hours stale.

# Rebuild-from-Git: restore ONLY the sensitive state, then let Git recreate the CRs.
# 1. Cluster + repo creds + argocd-secret (from Sealed Secrets, ESO, or a scoped import):
kubectl apply -n argocd -f sealed-cluster-secrets.yaml    # decrypts in-cluster
# 2. Seed the one root Application; it pulls the whole platform from Git:
kubectl apply -n argocd -f bootstrap/root-app.yaml
# 3. Argo CD recreates every child app FROM GIT and reconciles.

For most teams the answer is rebuild-from-Git for the CRs, plus a targeted restore of the sensitive Secrets — the best of both. Git gives current, reviewed app definitions with zero staleness; the Secret restore (or ESO) covers the one thing Git does not hold. Keep the full export as a forensic snapshot and break-glass fallback, not your primary path.

The re-adoption safety check: diff, then sync

Both paths converge on the single most important safety step in Argo CD DR — the one that separates a clean recovery from a self-inflicted fleet-wide outage. When the hub was lost but the spoke workloads kept running, your rebuilt Argo CD must re-adopt those live workloads — recognize that what Git says should exist already exists and is correct — and mark them Synced with nothing to change. It must not “restore” them by redeploying: redeploying every app across your fleet during an incident is its own incident.

The proof that re-adoption will be clean is argocd app diff returning empty before you let auto-sync run:

# BEFORE enabling auto-sync, diff the restored app against the live (surviving) cluster.
argocd app diff guestbook
# (no output) === an empty diff: Git matches live. Re-adoption is safe; nothing redeploys.

# If instead you see a full diff wanting to recreate everything, STOP — do not sync.
# Something is wrong: wrong targetRevision, wrong cluster Secret, or the live workloads
# are actually gone. Investigate before letting the controller act.

The operational discipline: install and restore with auto-sync disabled, run argocd app diff on a representative sample, confirm the diffs are empty (Git matches the surviving live state), then enable automated sync — turning “did my recovery just redeploy the fleet?” from a prayer into a checked precondition.

The DR runbook: the cluster hosting Argo CD is gone

Now assemble it into the procedure you run when the pager goes off. The scenario is the worst realistic one: the entire cluster hosting Argo CD is gone — a botched upgrade, a deleted cluster, a lost region — while the spoke clusters with your workloads stay up and serving. Your job: rebuild the control plane and re-adopt the fleet without touching the running apps.

The ordering is not cosmetic. Restore the cluster Secrets (step 3) before the root app (step 4), or the controller’s first reconcile hits every spoke Application with no credential and floods you with Failed/Unknown that masks whether recovery is working. And diff before sync (5 before 6) is the guardrail that keeps re-adoption from becoming redeployment. Run the steps in order and the fleet returns Synced/Healthy with not a single Pod restarted — the workloads never stopped; you just gave them their manager back.

# Steps 5-6 in practice: prove no redeploy, then hand control back to the controller.
argocd app list -o name | while read app; do
  echo "== $app =="; argocd app diff "$app" || true   # want: empty diffs
done
# All empty? Then re-enable automation fleet-wide:
argocd app list -o name | xargs -I{} argocd app set {} --sync-policy automated

RTO, RPO, and what DR does not cover

Set expectations honestly — “Argo CD DR” and “application DR” are different projects, and conflating them causes bad promises.

Two honest caveats. First, Argo CD’s RTO is dominated by cluster provisioning, not by anything Argo-specific — argocd admin import takes seconds; standing up a fresh AKS/EKS/GKE cluster and its identity takes most of the window. How much you pre-build decides the number:

Pre-baking a warm standby (or a fast, tested Terraform module) shrinks RTO — the same skill as bootstrapping. Second, and most important: DR for Argo CD is not DR for your applications. If a stateful app loses its database, Argo CD redeploys the app’s manifests but cannot restore its data — that is the app team’s backup/restore, separate from anything here. Argo CD guarantees your desired configuration comes back; it promises nothing about your application state. Keep the two runbooks separate.

Testing DR: the game day

A DR plan you have never executed is a hypothesis, not a plan. The only way to trust the runbook above is to run it on a schedule — a game day — against a throwaway environment.

Run the “full hub loss” drill at least quarterly and time it — that measured number is your real RTO, not the one in a wiki. The most valuable game day is the one where you deliberately forget the cluster Secrets, watch every spoke app go Unknown, and burn “restore Secrets before the root app” into muscle memory before it happens for real.

Hands-on lab

You will back up a running Argo CD, inspect what the backup contains, schedule it, then simulate the disaster — destroy the control plane while the workload keeps running — and rebuild it, proving with argocd app diff that recovery re-adopts rather than redeploys. It runs on a free local kind cluster. The commands and manifests are real and schema-correct for Argo CD 2.13+/3.x; the outputs are representative — nothing was run against a live cluster, so treat them as the shape to expect, not a transcript.

This lab is free — kind runs in Docker on your laptop, no cloud spend. Prerequisites: kind, kubectl, and the argocd CLI. New to installing Argo CD? Start with Installing Argo CD: Helm, Manifests, HA & First Login.

Step 1 — A cluster with Argo CD and one managed app. Give yourself real state to back up: an app Argo CD manages, running in its own namespace.

kind create cluster --name argo-dr           # free, local
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/v2.13.3/manifests/install.yaml
kubectl -n argocd rollout status deploy/argocd-server --timeout=180s

# A managed app with a distinct namespace so we can watch it survive the "disaster":
argocd login --core                          # talk to the API via kubeconfig (no server LB needed)
argocd app create guestbook \
  --repo https://github.com/argoproj/argocd-example-apps.git \
  --path guestbook --dest-server https://kubernetes.default.svc \
  --dest-namespace guestbook --sync-policy automated --auto-prune --self-heal
argocd app sync guestbook

What just happened: you have a real Argo CD managing guestbook, whose Pods run in the guestbook namespace. That namespace is your stand-in for a “surviving spoke” — it will keep running when we destroy the control plane.

Step 2 — Back up, and inspect the scope. Run the export and look at what it captured.

argocd admin export -n argocd -o argocd-backup.yaml
grep '^kind:' argocd-backup.yaml | sort | uniq -c
#    1 kind: Application        <-- guestbook is in the backup
#    6 kind: ConfigMap          <-- argocd-cm, argocd-rbac-cm, known-hosts, tls-certs, ...
#    5 kind: Secret             <-- argocd-secret + repo/cluster creds (PLAINTEXT)

What just happened: one command captured the control-plane state — the Application, config ConfigMaps, and Secrets. The Secret count is the tell: your backup holds plaintext credentials, which is why you encrypt it in production. The scope matches the “what to back up” table — apps, config, secrets; no Redis, because it is disposable.

Step 3 — Schedule it (and make an empty backup fail loudly). Apply the backup ServiceAccount/Role/CronJob from the Backup section. Add a guard so a wrong-namespace or empty export cannot pass silently.

# A verification you can bake into the CronJob's script — refuse to ship an empty backup:
n=$(grep -c '^kind:' argocd-backup.yaml)
if [ "$n" -lt 5 ]; then echo "BACKUP LOOKS EMPTY ($n objects) — failing"; exit 1; fi
echo "backup OK: $n objects"
# backup OK: 12 objects

What just happened: you turned the silent “export the wrong namespace and get an empty file” failure into a loud one. In the real CronJob this guard sits between argocd admin export and the upload, so a broken backup pages you instead of pretending to succeed.

Step 4 — Simulate the disaster. Destroy the control plane — delete the entire argocd namespace — but leave the workload running. This is the “hub is gone, spokes survive” scenario in miniature.

kubectl delete namespace argocd              # Argo CD is GONE
kubectl get pods -n guestbook                # ...but the workload is untouched:
# NAME                            READY   STATUS    RESTARTS   AGE
# guestbook-ui-6b5f...            1/1     Running   0          14m   <-- still serving

What just happened: you proved the thesis of the whole lesson on your own laptop. Argo CD is gone, yet guestbook keeps running with RESTARTS 0 — the control plane is not in the data path. You have lost reconciliation, not the app.

Step 5 — Rebuild the control plane. Reinstall Argo CD into the fresh (empty) namespace, restore the state from the backup, and — critically — do not sync yet.

# Reinstall Argo CD into the fresh namespace exactly as in Step 1, then restore:
argocd admin import -n argocd - < argocd-backup.yaml   # "-" reads the stream from stdin
kubectl get applications -n argocd
# NAME        SYNC STATUS   HEALTH STATUS
# guestbook   OutOfSync     Missing      <-- Argo CD sees the app spec but hasn't reconciled yet

What just happened: the rebuilt Argo CD has the guestbook Application back from the backup. It may briefly show OutOfSync/Unknown until it re-lists the live cluster — that is the controller catching up, not a problem. In a multi-cluster setup this is also where you would confirm the restored cluster Secrets are present so the hub can reach its spokes.

Step 6 — Prove re-adoption, not redeploy. This is the payoff. Diff the restored app against the still-running workload before allowing any sync.

argocd app diff guestbook
# (no output)   <-- EMPTY diff: Git matches the live, surviving workload.

argocd app get guestbook
# Name:  guestbook    Sync Status:  Synced      Health Status:  Healthy
kubectl get pods -n guestbook
# guestbook-ui-6b5f...  1/1  Running  0  20m     <-- RESTARTS still 0: never redeployed

What just happened: the empty diff proves Git matches what is already running, so Argo CD re-adopts the live Pods and marks the app Synced/Healthy without recreating anything. RESTARTS 0 and the unchanged age are the receipt: recovery restored management of the app, not the app itself — the entire promise of Argo CD DR, demonstrated end to end.

Step 7 — Teardown. Free the local resources.

kind delete cluster --name argo-dr           # removes the whole thing; zero cost anyway
rm -f argocd-backup.yaml

What just happened: kind clusters are local and ephemeral, so teardown is one command with nothing billable to leak — unlike a cloud DR drill, where you would also confirm no orphaned load balancers or disks remain.

Going deeper

The body gave you the load-bearing model and a runbook that works. This section is for the edges that bite once you run Argo CD DR for real: the config objects the obvious backup misses, a namespace-level backup tool, the assumption the whole plan quietly rests on, how a standby hub goes wrong, and what changes when the backup and the target run different Argo CD versions.

The backup scope has a few more objects than the “big three”

The scope table names the critical rows — cluster Secrets, repo creds, argocd-secret, argocd-cm, argocd-rbac-cm. Two or three more config objects hide behind “should be in Git” and are the ones teams discover missing after a rebuild, when the control plane is up but oddly lobotomized:

The robust move is the same one the body preaches for the CRs: keep all of this in Git as part of your self-managed install (Helm values, or a Kustomize overlay that Argo CD itself manages), so a rebuild-from-Git restores it for free. If instead you lean on argocd admin export, do not assume it captures every ConfigMap — the export recognizes a fixed set of Argo CD objects, and your notifications setup (its own topic in Notifications: Slack, Teams & Webhooks) may not ride along. Verify by grepping the export for the names above, exactly as you grep for a non-empty kind: count.

Velero: back up the namespace, not just Argo CD’s view of it

argocd admin export is Argo-aware — it knows which objects are Argo CD’s and writes a small, portable, greppable YAML. Velero comes at DR from the other direction: it backs up Kubernetes objects by namespace, so --include-namespaces argocd captures everything in the namespace — every ConfigMap and Secret (including the three above), the CRs, and any PVCs — without needing to know what Argo CD is.

# Velero: schedule a namespace backup to cloud object storage every 6 hours.
velero schedule create argocd-ns --schedule="0 */6 * * *" \
  --include-namespaces argocd --ttl 168h0m0s
# Restore into a fresh cluster (e.g. cluster migration after a region loss):
velero restore create --from-backup argocd-ns-20260908060000

They are complements, not rivals. A common production posture: Velero backs up the whole cluster (so the argocd namespace rides along with everything else) and a lightweight argocd admin export runs on its own schedule as a portable, reviewable, tool-independent snapshot you can grep and restore anywhere. If you already run Velero cluster-wide, you may not need a separate export — but keep one anyway for the greppability and the zero-dependency restore path.

The assumption the whole plan rests on: Git is up

Re-read the runbook and you will notice it quietly assumes Git is always reachable. “Rebuild from Git,” “the CRs come back from Git,” “targetRevision matches live” — all of it presumes the platform-gitops repo answers. Ask the uncomfortable question: what if the Git provider itself is gone — GitHub having a bad day, an org deleted, a self-hosted GitLab whose disk died?

First, the good news the data-path split still buys you: a Git outage is not an app outage. Your workloads keep serving, exactly as when Argo CD itself is down. What you lose is the ability to render new manifests — repo-server cannot clone a dead repo — so you cannot ship changes or self-heal drift until Git returns. A Git-provider outage degrades you to “frozen,” not “down.”

Second, the sharp correction to a natural misconception: argocd admin export does not save you from Git loss. The export preserves the intent — the Application CR with its repo URL, path, and targetRevision — but Argo CD still renders the actual manifests from Git at sync time. Restore the CR into a fresh hub while Git is gone and the app sits Unknown, unable to clone. The only real protection is to make Git itself redundant:

Treat Git the way this whole lesson treats Argo CD’s app repos: as the source of truth that must survive. The irony worth internalizing — the one component in this entire picture that behaves like a real, must-restore database is not Argo CD at all; it is your Git provider.

Standby-hub mechanics: one reconciler at a time, or you get split-brain

The standby-posture table offered “warm standby: a second cluster with Argo CD already installed, idle.” Idle is doing enormous work in that sentence, because Argo CD is agentless — the spokes have no controller of their own; they simply expose a kube-API that whichever hub holds their cluster Secret can drive. Stand up a second hub with the same cluster Secrets and the same Applications on automated sync, and now two controllers reconcile the same spokes at once. They fight: both write, both re-stamp the tracking annotation, syncs thrash, and an ownership tug-of-war plays out across the fleet. That is split-brain, and it is worse than having no standby.

The rule is absolute: exactly one hub reconciles a given spoke at any moment. A genuinely passive standby enforces that with a single lever — keep its controller from running:

# On the STANDBY hub: installed and credentialed, but NOT reconciling.
kubectl -n argocd scale statefulset argocd-application-controller --replicas=0
# Keep cluster Secrets + config pre-synced so failover is fast; with the controller
# at 0 replicas the standby watches nothing and touches no spoke.

# FAILOVER (primary confirmed dead): promote the standby, then diff before trusting it.
kubectl -n argocd scale statefulset argocd-application-controller --replicas=2
argocd app list -o name | while read a; do argocd app diff "$a" || true; done   # expect empty

The clean pattern reuses everything you built for bootstrapping (Cluster Bootstrapping & the Terraform Hand-off): the standby’s cluster Secrets and root app are staged by the same module, and failover is “scale the controller up” (or “apply the root app”) plus the same diff-before-sync check the body drilled into you. The confirm-the-primary-is-dead step is not optional — promoting a standby while the primary is merely partitioned is precisely how you create the split-brain you were trying to avoid.

Cross-version restore: when the backup and the target disagree

A DR event is the worst time to also do a major upgrade, yet it happens constantly — you back up on the version you were running and rebuild onto whatever the current manifests install, which may be a newer minor or a new major (Argo CD 2.x → 3.0 GA). Two things can bite:

• Resource tracking method. Argo CD marks the resources it owns either with a label (app.kubernetes.io/instance, the legacy default) or an annotation (argocd.argoproj.io/tracking-id); 3.0 makes annotation the default. Your live, surviving workloads are stamped with whatever the old hub used. If the rebuilt hub’s application.resourceTrackingMethod (in argocd-cm) disagrees, the controller does not recognize the running resources as belonging to the app — it treats them as strangers and wants to create duplicates. That is the “recovery redeployed the fleet” footgun wearing a new hat. Pin the tracking method to match what your live resources actually carry, before you apply the root app:

# argocd-cm — force the tracking method the LIVE resources already use, so a
# rebuilt (possibly newer) hub re-adopts instead of duplicating.
apiVersion: v1
kind: ConfigMap
metadata: { name: argocd-cm, namespace: argocd }
data:
  application.resourceTrackingMethod: annotation   # match your existing resources

• CRD schema drift. Importing older CRs into a newer CRD is normally safe (schema changes are additive); the reverse — restoring newer CRs onto an older CRD — can fail validation or silently drop fields. The safe rule: restore into the same-or-newer version you backed up from, then upgrade in place. Also skim the target version’s upgrade notes before restoring an old export — 3.0 dropped some deprecated config keys and flipped a few defaults, and an export written for 2.x may carry settings the new install ignores or rejects.

The DR-day discipline is the body’s guardrail with one extra move: restore, pin the tracking method, argocd app diff until empty, then enable sync — now proof against a version gap as well as a credential gap.

Common mistakes and troubleshooting

Every row here is a real failure mode with a real Argo CD symptom. The nastiest three get extra prose after the table.

1. “Restore redeployed everything” is the incident-during-an-incident. The instinct is to install Argo CD with auto-sync on and let it “fix” everything — but your workloads are already running on the surviving spokes; there is nothing to fix, only to re-adopt. If the restored state has a wrong targetRevision, or a cluster Secret points at a cluster whose live state you can’t read, an eager controller reads “I can’t confirm this matches” as “recreate it” — a fleet-wide rollout in the middle of a disaster. The discipline is non-negotiable: auto-sync off, argocd app diff until empty, then automation on. An empty diff is your written permission to let the controller act; being in a hurry is exactly when it saves you.

2. Losing the cluster Secrets is the failure that hides in plain sight. Teams back up the Application CRs (or keep them in Git) and feel safe, then find during a rebuild that the hub sees every app but can reach no cluster — the one thing that was neither in Git nor in the “obvious” backup was the credential to the spokes. The cluster Secrets matter most precisely because GitOps discipline does not protect them: they are sensitive, so they are (correctly) never in your app repos. Back them up explicitly — Sealed Secrets in Git, the cloud secret store via ESO, or inside the encrypted export — and rehearse restoring them first, before the root app.

3. Redis panic during recovery wastes your window. When you rebuild and Redis comes up empty, the UI is slow, apps flicker Progressing, and the reconcile queue spikes — and an operator who does not know Redis is disposable starts hunting for a backup that does not and should not exist. There is nothing to restore. The cache is supposed to be cold after a rebuild; it warms within minutes as repo-server recomputes manifests and the controller re-lists clusters. Spend your recovery window on what actually needs you — the cluster Secrets, the diff check — not a phantom Redis restore.

Common beginner mistakes

These are wrong mental models — the beliefs that send people down the wrong path before they ever hit a symptom. (For symptom → cause → fix, use the troubleshooting table above.)

• “I need to snapshot etcd and replicate Redis to protect Argo CD.” You are treating a reconciler like a database. Git is the source of truth, each spoke’s own etcd holds its live state, and Redis is a cache Argo CD rebuilds by itself. Back up the small credential-and-identity set; leave etcd and Redis alone.
• “If Argo CD is down, my apps are down.” The single most expensive misconception. Argo CD is a control plane, not a data-path component — no user request flows through it. When it dies you lose reconciliation, drift correction, and new deploys; every running Pod keeps serving. You have a recovery window, not an outage.
• “Backing up the Application CRs is my DR plan.” The CRs are the easy part — usually already in Git. The thing that is neither in Git nor obvious is the cluster Secret that lets the hub reach each spoke. Back that up first and rehearse restoring it before the root app, or you will rebuild a hub that sees every app and can reach none of them.
• “My manifests are in Git, so I’m fully covered.” Git holds your desired state, not your credentials — and rebuild-from-Git assumes Git is reachable. You still need the sensitive Secrets backed up (Sealed Secrets/ESO or an encrypted export), and you should make Git itself redundant with a mirror.
• “Restoring means letting Argo CD sync everything back into place.” Recovery is re-adoption, not redeployment. The workloads on the surviving spokes never stopped; Argo CD only needs to recognize them. Install with auto-sync off, argocd app diff until empty, then enable automation — an empty diff is your written permission to hand control back.
• “We have a nightly backup, so DR is handled.” A backup you have never restored is a hypothesis. The number that matters is a measured restore time from a game day, not the existence of a cron job — and the most valuable drill is the one where you deliberately forget the cluster Secrets and feel the failure safely, before it happens for real.

Cheat-sheet

The backup scope as a decision rule you can apply to any piece of Argo CD state:

The commands that do the work:

The HA + DR knobs:

The DR runbook, condensed to what you type when the hub is gone:

Practice challenges

Six exercises, beginner to expert. Each runs on the free local kind setup from the Hands-on lab (no cloud spend); representative outputs are shown. Try each before opening the solution.

1. (Beginner) Prove your backup is not empty. Export the control plane and print how many objects you captured — the one check that separates a backup from a hopeful cron line.

<details> <summary>Solution</summary>

argocd admin export -n argocd -o backup.yaml
grep -c '^kind:' backup.yaml        # e.g. 12 — any low number (<5) means something's wrong

Point it at the wrong namespace (-n default) and watch it exit 0 with a near-empty file — the exact silent failure the body warns about.

Why: an export you never counted is a backup you do not have; the count guard is the cheapest insurance in this lesson. </details>

2. (Beginner) Find the sensitive rows. From that same backup, list only the objects that are secret — the ones that are your entire DR responsibility and must be encrypted at rest.

<details> <summary>Solution</summary>

# The cluster-connection Secrets (the critical bit):
grep -B1 'argocd.argoproj.io/secret-type: cluster' backup.yaml
# Walk every Secret kind and its name (cluster creds, repo creds, argocd-secret):
awk '/^kind: Secret/{s=1} s&&/name:/{print; s=0}' backup.yaml

Everything else in the file is either already in Git or disposable; these Secrets are the part Git deliberately does not hold.

Why: naming the sensitive objects by hand is how “back up the credentials” stops being a slogan and becomes a checklist. </details>

3. (Intermediate) Feel the data-path split. Destroy the control plane and prove a managed workload keeps serving.

<details> <summary>Solution</summary>

kubectl delete namespace argocd          # the hub is gone
kubectl get pods -n guestbook            # workload untouched:
# guestbook-ui-...   1/1   Running   0   14m   <-- RESTARTS 0

Then reinstall Argo CD, argocd admin import -n argocd - < backup.yaml, and argocd app diff guestbook → empty.

Why: seeing RESTARTS 0 after deleting Argo CD turns “the control plane is not in the data path” from a claim you read into a fact you have witnessed. </details>

4. (Intermediate) Make an empty backup page you. Write the guard a scheduled export must pass before it uploads, and encrypt the upload.

<details> <summary>Solution</summary>

argocd admin export -n argocd -o backup.yaml
n=$(grep -c '^kind:' backup.yaml)
[ "$n" -lt 5 ] && { echo "BACKUP EMPTY ($n) — failing"; exit 1; }
gzip -c backup.yaml | aws s3 cp - "s3://REDACTED-argocd-backups/$(date +%FT%H%M%SZ).yaml.gz" \
  --sse aws:kms --sse-kms-key-id "$KMS_KEY_ID"

The --sse aws:kms is mandatory: the file holds plaintext cluster and repo credentials.

Why: a backup that “succeeds” nightly but ships an empty file is more dangerous than no backup, because it buys false confidence. </details>

5. (Advanced) Build a standby that cannot split-brain. Stand up a second hub that is credentialed and ready but reconciles nothing until you say so.

<details> <summary>Solution</summary>

# On the standby: install Argo CD, restore the sensitive Secrets, then PARK the controller:
kubectl -n argocd scale statefulset argocd-application-controller --replicas=0
kubectl apply -n argocd -f sealed-cluster-secrets.yaml   # decrypts in-cluster
# Failover — only after confirming the primary is truly dead:
kubectl -n argocd scale statefulset argocd-application-controller --replicas=2

Restore the sensitive state the GitOps-native way with Sealed Secrets / ESO / SOPS / Vault.

Why: two controllers reconciling the same spokes fight over ownership; parking the standby’s controller at 0 replicas guarantees exactly one reconciler at a time. </details>

6. (Expert) Survive a version gap. Your backup was taken on Argo CD 2.x; you are rebuilding onto a 3.0 install. Make re-adoption clean anyway.

<details> <summary>Solution</summary>

# Pin the tracking method to what the LIVE resources already carry, BEFORE the root app:
kubectl -n argocd patch cm argocd-cm --type merge \
  -p '{"data":{"application.resourceTrackingMethod":"annotation"}}'
kubectl -n argocd rollout restart statefulset argocd-application-controller
# Then the usual guardrail:
argocd app diff <app>            # must be empty before you enable automated sync

If diffs come back wanting to recreate everything, the tracking method is still mismatched — do not sync.

Why: 3.0 defaults tracking to annotation; if the rebuilt hub disagrees with how your live resources are stamped, it treats running workloads as strangers and redeploys the fleet. </details>

Interview and exam questions

Q: Why is disaster recovery for Argo CD fundamentally easier than for a database? A: Because Argo CD is not the source of truth and is not in the data path. Its desired state lives in Git (already backed up and replicated), the live state lives in the target clusters’ etcd, and its Redis is a disposable cache. The only unique state it holds is a small set of credentials and access-control config. So recovery is a rebuild (reinstall and re-point at Git), not a restore of authoritative data — and because the control plane isn’t in the request path, running workloads keep serving while you rebuild.

Q: If the entire cluster hosting Argo CD is destroyed, what happens to the applications it manages? A: They keep running. Argo CD is a control plane, not a data-plane component — no user traffic flows through it. When it’s gone you lose reconciliation, drift correction/self-heal, and the ability to ship new changes, but every already-running Deployment on every spoke keeps serving. You have a recovery window (typically minutes to an hour) before the lack of reconciliation actually bites.

Q: What state must you explicitly back up, given that Git already holds your manifests? A: The state that is not in your app Git repos, all of which is sensitive: the cluster Secrets (secret-type: cluster — credentials to reach the spokes), repo credentials (secret-type: repository), and argocd-secret (server signing key, OIDC client secret, TLS). If you keep your Application/ApplicationSet/AppProject CRs and argocd-cm/argocd-rbac-cm declaratively in Git, those are covered too; if not, back them up as well. Redis is never backed up — it’s disposable.

Q: What does argocd admin export capture, and why is the resulting file sensitive? A: It exports the whole Argo CD control-plane state from the namespace via the Kubernetes API: the config ConfigMaps, argocd-secret, the cluster/repo Secrets, and all Application/ApplicationSet/AppProject CRs. It’s sensitive because it includes those Secrets in plaintext — a complete backup is also a complete credential dump, so it must be encrypted in transit and at rest and never committed to Git raw.

Q: A colleague set up a nightly argocd admin export CronJob and it reports success every night, but the restore test found the backups empty. What likely happened? A: The export ran against the wrong namespace. argocd admin export does not error if it finds nothing in the target namespace — point it at default instead of argocd and it writes a near-empty file and exits 0. Fix: always pass -n argocd, and add a guard that asserts a minimum object count (grep -c '^kind:') so an empty backup fails the job loudly.

Q: Explain the difference between argocd admin import and rebuilding from Git, and when you’d choose each. A: import restores an exact point-in-time snapshot including secrets — fast and identical, but only as fresh as the last export and able to resurrect apps you deleted afterward. Rebuild-from-Git installs Argo CD, restores only the sensitive Secrets, and applies the root app-of-apps so the CRs come back from Git — current, reviewed, no staleness — but you must re-materialize the secrets separately. Most teams rebuild the CRs from Git and do a targeted restore of the sensitive Secrets, keeping a full export as a break-glass fallback.

Q: During recovery, how do you make sure Argo CD re-adopts running workloads instead of redeploying them? A: Install and restore with auto-sync disabled, then run argocd app diff on the restored apps against the surviving live clusters. An empty diff proves Git matches what’s already running, so Argo CD will mark the apps Synced and adopt the existing Pods with nothing to change. Only after diffs are empty do you re-enable automated sync. Skipping the diff risks a fleet-wide redeploy in the middle of an incident.

Q: What is the correct order of the DR runbook steps, and why does ordering matter? A: Provision the cluster → install Argo CD (HA, auto-sync off) → restore cluster Secrets/config → apply the root Application → diff → enable auto-sync. Restoring the cluster Secrets before the root app matters because otherwise the controller’s first reconcile hits every spoke app with no credential and floods you with failures. Diffing before enabling sync matters because it’s the guardrail that turns re-adoption into a checked precondition rather than a fleet-wide redeploy.

Q: What does losing Redis cost you, and what should you do about it? A: Nothing permanent. Redis is a disposable cache of computed manifests, resource trees, and app state that Argo CD regenerates from Git and the live clusters. Losing or wiping it causes a brief reconcile churn while the cache warms — not data loss — so you never back it up or restore it. The only Redis failure that hurts is a Sentinel quorum loss (2 of 3 down), which stalls reconciliation; prevent it with a minAvailable: 2 PDB and zone spread.

Q: How does the application-controller achieve HA, and what’s the single-replica risk? A: The controller runs as a StatefulSet that shards its managed clusters across replicas; you scale replicas and set ARGOCD_CONTROLLER_REPLICAS to match, choosing a --sharding-method (legacy/round-robin/consistent-hashing). A single controller replica is a single point of failure for reconciliation — if its node dies, drift correction and new syncs stop until it reschedules (your apps keep running regardless). Production runs at least two shards.

Q: What are realistic RTO and RPO for Argo CD, and what does its DR explicitly not cover? A: RTO is typically 15-60 minutes, dominated by provisioning the fresh cluster (the import/apply itself is seconds); shrink it with a warm standby or fast Terraform. RPO is roughly your backup cadence for the small sensitive state and near-zero for app definitions since Git is live. It does not cover application data — if a stateful app loses its database, Argo CD redeploys the app’s manifests but cannot restore its data; that’s the app team’s separate DR.

Glossary

• Control plane — the components that decide and enforce desired state (Argo CD’s application-controller, repo-server, API server). Distinct from the data path.
• Data path — the route a real user request travels (ingress → Service → Pod). Argo CD is not on it, which is why its failure is not an app outage.
• Source of truth — the authoritative copy of desired state. For Argo CD it is Git, not Argo CD itself.
• Reconciliation — the controller’s continuous loop of comparing live state to Git and closing the gap. What you lose (temporarily) when Argo CD is down.
• Drift — live state diverging from Git (a manual kubectl edit, a failed node). Self-heal corrects it; a dead controller lets it accumulate.
• Re-adoption — a rebuilt Argo CD recognizing already-running workloads as matching Git and marking them Synced with nothing to change — the goal of recovery, as opposed to redeploying.
• RTO (Recovery Time Objective) — how long recovery may take. For Argo CD, typically 15–60 min, dominated by provisioning the fresh cluster.
• RPO (Recovery Point Objective) — how much state you may lose, measured in time. ≈ your backup cadence for the small sensitive state; ~0 for app definitions because Git is live.
• Cluster Secret — the secret-type: cluster Secret holding the credentials and API endpoint for one managed spoke. The critical, sensitive, must-back-up object.
• argocd admin export / import — the commands that dump / restore Argo CD’s control-plane state via the Kubernetes API (not via argocd-server), so they work when the UI/API is down.
• Redis (in Argo CD) — a disposable cache of computed manifests, resource trees, and sync state. Never backed up; it warms from Git and the clusters.
• Sentinel / quorum — Redis HA’s failover electors; a majority (2 of 3) must be reachable to promote a new master. Losing quorum stalls reconciliation.
• PodDisruptionBudget (PDB) — a cap on how many replicas a voluntary disruption (a node drain) may remove at once; minAvailable: 2 on redis-ha protects Sentinel quorum.
• Pod anti-affinity — scheduling rules that spread replicas across nodes/zones so one node or zone loss cannot take a whole tier.
• Sharding (controller) — splitting managed clusters across application-controller replicas (a StatefulSet) for scale and HA; ARGOCD_CONTROLLER_REPLICAS must match the replica count.
• App-of-apps — a single root Application pointing at a Git repo of child Applications, so one kubectl apply re-creates the whole fleet — the DR bootstrap seed.
• Sealed Secrets / ESO — ways to keep sensitive material recoverable without plaintext in Git: Sealed Secrets stores an encrypted blob in Git that decrypts in-cluster; the External Secrets Operator materializes Secrets from a cloud secret store.
• Velero — a Kubernetes backup tool that snapshots resources (and volumes) by namespace, complementing Argo CD’s own export with a whole-namespace, cross-cluster restore.
• Split-brain — two Argo CD hubs reconciling the same spokes at once, fighting over ownership. Prevented by ensuring exactly one reconciler runs at a time.
• Resource tracking method — how Argo CD marks the resources it owns: label (legacy) or annotation (the 3.0 default). A mismatch on restore makes the controller fail to re-adopt live resources.
• Game day — a scheduled rehearsal of the DR runbook against a throwaway environment; the measured restore time is your real RTO.

Key takeaways

• Argo CD is mostly stateless and rebuildable from Git. Its desired state is your Git repos, live state is in the clusters, and Redis is a disposable cache. Recovery is a rebuild and re-point, not a database restore — which is why it’s forgiving.
• The control plane is not in the data path. If Argo CD (or its whole cluster) dies, running workloads keep serving; you lose reconciliation, drift correction, and new deploys until it’s back. That gives you a recovery window, not an outage.
• Back up only the small state that is not in your app repos: the cluster Secrets (credentials to spokes — the critical bit), repo creds, and argocd-secret. Everything else is in Git (if you’re declarative) or disposable (Redis).
• argocd admin export captures the whole control plane — CRs, config, and the sensitive Secrets in plaintext — via the Kubernetes API, so it works even when argocd-server is down. That also makes the export file itself a secret: encrypt it, and pass -n argocd (a wrong namespace silently produces an empty backup).
• HA so you rarely rebuild: sharded controller (ARGOCD_CONTROLLER_REPLICAS), ≥3 argocd-server, redis-ha (3 servers + Sentinel + HAProxy), PodDisruptionBudgets (redis minAvailable: 2 for quorum), anti-affinity, and multi-AZ/regional placement.
• Restore by rebuilding from Git for the CRs and doing a targeted restore of the sensitive Secrets — the GitOps-native path with zero staleness — keeping a full argocd admin import as a break-glass fallback.
• Re-adopt, don’t redeploy: diff before you sync. Install with auto-sync off, confirm argocd app diff is empty (Git matches the surviving live workloads), then enable automation. An empty diff is your permission to hand control back.
• Argo CD DR ≠ application DR. Its realistic RTO (15-60 min) is dominated by cluster provisioning and it guarantees your desired config comes back — never your application data, which is the app team’s separate responsibility. Test the whole runbook with quarterly game days; the measured time is your real RTO.