Sync Waves & Resource Hooks: Ordering, PreSync/Sync/PostSync/SyncFail & Jobs

In a nutshell

Kubernetes normally does not care what order you apply things in — you throw a pile of YAML at the API server, controllers wake up, and the cluster converges. Most of the time that is fine. This lesson is about the handful of times it is not fine, because one thing genuinely has to happen before another: a database migration before the pods that expect the new schema, a CRD before the custom resources that use it, a namespace before the quota inside it. Argo CD gives you exactly two tools for this, and they are the whole lesson. Sync waves are numbers you stick on resources to sort them into ordered groups, where Argo CD waits for each group to be Healthy before starting the next. Resource hooks are Jobs you attach to a moment in the sync — before it (PreSync), during it (Sync), after it is healthy (PostSync), or only if it fails (SyncFail).

A concrete analogy: think of a restaurant opening for the evening service. The kitchen preps its stations and the health inspector signs off before a single customer is served — that is your PreSync migration, and it has to pass or the doors stay shut. Then dishes go out in the right order and you do not fire the main course until the appetizer is actually on the table — that is sync waves, health-gated so wave 1 waits for wave 0. Once everything is served, the manager tastes a plate to confirm the food is good — that is the PostSync smoke test. And if the kitchen catches fire mid-service, the fire drill runs instead of the normal close — that is SyncFail. Same declared-in-Git deploy, but now it has a running order.

Level: Intermediate · Time: ~40 min

Prerequisites: you can create an Argo CD Application and you understand what “Healthy” means for a resource (the Sync Status & Health Assessment lesson). After this you can: number resources into ordered waves; write PreSync/PostSync/SyncFail hook Jobs; pick the right hook-delete policy; make a migration safely idempotent; and diagnose a sync that is stuck waiting on a wave.

---

Here is a problem that does not exist until, one day, it very much does. You point an Argo CD Application at a directory, hit sync, and Argo CD applies everything in it at once: the namespace, the ConfigMap, the Deployment, the Service, the Job that seeds your database. Ninety percent of the time this is exactly what you want — Kubernetes is declarative, controllers converge, order sorts itself out. Then you add one resource that has a prerequisite, and the whole thing falls over. The custom resource lands before its CRD is registered and the apply is rejected. The app pods start before the schema migration has run and they crash on a missing column. The ResourceQuota applies before the namespace exists and errors out. Nothing is wrong with your manifests — they are just being applied in the wrong order.

Argo CD gives you two mechanisms to fix this, and they are the entire subject of this lesson. Sync waves let you number your resources into ordered groups that apply one after another, with Argo CD waiting for each group to become Healthy before starting the next. Resource hooks let you run a resource — almost always a Job — at a defined point relative to the sync: before it (PreSync), during it (Sync), after everything is healthy (PostSync), or only when the sync fails (SyncFail). Used together they turn “apply a pile of YAML” into “run an ordered, gated deployment pipeline,” all declared in Git.

This is a cloud-neutral topic — waves and hooks behave identically on AKS, EKS, GKE, or a laptop kind cluster, because they are pure Argo CD sync mechanics with nothing cloud-specific in them. (The one real edge — a migration Job that needs credentials to reach a managed database — gets a short section of its own.) So the lab runs on a free local cluster and everything you learn transfers everywhere. We build the mental model, go phase by phase and policy by policy with real annotated manifests, then assemble the whole ordered deploy in the lab and watch the phases run in sequence.

---

Why this matters

Argo CD is a reconciliation engine: it renders your desired state from Git and drives the cluster to match it. For a single self-contained app, “apply it all and let Kubernetes converge” is the right model and you should not reach for ordering at all — adding waves you do not need is its own kind of bug. Ordering matters only when one resource cannot be healthy until another one already is, and that dependency is invisible to a plain kubectl apply -f ..

Three dependencies show up again and again, and every platform engineer meets all three eventually:

Notice what these have in common: the failure is not a typo in your YAML, and re-running the sync sometimes fixes it (the CRD registered on the first pass, so the CR applies on the second) and sometimes does not (the migration will never run after the pods, because Argo CD applied them together). “Just sync again” is not an ordering strategy — it is a coin flip that happens to land right when Kubernetes’ own retries paper over the gap.

The mental model to hold for the rest of this lesson: a sync is not one instant — it is a sequence you can shape. By default that sequence is Argo CD’s built-in ordering (namespaces and CRDs first, then everything else, roughly at once). Sync waves let you insert your own ordered checkpoints into it. Hooks let you attach run-once work — migrations, tests, cleanup — to the boundaries of it. Get these two tools right and a large chunk of “flaky deploy” tickets simply stop happening.

One sentence to anchor everything below: waves order the resources within a sync; hooks attach extra work to the phases of a sync. They compose — a hook can also carry a wave — but they answer different questions. Keep them as two words in your head.

---

Two mechanisms: sync waves and resource hooks

Before the details, get the shapes of the two tools clear, because beginners routinely reach for the wrong one. Both are driven by annotations on your Kubernetes resources — no new CRD, no controller flag, just metadata Argo CD reads at sync time.

The single most useful distinction: a normal resource with a sync-wave is still your permanent desired state — it lives in the cluster, self-heals, gets pruned when you delete it from Git. A hook is transient work — a migration that runs and finishes, a test that passes and is deleted. If the thing should keep running, it is a wave’d resource, not a hook. If the thing should run once at a boundary and go away, it is a hook.

When do you use which? Most real deploys use both, but the trigger for each is different:

Here is the whole model as one picture. Read it left to right: the application-controller starts a sync operation and consults phases (PreSync → Sync → PostSync) and, within each phase, waves. The PreSync migration Job runs and must succeed; then the Sync phase applies wave 0 (ConfigMap + Deployment) and waits for it to be Healthy before wave 1 (the dependent app); once everything is Healthy the PostSync smoke-test Job runs; and if the operation fails anywhere, the SyncFail hook fires instead.

The badges mark the ideas worth memorizing: waves order resources within a phase, lowest first (1); PreSync runs before the Sync phase and, because it runs on every sync, must be idempotent (2); waves are health-gated — wave 0 must be Healthy before wave 1 starts, which is how a database precedes its app (3); a resource that never becomes Healthy stalls the sync at that wave forever (4); PostSync runs only after the whole Sync phase is Healthy, so it is where smoke tests belong (5); and SyncFail fires only when the operation actually fails, with delete policies keeping hook Jobs from piling up (6). If you understand only this diagram, you already understand ordered deploys better than most people running Argo CD.

The rest of the lesson makes each of those six ideas concrete, then wires the whole pipeline together in the lab. The health-gating in badge 3 leans directly on the Sync Status & Health Assessment model — a wave “becomes Healthy” using exactly the per-kind health checks that lesson taught; if that idea is fuzzy, skim it first, because health is the gate that makes waves work.

---

Sync waves: ordering resources inside one sync

A sync wave is a number you attach to a resource with the annotation:

metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "-1"   # note: a STRING, quoted

The rules are few and precise:

That fifth rule — the health gate — is what makes waves more than cosmetic sorting. Argo CD does not merely apply wave −1 before wave 0; it applies wave −1, then blocks, watching health, and only proceeds to wave 0 once every resource in wave −1 reports Healthy. This is precisely how you express “the database must be up, not just created, before the app rolls.” A StatefulSet in wave 0 and a Deployment in wave 1 means the app pods do not even get applied until the database reports its replicas ready.

The full ordering: phase, then wave, then kind, then name

Waves are one term in a four-part sort key that Argo CD uses for every sync. Knowing all four explains behavior that otherwise looks random:

Two consequences fall straight out of this table. First, the phase always dominates the wave. A PreSync hook runs before every Sync-phase resource, no matter what wave numbers are involved — a PreSync hook in wave 5 still runs before a Sync resource in wave −5, because PreSync is a whole phase earlier. The wave on a hook only orders it against other hooks in the same phase. Second, even with no waves at all, Argo CD is not applying purely at random — the built-in kind ordering already puts Namespaces and CRDs early, which is why some ordering problems seem to fix themselves.

The implicit ordering, and why it is not enough

Argo CD’s built-in kind order handles the easy cases for free:

So a namespace and the ConfigMap inside it usually apply in the right order without a single wave, because the kind order already favors them. The trouble is that the kind order is fixed and generic — it knows nothing about your dependencies. It cannot know that your Deployment needs a migration first (both are “later” kinds), or that this operator must be Healthy before that custom resource. And the classic CRD-before-CR case is only partly solved: the kind order applies the CRD before the CR, but on a first-ever sync the CR’s dry-run can still fail because the API server has not finished establishing the freshly-applied CRD. That is where waves (CRD in wave −1, CR in wave 0) or the SkipDryRunOnMissingResource=true sync option make the ordering explicit and reliable rather than hopeful.

A pragmatic wave layout that scales — treat it as a starting convention, not law:

Waves do not cross Applications. This is the single most common wave misconception, and it is worth stating twice. sync-wave orders resources inside one Application’s sync. If your operator lives in Application A and your custom resources live in Application B, wave numbers on B’s resources will never wait for A — the two Applications sync independently. To order across Applications you put the waves on the child Application objects in an app-of-apps, or gate on health between stacked ApplicationSets. Do not expect two Applications to honor each other’s wave numbers; they do not.

---

Resource hooks: the four phases (plus Skip)

A hook is any resource — overwhelmingly a Job — annotated with a phase, telling Argo CD to run it at that point in the sync operation instead of treating it as permanent desired state:

metadata:
  annotations:
    argocd.argoproj.io/hook: PreSync

There are six phase values. The four in the lesson title are the ones you will use daily; the other two round out the picture:

The mental model for the run phases is a timeline: PreSync happens, and must succeed, before anything in your app changes. Sync applies your app. PostSync happens only after your app is fully Healthy. SyncFail is the catch that fires if any of that errors. A hook is not magic — a PreSync Job is a perfectly ordinary Kubernetes Job; the annotation just tells Argo CD to run it, wait for it to complete, and only then proceed to the Sync phase.

Three properties of hooks trip people up, so name them now:

• Hooks run on every sync, not just the first. A PreSync migration runs again on the next sync, and the next. This is why idempotency is non-negotiable (next section). It is not “run once ever” — it is “run once per sync.”
• A hook is identified purely by the annotation. Remove the annotation and the same Job becomes a permanent Sync-phase resource that Argo CD will try to keep alive and prune. Add it and the Job becomes transient hook work. The manifest body is identical.
• You can put a resource in multiple phases with a comma-separated list: argocd.argoproj.io/hook: PreSync,PostSync runs the same manifest in both phases. Useful for a check you want both before and after.

A hook can technically be any kind, but in practice:

Because a hook Job reports completion through its own Complete/Failed conditions, Argo CD knows whether the phase succeeded by reading the Job’s status — the same health assessment machinery that governs everything else. A PreSync Job that never completes leaves the sync stuck in the PreSync phase; a PostSync Job that fails flips the whole sync to failed. Both are covered in troubleshooting.

Here is a complete, real PreSync hook — a schema migration Job:

apiVersion: batch/v1
kind: Job
metadata:
  name: db-migrate
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
    argocd.argoproj.io/sync-wave: "-1"
spec:
  backoffLimit: 2                 # retry a couple of times before failing the hook
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: migrate
          image: registry.example.com/myapp-migrations:1.8.0
          command: ["/bin/sh", "-c"]
          args:
            - |
              # A real migration tool tracks which migrations have already run,
              # so re-running this on every sync is safe (idempotent).
              migrate -path /migrations -database "$DATABASE_URL" up
          envFrom:
            - secretRef:
                name: db-credentials   # provided out-of-band; never in this manifest

And a SyncFail hook that cleans up when a sync goes wrong:

apiVersion: batch/v1
kind: Job
metadata:
  name: rollback-canary
  annotations:
    argocd.argoproj.io/hook: SyncFail
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
spec:
  backoffLimit: 1
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: cleanup
          image: bitnami/kubectl:1.31
          command: ["/bin/sh", "-c"]
          args:
            - |
              echo "Sync failed — removing the half-applied canary."
              kubectl delete deployment myapp-canary --ignore-not-found

Note that neither manifest contains a plaintext secret — the migration credentials come from a Secret (db-credentials) created out-of-band by Sealed Secrets, External Secrets, or your cloud’s secret store, never committed in the Job. That rule does not relax just because a resource is a hook.

---

Hook delete policies: taming the Job pileup

A hook Job runs, finishes, and then… sits there. Run ten syncs and, unless you say otherwise, you can accumulate ten completed db-migrate Jobs (and their pods) cluttering the namespace. The hook delete policy controls when Argo CD cleans a hook resource up:

metadata:
  annotations:
    argocd.argoproj.io/hook-delete-policy: HookSucceeded

There are three policies, and choosing the right one per hook is the difference between a tidy namespace and a graveyard of finished Jobs:

The subtlety that catches everyone: the default policy is BeforeHookCreation. If you specify nothing, Argo CD deletes the previous hook instance right before creating the new one on the next sync. That keeps at most one instance around — but it means a completed hook Job stays visible until the next sync runs, and it never cleans up promptly on success. People expect finished hooks to vanish and are surprised to find yesterday’s db-migrate still sitting Completed in the namespace.

The real pileup happens in one specific case, and it is worth understanding exactly:

The takeaway is a rule of thumb: for a hook you do not need to inspect on success, set HookSucceeded. For a hook whose failures you want to debug (almost all of them), do not set HookFailed — let the failed Job stick around so you can kubectl logs it. You can combine policies with a comma (HookSucceeded,BeforeHookCreation), but for most migrations and tests a plain HookSucceeded is exactly right: it cleans up the happy path and preserves the evidence on failure.

---

Waves, hooks, and the classic patterns

Waves and hooks compose. A hook is placed in a phase by its hook annotation, and ordered within that phase by its sync-wave annotation. So a PreSync Job with sync-wave: "-1" runs in the PreSync phase (before any Sync-phase resource) and, if there were other PreSync hooks, would run before those with higher waves. In practice you use the wave on hooks mostly to order multiple hooks in the same phase — a backup PreSync (wave −2) before a migration PreSync (wave −1), say.

Three patterns account for the vast majority of real hook usage. Learn them as recipes.

Pattern 1 — the schema migration (PreSync Job, and the idempotency rule)

The archetype. Your new app version needs a database change; the change must land before the new pods, or they crash. A PreSync Job runs the migration; the Sync phase then rolls the app.

The rule that makes or breaks this pattern: the migration must be idempotent, because the PreSync hook runs on every single sync. Auto-sync on a new commit, a manual sync, or a self-heal — each is a sync, and each re-runs your PreSync migration. A naive CREATE TABLE orders (...) succeeds the first time and then fails on the second sync with “table already exists,” which fails the PreSync hook, which aborts the whole sync. Real migrations are idempotent by construction:

Use a real migration tool if you possibly can — hand-rolled IF NOT EXISTS scripts drift out of sync with reality fast. The point to internalize is that Argo CD gives you no “run once ever” hook; every hook is “run once per sync,” so idempotency is the price of admission for PreSync work.

Pattern 2 — the post-deploy smoke test (PostSync Job)

After the deploy is fully Healthy, prove it actually serves traffic before you call the sync a success. A PostSync Job curls the service and exits non-zero if the response is wrong:

apiVersion: batch/v1
kind: Job
metadata:
  name: smoke-test
  annotations:
    argocd.argoproj.io/hook: PostSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
spec:
  backoffLimit: 3            # tolerate a little startup jitter before failing
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: smoke
          image: curlimages/curl:8.10.1
          command: ["/bin/sh", "-c"]
          args:
            - |
              # Same namespace, so the short service name resolves.
              curl -sf --retry 5 --retry-delay 3 http://web/ | grep -q "ordered deploy OK"

The power of this pattern is that a failing PostSync fails the whole Application — the sync goes to a failed state, which you can alert on, and which triggers any SyncFail hook. The danger is the flip side: a flaky smoke test fails perfectly good deploys. If your test hits a still-warming cache or a dependency with its own startup lag, use --retry, a generous backoffLimit, and assert on something stable. A smoke test that fails 5% of the time will erode trust in the whole pipeline faster than no test at all.

Pattern 3 — the one-time data seed (guarded hook)

Seeding reference data (country codes, default roles) is “run once ever,” which Argo CD does not offer — so you build it out of a “run once per sync” hook plus a guard. A PreSync or Sync-phase Job that first checks whether the data exists and exits early if it does:

args:
  - |
    if [ "$(count-rows countries)" -gt 0 ]; then
      echo "Reference data already present — nothing to seed."; exit 0
    fi
    seed-reference-data

Same idempotency discipline as the migration: the seed must be safe to re-run because it will be re-run.

Is any of this cloud-specific?

Sync waves and hooks are pure Argo CD sync mechanics — identical on AKS, EKS, GKE, and a local kind cluster. There is no per-cloud behavior in the ordering itself. The one genuine edge appears only when a hook needs to reach a managed dependency: a PreSync migration Job connecting to a managed database needs an identity and credentials, and that is cloud-specific — not the hook, but how the Job’s ServiceAccount authenticates.

The hook manifest is the same on all three; only the ServiceAccount annotation and the secret source differ, and those are covered in the identity and secrets lessons. Everywhere else in this topic, “cloud-neutral” is the honest and complete answer.

---

From Helm hooks to Argo CD hooks

If you deploy Helm charts through Argo CD, you inherit a translation you should understand, because charts carry their own lifecycle hooks (helm.sh/hook) and Argo CD never runs helm install. As the Helm integration lesson explains, the repo-server runs helm template to render plain YAML — so Argo CD re-expresses Helm’s install/upgrade hooks as its own sync phases at render time:

The ordering and cleanup annotations map across too, which is what makes the translation usable rather than merely approximate:

The practical rule: a chart whose hooks do install/upgrade work (migrations, seeds) generally just works on Argo CD, because those map cleanly to PreSync/PostSync and hook-weight behaves like a wave. A chart that leans on helm test, delete/rollback hooks, or .Release.IsUpgrade branching needs attention — you either re-express the behavior as native Argo CD hooks yourself or accept that those paths never fire. If a chart’s hook Job mysteriously never runs, “it is a delete/rollback/test hook” is the first thing to check.

---

Hands-on lab

Time to build the whole ordered pipeline and watch the phases run in sequence. This lab runs on a free local cluster (kind or minikube) with Argo CD installed — nothing here bills, and everything is cloud-neutral. It assumes you can create an Argo CD Application from a Git repo, exactly as in Your First Application. We deploy: a PreSync migration Job (wave −1, a fake migration that sleeps then succeeds, cleaned up with HookSucceeded), a wave 0 ConfigMap + Deployment + Service, and a PostSync smoke-test Job that curls the service. Then we sync and watch PreSync → Sync → PostSync run in order.

The outputs below are representative shapes, not a transcript from your exact cluster — revisions, pod suffixes, and timings will differ. The ordering and the states are what matter. There is no live cluster behind this text; the manifests are schema-correct and the output is what Argo CD produces for this shape of app.

Step 0 — Lay out the repo. Put four files in a directory (ordered-demo/) in a Git repo Argo CD can read:

ordered-demo/
  01-presync-migrate.yaml    # PreSync hook, wave -1
  02-config.yaml             # ConfigMap, wave 0
  03-deploy.yaml             # Deployment + Service, wave 0
  04-postsync-smoke.yaml     # PostSync hook

01-presync-migrate.yaml — the migration hook. It sleeps 5 seconds to simulate a real migration, then exits 0:

apiVersion: batch/v1
kind: Job
metadata:
  name: db-migrate
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
    argocd.argoproj.io/sync-wave: "-1"
spec:
  backoffLimit: 2
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: migrate
          image: busybox:1.36
          command: ["/bin/sh", "-c"]
          args:
            - |
              echo "PreSync: running idempotent schema migration..."
              sleep 5
              echo "PreSync: migration complete."

02-config.yaml — the ConfigMap the app serves, wave 0:

apiVersion: v1
kind: ConfigMap
metadata:
  name: web-config
  annotations:
    argocd.argoproj.io/sync-wave: "0"
data:
  index.html: |
    <h1>KloudVin demo — ordered deploy OK</h1>

03-deploy.yaml — the Deployment (mounting that ConfigMap) and its Service, wave 0:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  annotations:
    argocd.argoproj.io/sync-wave: "0"
spec:
  replicas: 2
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: web
          image: nginx:1.27-alpine
          ports:
            - containerPort: 80
          volumeMounts:
            - name: html
              mountPath: /usr/share/nginx/html
      volumes:
        - name: html
          configMap:
            name: web-config
---
apiVersion: v1
kind: Service
metadata:
  name: web
  annotations:
    argocd.argoproj.io/sync-wave: "0"
spec:
  selector:
    app: web
  ports:
    - port: 80
      targetPort: 80

04-postsync-smoke.yaml — the smoke test, which curls the service after everything is Healthy:

apiVersion: batch/v1
kind: Job
metadata:
  name: smoke-test
  annotations:
    argocd.argoproj.io/hook: PostSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
spec:
  backoffLimit: 3
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: smoke
          image: curlimages/curl:8.10.1
          command: ["/bin/sh", "-c"]
          args:
            - |
              echo "PostSync: smoke-testing the web service..."
              curl -sf --retry 5 --retry-delay 3 http://web/ | grep -q "ordered deploy OK"
              echo "PostSync: smoke test passed."

What just happened: four manifests, three ordering annotations. The migration is a PreSync hook so it runs before the app; the ConfigMap and Deployment share wave 0 so they apply together in the Sync phase; the smoke test is PostSync so it runs only after the Deployment is Healthy. Commit and push all four.

Step 1 — Create the Application and sync it.

# Point an app at the directory and sync (adjust repo/path to your own)
argocd app create ordered-demo \
  --repo https://github.com/acme/argocd-demo.git \
  --path ordered-demo \
  --dest-server https://kubernetes.default.svc \
  --dest-namespace demo \
  --sync-option CreateNamespace=true \
  --sync-policy manual
argocd app sync ordered-demo

# (representative — trimmed)
Operation:   Sync
Phase:       Succeeded
Message:     successfully synced (all tasks run)

GROUP  KIND        NAMESPACE  NAME        STATUS     HEALTH   HOOK      MESSAGE
       Job         demo       db-migrate  Succeeded           PreSync   job.batch/db-migrate created
       ConfigMap   demo       web-config  Synced
apps   Deployment  demo       web         Synced     Healthy
       Service     demo       web         Synced     Healthy  service/web created
       Job         demo       smoke-test  Succeeded           PostSync  job.batch/smoke-test created

What just happened: read the HOOK column. db-migrate ran in the PreSync phase and shows Succeeded; the ConfigMap/Deployment/Service applied in the Sync phase (no hook); smoke-test ran in PostSync and Succeeded. The operation-level Phase: Succeeded means the whole sequence completed. Argo CD did not apply these all at once — it ran PreSync, waited, ran Sync, waited for Healthy, then ran PostSync.

Step 2 — Watch the ordering happen (the real payoff). If you sync again with the pods being watched, you see the sequence in wall-clock time:

# In one terminal, watch pods; in another, re-sync
kubectl -n demo get pods -w

# (representative — note the ORDER things appear)
NAME                    READY   STATUS      RESTARTS   AGE
db-migrate-abc12        0/1     Pending     0          0s     # PreSync: migration first
db-migrate-abc12        1/1     Running     0          2s
db-migrate-abc12        0/1     Completed   0          7s     # migration done → Sync phase begins
web-7d9c8b6f5-2xk4t     0/1     Pending     0          7s     # wave 0: app pods now
web-7d9c8b6f5-2xk4t     1/1     Running     0          12s
web-7d9c8b6f5-9mzq8     1/1     Running     0          12s    # Deployment Healthy → PostSync
smoke-test-def34        0/1     Pending     0          18s    # PostSync: smoke test last
smoke-test-def34        0/1     Completed   0          23s

What just happened: the timeline is the lesson. The db-migrate pod runs and completes before any web pod appears (PreSync gates the Sync phase). The web pods come up next and reach Ready. Only after the Deployment is Healthy does smoke-test appear (PostSync waits for health). You are watching phases and the wave-0 health gate execute in order.

Step 3 — Prove hooks re-run every sync (the idempotency lesson, live). Sync a third time and watch db-migrate run again:

argocd app sync ordered-demo
kubectl -n demo get jobs

# (representative) — db-migrate ran again this sync
NAME         STATUS     COMPLETIONS   DURATION   AGE
db-migrate   Complete   1/1           6s         8s
smoke-test   Complete   1/1           5s         2s

What just happened: the PreSync migration executed on this sync too — hooks are “run once per sync,” not “run once ever.” If our fake migration had been a non-idempotent CREATE TABLE, this second run would have failed and aborted the sync. This is exactly why real migrations must be idempotent.

Step 4 — Confirm the delete policy cleaned up. Because both hooks carry hook-delete-policy: HookSucceeded, their Jobs are deleted shortly after they succeed:

kubectl -n demo get jobs
# (a moment later — the succeeded hook Jobs are gone)
# No resources found in demo namespace.

What just happened: HookSucceeded removed each hook Job once it passed, so the namespace is not accumulating finished migration/test Jobs. Had we left the default (BeforeHookCreation), each Job would linger until the next sync replaced it. Had we used generateName with the default, they would pile up.

Teardown.

# Delete the app and everything it created (including the namespace)
argocd app delete ordered-demo --cascade

What just happened: --cascade removes the managed resources (ConfigMap, Deployment, Service) along with the Application. The hook Jobs were already gone via HookSucceeded. Your cluster is back to a clean Argo CD. You have now built and observed a real ordered deploy end to end.

---

Going deeper

The core mechanics above are enough to ship an ordered deploy. This section is for the reader who wants to know why the machinery behaves the way it does at the edges — the parts that turn a “works on my cluster” pipeline into one you trust in production.

What “Healthy” actually gates on — and the resources that gate on nothing

The health gate between waves is only ever as strong as the health check for the kinds sitting in that wave. The lesson already noted that a ConfigMap is Healthy the instant it applies, so a wave full of ConfigMaps gates on nothing and advances immediately. The sharper, less obvious version of that rule: a custom resource whose kind has no health check registered is also treated as Healthy the moment it applies. Argo CD ships built-in health assessments for the common kinds and a Lua library for many popular CRDs, but if you drop your own Foo/bar CR into wave 0 hoping to gate wave 1 on it, and Foo has no built-in check and no entry under resource.customizations.health.* in argocd-cm, then wave 0 reports Healthy instantly and your gate does nothing. If you need a custom resource to actually hold a wave, you must give its kind a health check — a Lua script that reads the CR’s status and returns Healthy/Progressing/Degraded.

The other bound worth knowing: a Deployment’s contribution to the health gate is capped by its own progressDeadlineSeconds (Kubernetes default 600s). A wave-0 Deployment that never becomes Ready does not truly wait forever — after the progress deadline it flips to Degraded, and that fails the wave and stops the sync. So “the wave that waits forever” is really “the wave that waits up to each resource’s progress deadline, then fails.” Lowering progressDeadlineSeconds is how you make a stuck rollout surface as a failure quickly instead of hanging.

Two different clocks: the sync operation vs the reconciliation loop

Newcomers routinely conflate two timeouts that have nothing to do with each other:

The consequence: a hook Job with no activeDeadlineSeconds can wedge a phase indefinitely, and neither --timeout (that only ends the CLI’s wait) nor timeout.reconciliation (that only sets the poll interval) will rescue you. Give every hook a real deadline:

spec:
  activeDeadlineSeconds: 600   # hard stop — the Job fails instead of hanging the phase
  backoffLimit: 2              # retry a couple of times, then report Failed

For automated sync, the Application carries its own retry block so a transient failure re-tries with backoff instead of parking Degraded:

spec:
  syncPolicy:
    retry:
      limit: 3
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 3m

Hooks in the same phase run in parallel — waves serialize them

Argo CD applies every hook that shares a phase and a wave together, then waits for all of them before moving on. So two PreSync hooks with no sync-wave do not run one-after-another — they run concurrently. If you need a backup to finish before a migration starts, and both are PreSync, giving them the same phase is not enough: you must put them in different waves (backup -2, migrate -1). This is the real reason the patterns table earlier assigned waves to same-phase hooks — the wave is the only lever that serializes hooks within one phase.

There is no automatic rollback — plan for partial application

A crucial expectation to correct: Argo CD does not roll back a failed sync. The behaviour splits on where the failure lands. A failed PreSync hook aborts the operation before the Sync phase, so your running app is untouched — the best-case failure. But if a later wave or a PostSync hook fails, the resources that already applied stay applied: the Application is left OutOfSync/partially-updated, and your only automatic responses are a SyncFail hook (cleanup you wrote yourself) and, if enabled, self-heal re-syncing toward Git. Rolling back to the previous good revision is a new sync (argocd app rollback, or reverting the commit), not something the failed operation does for you. When you genuinely need automatic, health-driven rollback of a workload, that is the job of progressive delivery — Argo Rollouts canary/blue-green — not sync waves. Waves order the apply; they do not supervise it afterwards.

Ordering across Applications, PostDelete, and finalizers

Waves never cross Applications — stated earlier, but the mechanism for ordering across them is worth naming. Because a child Application is itself just a resource in the parent, you can put sync-wave annotations on the child Application objects in an app-of-apps to bring up an operator app before the app that depends on it; ApplicationSet offers a progressive alternative (spec.strategy.type: RollingSync, enabled on the controller) that rolls a fleet in ordered steps. Finally, the PostDelete phase runs when the Application is deleted, and it is entangled with the app-level finalizer resources-finalizer.argocd.argoproj.io: that finalizer is what makes a delete cascade to the managed resources, and PostDelete hooks run as part of that teardown. Without the finalizer a delete is non-cascading (the Application object goes, the resources stay) and your PostDelete cleanup will not behave the way you expect. If PostDelete is your hook for tearing down external state (a cloud bucket, a DNS record), verify the finalizer is present or the hook may never fire. Hook-level troubleshooting for all of this — stuck phases, pruning order, orphaned hook Jobs — lives in Troubleshooting sync failures, hooks & pruning.

---

Common mistakes and troubleshooting

Every row here is a real symptom with a real Argo CD state or message behind it. Keep it close during an incident.

Three gotchas cost the most hours, so give them extra words:

1. The migration that poisons every future sync. A PreSync migration that is not idempotent works beautifully on the deploy you tested — the first sync — and then silently becomes a landmine. The next sync (a new commit, a self-heal, anyone clicking Sync) re-runs it, it fails on “table already exists,” the PreSync phase fails, and now the whole app cannot sync at all until someone notices the migration Job’s logs. The blast radius is much larger than “one migration failed”: a non-idempotent PreSync hook can wedge your entire deployment pipeline. Treat idempotency as a hard requirement, not a nicety, and prefer a real migration tool that records applied versions.

2. The wave that waits forever. The health gate between waves is a feature — it is why your app waits for the database — but it has no built-in escape hatch. If a wave-0 resource never becomes Healthy (an image that will never pull, a probe that will never pass), Argo CD sits at wave 0 indefinitely, and every later wave and every PostSync hook never runs. From the outside this looks like “the sync is stuck,” and people go hunting in Argo CD when the cause is a Degraded pod in the current wave. The discipline: when a sync stalls, argocd app get and read the tree — the wave is almost certainly parked on one unhealthy resource, and that resource, not Argo CD, is your bug.

3. The delete policy that eats the evidence. HookFailed sounds symmetric with HookSucceeded, so people set both “to be tidy.” But HookFailed deletes a hook Job the moment it fails — including the logs you need to find out why it failed. When a migration or smoke test fails, you almost always want the failed Job and its pod to stay so you can kubectl logs it. Use HookSucceeded to clean the happy path; leave failures on the floor where you can inspect them. Reserve HookFailed for cleanup work whose failure genuinely does not matter.

---

Common beginner mistakes

These are misconceptions about the model, not symptoms to grep for (those live in the troubleshooting table). Each one is a wrong mental picture that sends people down the wrong path before anything even breaks.

• “Sync waves are depends_on — I’m declaring that X depends on Y.” They are not dependency edges. A wave is a single global integer that sorts everything, with a health gate between groups. You cannot say “the Deployment depends on the Secret”; you can only say “the Secret is in an earlier wave than the Deployment,” and every resource in that earlier wave must be Healthy before any later resource applies. Right model: ordered buckets with a health checkpoint between them, not a per-resource DAG.

• “Hooks run once, ever.” A hook runs once per sync — and a new commit, a manual click, and a self-heal are each a sync. This is the single costliest beginner error, because a non-idempotent PreSync migration works on the deploy you tested and then wedges every future sync. Right model: every hook body must be safe to run again, unconditionally.

• “More waves means a safer deploy.” Waves you do not need make every deploy slower (each adds a health-gate wait) and paper over the fact that Kubernetes already converges on its own. Ordering is a cost you pay only to satisfy a real prerequisite. Right model: default to no waves; add one only where a genuine “must be Healthy first” exists.

• “A hook is a special kind of object.” A hook is an ordinary Job (or Pod) with one annotation. Delete argocd.argoproj.io/hook and the exact same manifest becomes permanent desired state that Argo CD keeps alive and prunes; add it back and it becomes transient run-once work. Right model: the annotation, not the manifest, decides transient-vs-permanent.

• “Wave numbers must be 0, 1, 2, contiguous.” Only the relative order matters. Spread them out — -10, 0, 10 — so you can insert a new tier later without renumbering everything. Right model: gaps are free; leave room.

• “A PostSync hook with a low wave runs early.” The phase always dominates the wave. A PostSync hook annotated sync-wave: "-5" still runs after every Sync-phase resource, because PostSync is a whole phase later; the -5 only orders it against other PostSync hooks. Right model: phase first, then wave within the phase — never mix the two axes in your head.

---

Cheat-sheet

Bookmark this. It answers “what annotation do I need, and how do I watch it run?”

The annotations (all under metadata.annotations):

The phases, in run order:

Commands to drive and watch ordering:

Rules of thumb: migrations → PreSync + idempotent + HookSucceeded; tests → PostSync + retries; cleanup → SyncFail; “database Healthy before app” → waves, not hooks; and remember waves never cross Applications.

---

Practice challenges

Six exercises, beginner to advanced. Try each before opening the solution; every answer is a real annotation, manifest, or command you can run against a local kind cluster with Argo CD installed.

Challenge 1 (beginner) — put a namespace first. You have a Namespace and a ResourceQuota in the same Application, and the quota sometimes errors with namespaces "x" not found. Add one annotation so the namespace always applies in an earlier group.

<details> <summary>Solution</summary>

Annotate the namespace with an earlier wave (the quota stays at the default 0):

metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "-1"   # quoted string, lower applies first

Why: lower wave numbers apply first, and the namespace becomes Healthy before wave 0 (the quota) starts.

</details>

Challenge 2 (beginner) — turn a Job into a PreSync migration that tidies up. You have an ordinary migration Job. Make it run before the app rolls, and delete itself when it succeeds while keeping its pod on failure for debugging.

<details> <summary>Solution</summary>

metadata:
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded

Why: hook: PreSync flips the Job from permanent state to run-once work that gates the Sync phase; HookSucceeded cleans the happy path but leaves failed Jobs so you can read their logs.

</details>

Challenge 3 (intermediate) — serialize two PreSync hooks. A backup Job and a migration Job are both annotated PreSync, but they run at the same time and the migration occasionally starts before the backup finishes. Force the backup to complete first.

<details> <summary>Solution</summary>

Give them different waves — the phase alone does not serialize them:

# backup Job
argocd.argoproj.io/hook: PreSync
argocd.argoproj.io/sync-wave: "-2"
# migration Job
argocd.argoproj.io/hook: PreSync
argocd.argoproj.io/sync-wave: "-1"

Why: hooks that share a phase and wave run in parallel; the sync-wave is the only lever that orders hooks within one phase.

</details>

Challenge 4 (intermediate) — stop a stuck rollout from hanging forever. A wave-0 Deployment references an image tag that will never pull, so the sync sits in Progressing indefinitely. Make the wave fail within about two minutes instead of hanging.

<details> <summary>Solution</summary>

Lower the Deployment’s own progress deadline:

spec:
  progressDeadlineSeconds: 120   # after this, the Deployment reports Degraded

Why: the health gate is bounded by each resource’s progress deadline (default 600s); once it elapses the Deployment goes Degraded, which fails the wave and surfaces the real bug fast. (For a hook Job, activeDeadlineSeconds plays the same role.)

</details>

Challenge 5 (advanced) — fix CRD-before-CR on a first sync. A single Application ships a CustomResourceDefinition and an instance of that new kind. The very first sync fails with no matches for kind "Widget". Give two independent fixes.

<details> <summary>Solution</summary>

Either separate them by wave:

# the CRD
argocd.argoproj.io/sync-wave: "-1"
# the Widget custom resource
argocd.argoproj.io/sync-wave: "0"

or tell Argo CD not to dry-run the CR against a not-yet-known kind:

# on the custom resource
argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true

Why: both approaches stop the CR from being validated before the freshly-applied CRD is established — the wave adds a health-gated pause, the sync-option skips the failing dry-run.

</details>

Challenge 6 (advanced) — order two Applications. Your operator lives in Application A and the custom resources that need it live in Application B. You add sync-wave annotations to B’s resources, but B never waits for A. Explain why, and order them correctly.

<details> <summary>Solution</summary>

Waves only order resources within one Application’s sync, so B’s waves are invisible to A. Order the child Application objects instead — an app-of-apps where the parent puts A in an earlier wave than B:

# the child Application object for A
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "-1"
# the child Application object for B stays at wave "0"

Why: the child Application is itself a resource in the parent’s sync, so waving it is what serializes the two stacks; ApplicationSet RollingSync is the progressive-sync alternative.

</details>

---

Interview and exam questions

Q: What problem do sync waves and hooks solve, and how do they differ? A: Argo CD applies an Application’s resources roughly at once, but some resources have prerequisites (a CRD before its CR, a migration before the app, a namespace before its quota). Sync waves order normal resources into numbered groups that apply in sequence, with Argo CD waiting for each group to be Healthy before the next. Hooks attach run-once work (usually a Job) to a phase of the sync — before it (PreSync), during (Sync), after all-Healthy (PostSync), or on failure (SyncFail). Waves order permanent state; hooks run transient work at sync boundaries.

Q: A resource has sync-wave: "-1". What does that mean, and what is the default wave? A: It applies in an earlier group than the default. Argo CD applies lower wave numbers first (−1 before 0 before 1), and waits for each wave to become Healthy before starting the next. The default for any un-annotated resource is wave 0. The value is a quoted string.

Q: Explain the full ordering precedence Argo CD uses in a sync. A: Four keys, coarsest to finest: phase (PreSync → Sync → PostSync, run strictly in sequence), then wave (lower first within a phase), then kind (a built-in order — Namespaces and CRDs early, custom resources late), then name (alphabetical). So the phase always dominates the wave: a PreSync hook runs before every Sync-phase resource regardless of wave numbers.

Q: Why must a PreSync migration Job be idempotent? A: Because hooks run on every sync, not just the first — a new commit, a manual sync, or a self-heal each re-runs the PreSync hook. A non-idempotent migration (CREATE TABLE with no guard) succeeds the first time and then fails on “already exists,” which fails the PreSync phase and can wedge the entire app’s syncing. Use a migration framework that records applied versions, or IF NOT EXISTS/guard queries.

Q: Name the three hook delete policies and when each deletes the hook. A: HookSucceeded deletes the hook after it succeeds; HookFailed deletes it after it fails; BeforeHookCreation (the default) deletes the previous instance right before the next sync creates a new one. For a passing one-shot you want HookSucceeded; you usually avoid HookFailed because it deletes the logs you need to debug a failure.

Q: A sync is stuck — it never leaves one wave. How do you diagnose it? A: The wave is health-gated, so it is almost certainly parked on a resource that never became Healthy. Run argocd app get, find the Degraded/Progressing child in that wave (bad image → ImagePullBackOff, failing probe, exceeded progress deadline), and fix that resource — Argo CD is correctly refusing to advance until the wave is Healthy. Every later wave and PostSync hook is blocked behind it.

Q: Your CRD and its custom resources are in the same sync and the CR apply fails with “no matches for kind.” Why, and how do you fix it? A: The custom resource was dry-run/applied before the API server finished establishing the freshly-applied CRD. Fix it by putting the CRD in an earlier sync wave than the CR, or by adding argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true to the custom resource so Argo CD does not fail the dry-run on the not-yet-known kind.

Q: What is the difference between a Sync-phase hook and a plain resource? A: Almost none in timing — an un-annotated resource is implicitly in the Sync phase. The difference is lifecycle: a hook is tracked as run-once work and governed by a delete policy, whereas a plain resource is permanent desired state that self-heals and is pruned when removed from Git. The annotation is what flips a Job between the two.

Q: Do sync waves order resources across two different Applications? A: No. Waves order resources within a single Application’s sync. Two Applications sync independently and do not honor each other’s wave numbers. To order across Applications, put waves on the child Application objects in an app-of-apps, or gate on health between stacked ApplicationSets.

Q: A Helm chart has a helm.sh/hook: pre-upgrade Job. Does it run under Argo CD? A: Yes — Argo CD renders the chart with helm template and re-expresses install/upgrade hooks as its own phases: pre-install/pre-upgrade → PreSync, post-install/post-upgrade → PostSync, and helm.sh/hook-weight behaves like a sync-wave. What does not translate: helm test, pre-delete, and *-rollback hooks, because there is no Helm test/delete/rollback lifecycle in the render model.

Q: When does a SyncFail hook fire, and when does it surprisingly not? A: It fires when the sync operation reaches a failed phase — for example a PreSync hook failed, or a resource could not apply. It does not fire on a healthy sync, and it can appear not to fire if the sync merely hung (a hook Job running forever) rather than failing, or if the operation timed out in a state Argo CD did not classify as failed. Give hooks an activeDeadlineSeconds/backoffLimit so they fail cleanly and let SyncFail trigger.

---

Glossary

• Sync wave — an integer you set with argocd.argoproj.io/sync-wave: "N" that sorts a resource into an ordered group. Lower applies first; default is 0; the value is a quoted string.
• Resource hook — an ordinary resource (almost always a Job) annotated argocd.argoproj.io/hook: <phase> so Argo CD runs it at a specific point in the sync instead of treating it as permanent state.
• Phase — the coarsest ordering axis of a sync: PreSync → Sync → PostSync, run strictly in sequence, with SyncFail as the failure branch. The phase always dominates the wave.
• PreSync / Sync / PostSync / SyncFail — run before the app changes / with the app / after the app is Healthy / only when the operation fails. An un-annotated resource is implicitly in the Sync phase.
• Skip / PostDelete — Skip tells Argo CD not to apply a manifest at all; PostDelete runs when the whole Application is deleted (tied to the cascade finalizer).
• Health gate / health-gated — Argo CD applies a wave, then waits for every resource in it to report Healthy before starting the next wave. This is what makes ordering meaningful rather than cosmetic.
• Hook delete policy — argocd.argoproj.io/hook-delete-policy, one of HookSucceeded, HookFailed, or BeforeHookCreation (the default), controlling when a finished hook resource is removed.
• Idempotent — safe to run repeatedly with the same end result. Mandatory for PreSync work because hooks run on every sync, not just the first.
• Sync operation — one execution of applying Git’s desired state to the cluster, spanning all phases. Triggered by a manual sync, a new commit under auto-sync, or a self-heal.
• Sort key (phase → wave → kind → name) — the four-part order Argo CD applies within a sync: phase, then wave, then a built-in kind order, then alphabetical name.
• Kind order — Argo CD’s fixed, built-in ordering that already puts Namespace and CustomResourceDefinition early and custom resources late, before any wave you add.
• generateName vs name — a Job with a fixed name is replaced each sync (one leftover at a time); one with generateName gets a fresh random name each sync and will pile up unless a delete policy cleans it.
• activeDeadlineSeconds / backoffLimit — a Job’s hard wall-clock limit and its retry count; together they stop a hook from hanging a phase and make it fail cleanly so SyncFail can fire.
• resources-finalizer.argocd.argoproj.io — the Application-level finalizer that makes a delete cascade to managed resources; it also governs when PostDelete hooks run.
• SkipDryRunOnMissingResource — a sync option that stops Argo CD failing a resource’s dry-run when its kind is not yet known to the API server — the classic CRD-before-CR fix.

---

Key takeaways

• Two mechanisms, two questions. Sync waves order the normal resources within one sync (lowest wave first, health-gated between waves); hooks attach run-once work to a sync phase. Waves are for permanent state that needs ordering; hooks are for migrations, tests, and cleanup.
• The full order is phase → wave → kind → name, and phase always wins: a PreSync hook runs before every Sync-phase resource no matter the wave. Even with no waves, Argo CD’s built-in kind order already applies Namespaces and CRDs first.
• Waves gate on health. Argo CD applies a wave, waits for every resource in it to be Healthy, then starts the next — which is how a database (wave 0) precedes its app (wave 1). A resource that never goes Healthy stalls the sync at that wave forever.
• The four phases: PreSync (before the sync — migrations, backups; must succeed), Sync (with your resources), PostSync (after all-Healthy — smoke tests, notifications), SyncFail (on a failed operation — cleanup). Plus Skip (do not apply) and PostDelete (on app deletion).
• Hooks run on every sync, so PreSync work must be idempotent. There is no “run once ever” hook; a non-idempotent migration will fail on the second sync and can wedge the whole app. Use a migration framework or IF NOT EXISTS/guards.
• Set a delete policy or drown in Jobs. Default is BeforeHookCreation (keeps one, cleans on the next sync, not on success); use HookSucceeded to clean passing one-shots promptly; avoid HookFailed so you keep failure logs to debug.
• Helm hooks become Argo CD hooks at render time: pre-* → PreSync, post-* → PostSync, hook-weight → sync-wave, delete policies map across; test/delete/rollback hooks are not honoured.
• Waves never cross Applications, and this whole topic is cloud-neutral — identical on AKS, EKS, GKE, and kind. The only per-cloud edge is how a migration Job’s ServiceAccount authenticates to a managed database (Workload Identity / IRSA / Workload Identity), not the ordering itself.