Troubleshooting Argo CD Part 1 — OutOfSync, Degraded, Unknown & ComparisonError

In a nutshell

A red badge in Argo CD is a symptom, not a diagnosis — and the four states this lesson covers (OutOfSync, Degraded, Unknown, and the ComparisonError behind Unknown) are the loud, common, usually-harmless ones you meet first. “Loud but harmless” means they shout on the dashboard yet rarely mean Argo CD itself broke: nine times in ten the real cause is your own workload, another controller, or your Git source. The skill this lesson builds is turning that noise into a precise answer — which component is at fault and which single command proves it — instead of guessing and fixing the wrong half of the system.

The mental model is a car’s dashboard. A check-engine light tells you something is wrong, not what; a good mechanic plugs in the reader and follows the trouble codes in order before touching a part, while a bad one swaps random components until the light happens to go off. Argo CD’s “trouble codes,” in order, are: the state (argocd app get), the diff or conditions (argocd app diff), the Kubernetes events (kubectl describe), and finally the component logs. Read them in order and every red badge names its own cause.

Level: Advanced · Time: ~58 min

You’ll need first: the two status axes from Sync Status & Health Assessment and basic argocd/kubectl. After this you can: look at any OutOfSync / Degraded / Unknown / ComparisonError app, run the one command that confirms the cause, and name the exact component to blame — the application-controller, the repo-server, or your own workload — before you change anything.

Every Argo CD incident starts the same way: a badge turns red, someone pings the platform channel, and the next thirty minutes decide whether this is a two-minute fix or a two-hour goose chase. The difference is almost never knowledge of some obscure flag. It is method. The engineer who fixes it fast doesn’t know more Argo CD trivia than you — they refuse to guess. They read the state, read the events, read the component logs, and isolate the failing layer before they touch anything.

The engineer who burns the afternoon does the opposite. They see OutOfSync, assume the repository is broken, and start rotating credentials — while the real cause is an HPA quietly rewriting replicas. Or they see a red health badge, assume Argo CD “broke the deploy,” and reinstall the controller — while the real cause is a typo in their own image tag that Argo CD applied perfectly. Guessing sends you to the wrong half of the system, and the wrong half of the system is a big place.

This lesson is the systematic debugging playbook for the four states you will meet most: OutOfSync, Degraded, Unknown, and the ComparisonError condition behind Unknown. We build the diagnostic method first — a decision tree you can run in your head — then go deep on each failure class, always answering the same three questions: what does this state actually mean, which single command confirms the cause, and which component do I blame? Every state, error string, and command here is real for Argo CD 2.13+/3.x on Kubernetes 1.29+; the console output is representative and labelled as such, because the shape of the output — not one captured transcript — is what you learn to read. Part 2 of this playbook covers sync-time failures — SyncFailed, hook failures, stuck operations, and pruning disasters — in Troubleshooting Argo CD Part 2.

---

Why this matters

Argo CD is a distributed system with several moving parts, and a red badge is a symptom, not a diagnosis. The same OutOfSync badge can mean “a commit hasn’t synced yet,” “someone ran kubectl edit,” or “a controller is fighting the diff forever.” The same Unknown can mean “the repo is down,” “the Helm chart won’t render,” or “the repo-server pod is OOMKilling.” You cannot fix a symptom; you can only fix the specific cause behind it, and the whole skill is getting from symptom to cause quickly and without thrashing.

The trap is that Argo CD’s states are reassuringly specific-looking. Degraded sounds like Argo CD is degraded. OutOfSync sounds like a sync problem. Both readings are usually wrong: Degraded almost always means your workload failed, and a perpetual OutOfSync usually means another controller — not Argo CD, not your Git — owns a field. The words point you at Argo CD; the causes live somewhere else. A method that forces you to read the evidence instead of trusting the label is the only reliable defence.

This lesson leans hard on one earlier idea: the two independent status axes from Sync Status & Health Assessment. That lesson taught you what Synced/OutOfSync/Unknown and Healthy/Progressing/Degraded mean. This one turns that model into a debugging tool: given a combination, where do you look, what do you run, and who is at fault? If the two-axes idea is fuzzy, skim that lesson first — everything below assumes you can already say, without thinking, “sync is the diff against Git; health is whether the resources work.”

The one-sentence discipline for the whole lesson: read the state, then the events, then the component logs — and name the layer before you touch it. If you can’t yet name which of the three components (application-controller, repo-server, or your own workload) is at fault, you have not finished diagnosing, and any change you make is a guess.

---

The diagnostic method: read, don’t guess

Before any specific failure class, internalise the loop. Every Argo CD problem yields to the same four-layer read, from cheapest and most general to deepest and most specific. You almost never need all four — most incidents are solved at layer 1 or 2 — but doing them in order is what stops you from thrashing.

The order encodes a principle: start where the answer is most likely and cheapest, and only descend when the layer above doesn’t resolve it. Ninety percent of incidents are named at layer 1 (the resource tree points straight at the broken child) or layer 2 (the diff or the ComparisonError string). You reach for component logs (layer 4) only when the app-level and Kubernetes-level views don’t explain it — a repo-server that won’t render, a controller that isn’t reconciling. Jumping straight to kubectl logs on the repo-server for a Degraded app is the classic rookie move: the repo-server had nothing to do with it, and you’ll read a thousand lines of irrelevant gRPC chatter.

The second half of the method is isolation: once you know which axis is red, you know which layer can possibly be responsible, which slashes the search space before you run a single deep command.

The whole method collapses into one picture — the decision tree you run from any red badge. Read it left to right: you see a red symptom, you read the state first rather than guessing, that read routes you to the one probe that fits the symptom, the probe isolates the component to blame, and the component determines the fix.

The badges mark the load-bearing moves: read the state before you guess (1); an unexplained OutOfSync goes to argocd app diff (2); Unknown/ComparisonError goes to the repo-server logs (3); a Degraded resource is your workload’s bug, not Argo CD’s (4); the fix follows the component you isolated (5); and the application-controller owns the diff verdict while the repo-server owns rendering (6). The three coloured lanes — amber for the OutOfSync path, purple for Unknown, red for Degraded — never cross: each symptom has its own probe, its own component, and its own fix, which is exactly why guessing across lanes wastes the incident.

---

The state matrix, now for debugging

The sync/health lesson taught the two axes as a concept. Here we weaponise them: a lookup table from (sync, health) to “what it means and where to look.” Keep this open during an incident and layer 1 of the method becomes mechanical.

First, re-read the two axes with a debugging lens — what each value tells you about where the fault can be:

Now the payload — the combination lookup. This is the “this state means → look here” table the whole method hangs on:

Two rows deserve to be memorised because they are the two most misdiagnosed states in all of Argo CD. Synced + Degraded is the one that ends careers-worth of wasted afternoons: the big green Synced is so reassuring that people hunt everywhere except their own manifest — but Synced is proof Argo CD did exactly what Git said, so a broken result means the bug is in what you committed. And Unknown is the one people escalate to “Argo CD is down” — but Unknown is a narrow, specific failure of the comparison (the repo-server couldn’t render or reach the source), and it never implicates your running pods.

Finally, tie each verdict to the component that produces it — because the component is what you’ll eventually read logs from. This mapping comes straight from the Argo CD architecture:

---

Perpetual OutOfSync: the causes catalogue and the fix ladder

An OutOfSync that a sync fixes is not a problem — you sync, it goes green, done. The pathological case is perpetual OutOfSync: the app flips back to OutOfSync seconds after every sync, no Git change involved, and with selfHeal on it flaps visibly every reconcile. This is the single most common Argo CD support ticket, and the diagnosis is always the same shape: a field exists on the live object that isn’t in your rendered manifest, and something keeps putting it back.

The full mechanics, the three selection tools, and the lab that reproduces it live in the dedicated Diffing & Drift lesson. Here, the troubleshooting cut: recognise the actor from the symptom in the diff, then climb the fix ladder. Start by naming the culprit — these are the offenders you’ll actually meet, ranked by ticket volume:

The tell that distinguishes perpetual OutOfSync from an ordinary un-synced commit: you sync, it goes green, then it goes yellow again with nothing in Git changed. An un-synced commit stays Synced after you sync it; a controller-owned field can’t stay Synced because the controller writes it right back. argocd app diff shows you the field; --show-managed-fields shows you the owner. Once you have both, climb the ladder — reach for the highest rung that fixes it, because each rung down is more manual and more brittle:

The most important line in that table is the last one. When an app flaps, the tempting one-liner is to disable self-heal — and it’s wrong every time. The flap is a signal that the diff is wrong about ownership; turning off self-heal silences the smoke detector while the app stays OutOfSync and every genuine kubectl edit in prod now goes uncorrected. Fix the diff (rungs 1–3) and leave self-heal doing its job on every other field.

The perpetual-OutOfSync reflex, in three commands: argocd app diff <app> (which field?), kubectl get <kind> <name> --show-managed-fields -o yaml | grep -A6 managedFields (which owner?), then the highest ladder rung that covers it. If you find yourself typing selfHeal: false, stop — you’ve mistaken the smoke detector for the fire.

---

Unknown and ComparisonError: the repo-server couldn’t render

Unknown sync status is categorically different from OutOfSync. OutOfSync means Argo CD ran the comparison and found a difference. Unknown means Argo CD could not run the comparison at all — and that failure is recorded as a ComparisonError condition. Because the comparison begins with the repo-server cloning your repo and rendering manifests, Unknown almost always means the repo-server failed, and the error string tells you exactly how.

Read the condition first. It’s right there in argocd app get:

# The Conditions block names the ComparisonError verbatim
argocd app get payments-api

# (representative output)
Name:            argocd/payments-api
Sync Status:     Unknown
Health Status:   Healthy
CONDITIONS:
  TYPE             MESSAGE                                                        LAST TRANSITION
  ComparisonError  rpc error: code = Unknown desc = failed to generate           2026-07-15T09:14:22Z
                   manifests in deploy/overlays/prod: app path does not exist

The MESSAGE is the whole diagnosis. Here it says app path does not exist — the path in your Application source points at a directory that isn’t in the repo at that revision. No amount of poking at pods or clusters will help; the fix is the path or targetRevision. Learn to map the common ComparisonError strings to their cause:

When the ComparisonError names a render failure (Helm, Kustomize, a plugin), the ground truth is in the repo-server logs. This is the one component whose logs you’ll read most in troubleshooting, so learn to read them:

# The repo-server renders manifests — its log is the source of truth for Unknown/ComparisonError
kubectl -n argocd logs deploy/argocd-repo-server --tail=200 | grep -iE "error|failed|panic"

# (representative repo-server error line, structured logging)
level=error msg="finished unary call with code Unknown"
  error="`helm template ...` failed exit status 1: Error: template: web/templates/deployment.yaml:23:18:
  executing \"web/templates/deployment.yaml\" at <.Values.image.tag>: nil pointer evaluating interface {}.tag"
  grpc.method=GenerateManifest grpc.service=repository.RepoServerService

Two fields orient you instantly: grpc.method=GenerateManifest tells you this is the render path (as opposed to ResolveRevision for revision lookups or listing refs for repo access), and the quoted error= is the raw tool failure — here a Helm template dereferencing a nil .Values.image.tag. That is a values problem in your chart, reproducible offline with helm template, and it has nothing to do with the cluster. The failure modes a repo-server can be in:

The heuristic that saves the most time: if one app is Unknown, suspect that app’s source (path, revision, chart, values). If many apps go Unknown at once, suspect the repo-server itself (OOM, a repo host outage, a bad credential shared across apps). One-app-Unknown is a config bug; all-apps-Unknown is an infrastructure bug, and they send you to completely different fixes.

---

Degraded: your app’s own bug, not Argo CD’s

Degraded is the most emotionally misread state, because a red heart on the Argo CD dashboard feels like Argo CD broke. It didn’t. Health is computed by inspecting each resource’s own live state against a per-kind rule — a Deployment’s available replicas, a Pod’s phase, a Job’s completion. When health is Degraded, Argo CD is a thermometer reporting a fever, not the cause of the fever. And when the app is Synced + Degraded, the point is sharper still: Argo CD applied exactly what Git contained, so the failing result is in what you committed.

The diagnosis flows straight down the resource tree. argocd app get gives you the app rollup and the per-resource breakdown, and the MESSAGE column usually names the cause outright:

argocd app get payments-api

# (representative output — Synced but Degraded)
Sync Status:     Synced to main (a1b2c3d)
Health Status:   Degraded

GROUP  KIND        NAMESPACE  NAME         STATUS  HEALTH     MESSAGE
       Service     payments   api          Synced  Healthy    service/api created
apps   Deployment  payments   api          Synced  Degraded   Deployment "api" exceeded its progress deadline
       Pod         payments   api-7c9-xh2  Synced  Degraded   Back-off pulling image "registry.example.com/api:v9.9.9-typo"

Read it as a drill-down. The app is Synced/Degraded; the tree says the Service is fine but the Deployment hit ProgressDeadlineExceeded because its Pod can’t pull registry.example.com/api:v9.9.9-typo. The worst-child rollup landed the whole app on Degraded, and the MESSAGE hands you the cause: a bad image tag. The fix is in Git (correct the tag), not in Argo CD. The per-kind rules that produce Degraded are worth knowing so you recognise the cause from the message:

When the MESSAGE column isn’t enough — a crash loop whose reason is in the application’s own output — descend to layer 3 (events and logs). Three tools, in rough order of how deep they take you:

The events are where “the pod is Degraded” becomes “the pod can’t reach its database because the ConfigMap has the wrong host.” A representative describe for the image case above:

kubectl -n payments describe pod api-7c9-xh2

# (representative Events section)
Events:
  Type     Reason     Age                 From     Message
  ----     ------     ----                ----     -------
  Normal   Scheduled  2m                  default-scheduler  Successfully assigned payments/api-7c9-xh2 to node-1
  Normal   Pulling    2m (x4 over 2m)     kubelet  Pulling image "registry.example.com/api:v9.9.9-typo"
  Warning  Failed     2m (x4 over 2m)     kubelet  Failed to pull image "...": not found: manifest unknown
  Warning  Failed     2m (x4 over 2m)     kubelet  Error: ErrImagePull
  Warning  Failed     40s (x6 over 2m)    kubelet  Error: ImagePullBackOff

manifest unknown from the registry confirms the tag doesn’t exist — a Git-side typo, fixed by a commit. Note what you did not do: you never touched the repo-server, the controller, or Argo CD config, because health problems don’t live there. This is the discipline the whole lesson is built to instil — the red badge is on the workload lane, so you stayed in the workload lane.

---

Connectivity failures: repo, cluster, and auth

A distinct failure family has nothing to do with your manifests: Argo CD can’t reach something. There are exactly three network relationships that can break, and they surface as different states, so knowing which relationship failed tells you which state you’ll see and which fix applies.

Read the error code, not just the message — the code localises the failure:

The repo side is cloud-neutral — a Git host is a Git host. The cluster side is where a genuine cloud edge appears, because Argo CD authenticates to each managed control plane differently, and the failure signatures differ. Argo CD stores per-cluster credentials in a Secret (labelled argocd.argoproj.io/secret-type: cluster) — a bearer token, client cert, or an execProviderConfig that mints short-lived tokens. When the controller suddenly can’t reach a cluster it synced yesterday, this is the table to reach for:

The one-line rule for the cloud edge: i/o timeout is a network problem (no path to the API server — usually a private control plane), while Unauthorized/Unauthenticated is a credentials problem (expired token, unmapped identity). They look similar in a dashboard and have completely different fixes, and the error string is what tells them apart. The full mechanics of registering clusters and rotating their credentials belong to the multi-cluster registration material; here, recognise the signature and know which knob to turn.

---

Cache staleness: when the truth is stale, not wrong

Occasionally the state Argo CD shows you is simply old. Argo CD caches two expensive things in Redis: the rendered manifests (keyed by repo/revision/path, so it doesn’t re-run helm template every three minutes) and the live cluster state (so the UI is fast). When a cache entry is stale, you get a confusing class of symptom: the app shows OutOfSync for a change you already reverted, or Synced against a commit you know you changed, or a diff that doesn’t match reality. The cache is an optimisation, never a source of truth — Git and the live cluster are authoritative — so the fix is always “invalidate the cache and recompute.”

The refresh spectrum, from lightest to heaviest, matters because each level invalidates a different cache:

--hard-refresh is the one to know: it’s the correct response to “I edited argocd-cm (a health check, an ignoreDifferences rule) and the app doesn’t reflect it,” because those inputs feed the cached render. And because Redis holds no source-of-truth data, restarting or flushing it is safe — you lose speed for a minute while caches rebuild, never data:

If a state looks impossible — OutOfSync on a field you reverted, Synced on a commit you changed, a diff that contradicts kubectl get — suspect the cache before you suspect a bug. argocd app get <app> --hard-refresh re-renders from scratch and resolves the large majority of “Argo CD is showing me something that isn’t true” reports.

---

Going deeper

The body above covers the four states you meet daily and the four-layer read that cracks them. This section is for the cases that break the tidy rules — where Unknown is not the repo-server, where the CLI you rely on is the thing that’s down, and where the red badge is one level removed from the resource actually at fault. None of it replaces the method; it extends the method to the edges.

Unknown that isn’t a ComparisonError: the other conditions

The body equates Unknown with ComparisonError — the repo-server couldn’t render. That is the common case, but the Conditions: block emits several other types, and each routes you somewhere completely different. The type is the routing key; read it before you assume “repo-server.”

The one that trips up experienced engineers is InvalidSpecError, because it also shows Sync Status: Unknown — so the reflex (“Unknown → repo-server logs”) sends you to a component that did nothing wrong. InvalidSpecError means the application-controller refused the spec during validation, before the repo-server was ever asked to render. Representative:

# argocd app get checkout   (representative)
Sync Status:     Unknown
CONDITIONS:
  TYPE              MESSAGE
  InvalidSpecError  application destination server https://kubernetes.example.com and
                    namespace prod do not match any of the allowed destinations in
                    project 'team-a'

That is not a rendering problem, and no repo-server log will mention it. The fix is in the AppProject (its destinations, sourceRepos, or clusterResourceWhitelist) or in the app’s own destination/source — the boundary rules covered in AppProjects & multi-tenancy. Other InvalidSpecError strings you’ll meet: Application referencing project 'X' which does not exist, application repo … is not permitted in project 'X', and cluster 'https://…' has not been configured (the destination cluster was never registered). All are spec/project bugs; none is a repo-server bug.

Git-provider rate limits: many apps, one repo, intermittent Unknown

The body’s connectivity table has “repo down” and “repo auth.” A third, sneakier repo failure is throttling. Every Application polls its source on the reconciliation timer (default timeout.reconciliation: 180s) and each ResolveRevision is a git ls-remote; point a few hundred apps at one GitHub/GitLab org and you can trip the provider’s API rate limit. The signature is distinctive — apps flip to Unknown intermittently and in groups, clearing and returning, with a condition like:

ComparisonError  failed to list refs: ... 429 Too Many Requests
                 (or: 403 API rate limit exceeded / secondary rate limit)

Any single app tested alone is fine, which is exactly why this looks like a ghost. The fix is to stop polling so hard: drive refreshes with a Git push webhook and raise timeout.reconciliation so background polling drops, and authenticate with a credential that carries a higher limit (a GitHub App installation token far exceeds a personal access token). The credential and webhook mechanics live in Repositories & private-repo credentials. The diagnostic tell holds constant: one app Unknown is that app’s source; many apps Unknown together is shared infrastructure — and rate-limiting is the shared-infra cause the body’s OOM/outage rows don’t name.

When the CLI itself is down: read status straight off the CR

Every command so far assumes argocd app get works. But one failure mode the body lists is argocd-server (the API/UI) down — and when it is, the CLI and dashboard go dark with it. You are not blind. The application-controller writes every verdict directly onto the Application custom resource’s .status, independent of the API server, so kubectl still reads the ground truth:

# Sync + health without the Argo CD API at all
kubectl -n argocd get application payments-api \
  -o jsonpath='{.status.sync.status} / {.status.health.status}{"\n"}'

# The conditions (ComparisonError, InvalidSpecError, …) and the last sync operation
kubectl -n argocd get application payments-api -o yaml | yq '.status.conditions, .status.operationState'

kubectl get application (short name app) works whenever the controller and the Kubernetes API are alive — which is almost always, because argocd-server being down doesn’t stop reconciliation. .status.operationState.phase/.message even tells you how the last sync finished. This is the fallback that turns “the UI is down, I’m blind” into “the UI is down, so I read the CR.”

The red is one level up: app-of-apps and the Application health check

Argo CD ships a built-in health check for its own Application kind, which is what makes the app-of-apps pattern legible: a parent Application manages child Application resources, and the parent’s sync/health roll up the children’s. So a parent that reads OutOfSync/Degraded while its own manifests (a folder of Application YAMLs) are untouched is almost never a bug in the parent — it’s a child app that went red, bubbling up. Drilling the parent’s manifests is wasted effort; the move is to open the tree, find the child Application that’s unhealthy, and run the same four-layer method on that app, which has its own sync and health axes. The pattern and its failure modes are covered in the app-of-apps lesson; the troubleshooting cut is simply: a parent’s red is a pointer, not a diagnosis — follow it down one level.

Detecting these states before a human does — metrics and scale

At fleet scale you don’t want to discover a Degraded app from a Slack ping. The application-controller exports argocd_app_info{sync_status="…",health_status="…"} (a gauge per app) and argocd_app_reconcile (a reconcile-duration histogram); alerting on “app OutOfSync/Degraded/Unknown for > N minutes” turns every state in this lesson into a proactive signal — the wiring is in the observability lesson. One scale-specific failure worth naming: the application-controller can be sharded across clusters, and if a shard is crashed or unbalanced, the apps it owns simply stop reconciling — they look frozen (stale state, a --refresh that never lands) rather than red. argocd admin cluster stats shows shard assignment; a whole shard’s worth of apps going stale at once is a controller/sharding problem, not a per-app one (its own scaling & sharding lesson goes deep).

The essential tools: which command, which log, which component

You’ve now met every tool in context. This section is the consolidated reference — the toolbox, and the map from symptom to the component whose logs actually help. Start with the command toolbox, ordered by how often a real incident uses it:

The single most valuable reference in this whole lesson is the which-log-for-which-symptom map. Reading the wrong component’s logs is the most common way to waste time after you’ve already identified the symptom — this table stops that:

Finally, argocd admin — the operator-side subcommands that answer questions the app-level CLI can’t, indispensable when the problem is Argo CD’s own configuration or its cluster-level bookkeeping:

---

Walk a real OutOfSync to root cause

Method beats memorisation, so here is the method run end to end on a realistic incident — including the wrong guess it prevents. The alert: payments-api has been OutOfSync for an hour and re-syncing doesn’t fix it. The tempting guess (“the repo credentials expired”) is plausible and wrong, and the method is what saves you from chasing it.

Layer 1 — read the state. Never open a log before this.

argocd app get payments-api

# (representative output)
Sync Status:     OutOfSync from main (a1b2c3d)
Health Status:   Healthy
CONDITIONS:      <none>

GROUP  KIND        NAMESPACE  NAME  STATUS     HEALTH   MESSAGE
apps   Deployment  payments   api   OutOfSync  Healthy  deployment.apps/api configured

Two facts already kill the “repo credentials” guess. First, the sync status is OutOfSync, not Unknown — Argo CD ran the comparison and found a real diff, so the repo-server reached and rendered the source fine (a credential failure would be Unknown + ComparisonError). Second, there are no conditions. And health is Healthy, so nothing is crashing. This is a pure diff problem on one Deployment, and the app is running fine — which also means it’s not urgent in the way a crash loop would be. Isolation done: the fault is on the sync axis, the diff itself.

Layer 2 — read the diff. What field actually differs?

argocd app diff payments-api

# (representative output)
===== apps/Deployment payments/api ======
5c5
<   replicas: 6      # live: the cluster has 6
---
>   replicas: 2      # desired: Git says 2

The < line is live, the > line is Git. The only difference is replicas — live has 6, Git says 2. Nothing else drifted. And critically, you didn’t put 6 anywhere. Something is writing replicas on the live Deployment. This is the perpetual-OutOfSync fingerprint: a single field, owned by not-you, that a sync can’t hold because it’s rewritten immediately.

Layer 3/4 — find the owner. Ask Kubernetes who last wrote that field.

kubectl -n payments get deploy api --show-managed-fields -o yaml | grep -A6 managedFields

# (representative output)
  managedFields:
  - manager: kube-controller-manager
    operation: Update
    subresource: scale
    fieldsV1:
      f:spec:
        f:replicas: {}

There it is: manager: kube-controller-manager, subresource: scale. An HPA is scaling the Deployment to 6 through the scale subresource, and kube-controller-manager is the field-manager on record. Confirm the HPA exists (kubectl -n payments get hpa), and the root cause is nailed: the app declares replicas: 2 in Git, an HPA owns the real replica count, and the two disagree permanently. With selfHeal on, this would visibly flap; here it just sits OutOfSync.

The fix — the right ladder rung. Server-Side Diff (rung 1) doesn’t reproduce the HPA’s scale-subresource write, so the correct, self-maintaining fix is rung 2, managedFieldsManagers:

# On the Application spec — stop diffing whatever kube-controller-manager owns
spec:
  ignoreDifferences:
    - group: apps
      kind: Deployment
      name: api
      managedFieldsManagers:
        - kube-controller-manager

Commit it (the Application is itself in Git), let it sync, and argocd app get reads Synced/Healthy — stably, no flap. Total elapsed: three commands to root cause, one small config change to fix. The “repo credentials” guess would have had you rotating tokens, re-adding the repo, and restarting the repo-server — three fixes for a problem in none of those places. That is the entire value of the method: OutOfSync (not Unknown) plus no conditions ruled out the repo in the first ten seconds, and argocd app diff named the field in the next ten.

---

Hands-on lab: diagnostic drills

This lab is deliberately analysis-level and needs no live cluster — troubleshooting is a reading skill, and these drills train the read. You are handed four broken apps, each as the representative output you’d actually see, and your job is to run the method in your head: name the red axis, predict the confirming command, identify the component to blame, and state the fix. Work each one before reading the resolution. (If you want to reproduce the states for real, the Diffing lesson has a full kind-based lab for the HPA case — nothing here provisions anything, so there is nothing to tear down.)

Drill 1 — perpetual OutOfSync. You’re handed:

# argocd app get shop-web  (representative)
Sync Status:     OutOfSync from main (7f3a1e0)
Health Status:   Healthy
CONDITIONS:      <none>

# argocd app diff shop-web  (representative)
===== apps/Deployment shop/web ======
5c5
<   replicas: 4
---
>   replicas: 2

Diagnose: Red axis is sync, and it’s OutOfSync with no conditions and Healthy — so the repo rendered fine and nothing is crashing. The diff shows a single field, replicas, live (4) ≠ Git (2), and you didn’t write 4. Confirming command: kubectl get deploy web --show-managed-fields -o yaml | grep -A6 managedFields → expect kube-controller-manager/subresource: scale. Component to blame: an HPA (not Argo CD, not the repo). Fix: climb the ladder — managedFieldsManagers: [kube-controller-manager] (rung 2), not selfHeal: false.

Drill 2 — ComparisonError from a bad path. You’re handed:

# argocd app get billing  (representative)
Sync Status:     Unknown
Health Status:   Healthy
CONDITIONS:
  TYPE             MESSAGE
  ComparisonError  rpc error: code = Unknown desc = failed to generate manifests
                   in deploy/overlays/production: app path does not exist

Diagnose: Red axis is sync = Unknown, which means the comparison never ran — a ComparisonError. You don’t touch pods or clusters for Unknown. The message names it: app path does not exist for deploy/overlays/production. Confirming command: kubectl -n argocd logs deploy/argocd-repo-server | grep GenerateManifest, and check the repo — the directory is probably deploy/overlays/prod, not production, or the folder was moved. Component to blame: the repo-server (rendering), driven by a wrong spec.source.path or targetRevision. Fix: correct the path (or the revision) on the Application; hard-refresh.

Drill 3 — Degraded from a bad image. You’re handed:

# argocd app get orders  (representative)
Sync Status:     Synced to main (c4d5e6f)
Health Status:   Degraded

GROUP  KIND        NAMESPACE  NAME         STATUS  HEALTH     MESSAGE
apps   Deployment  orders     api          Synced  Degraded   Deployment "api" exceeded its progress deadline
       Pod         orders     api-5b8-qk9  Synced  Degraded   Back-off pulling image "ghcr.io/acme/orders:v2.4.O"

Diagnose: Red axis is health = Degraded, and sync is Synced — the career-saver combination. Synced proves Argo CD applied exactly what Git said, so the bug is in the commit. The MESSAGE shows an image pull backing off on tag v2.4.O — note the letter O where a zero 0 belongs. Confirming command: kubectl -n orders describe pod api-5b8-qk9 → expect ErrImagePull/manifest unknown. Component to blame: your workload (a typo’d image tag), not Argo CD. Fix: correct the tag to v2.4.0 in Git, commit, sync. You never open a repo-server or controller log — health problems aren’t there.

Drill 4 — Unknown from a repo-auth failure. You’re handed:

# argocd app get analytics  (representative)
Sync Status:     Unknown
Health Status:   Healthy
CONDITIONS:
  TYPE             MESSAGE
  ComparisonError  rpc error: code = Unknown desc = failed to list refs:
                   authentication required

Diagnose: Red axis is sync = Unknown again, but the message is different: failed to list refs: authentication required — Argo CD couldn’t even list the repo’s refs, so this is repo access/credentials, one step earlier than a render error. Confirming command: argocd repo get https://git.acme.com/analytics.git (expect a failed connection state), and kubectl -n argocd logs deploy/argocd-repo-server | grep -i "authentication required". Component to blame: the repo-server, failing on a bad/expired credential (rotated token, wrong username, revoked deploy key). Fix: update the repository Secret (re-argocd repo add with a valid token); if many repos failed at once, suspect an org-wide token/SSO change rather than one app.

Now the pattern across all four — the whole point of the drills is that the method is identical every time, only the lane changes:

If you can look at each block, name the axis, and predict the confirming command before reading the resolution, you have the skill this lesson exists to build. The states change; the four-layer read does not.

---

Common mistakes and troubleshooting

This is the reference table the lesson is built around — every state and failure class, its likely cause, the one command that confirms it, and the fix. Keep it open during real incidents.

Three gotchas cost the most hours, and each is a failure of method, not knowledge:

1. Trusting the label instead of reading the evidence. Degraded reads like “Argo CD is degraded”; OutOfSync reads like “a sync problem.” Both readings send you to the wrong lane. The fix is mechanical: before forming any hypothesis, run argocd app get and ask which axis is red and which resource. Synced/Degraded is your workload; Unknown is the repo-server; perpetual OutOfSync is another controller. The label is a symptom; the tree is the evidence.

2. Reading the wrong component’s logs. Once people learn kubectl -n argocd logs exists, they run it for everything — including Degraded apps, where the repo-server and controller are entirely innocent and you’ll scroll a thousand irrelevant gRPC lines. Use the which-log-for-which-symptom map: Unknown → repo-server, stuck reconcile → controller, Degraded → not Argo CD at all (kubectl describe/logs the workload), UI down → server, SSO → Dex. Match the component to the symptom before you tail anything.

3. Reaching for selfHeal: false to stop a flap. It’s the most tempting wrong fix in Argo CD. Disabling self-heal doesn’t make the app Synced — it’s still OutOfSync — and it switches off drift correction for every field, so the next real out-of-band change in prod goes silently uncorrected. A flap is a diagnosis: the diff is wrong about who owns a field. Fix the diff (Server-Side Diff, managed fields, or a scoped pointer) and keep self-heal protecting everything else.

---

Cheat-sheet

Bookmark this. It compresses the whole method into three lookups: the decision tree, the component map, and the state matrix — plus the commands that drive them.

The diagnostic decision tree (run it top to bottom):

Which log for which component:

The state matrix, condensed:

Diagnostic commands, the short list:

---

Interview and exam questions

Q: An app is OutOfSync and re-syncing doesn’t fix it. Walk me through your diagnosis without guessing. A: Layer 1, argocd app get: confirm it’s OutOfSync (not Unknown) and note the conditions and health — OutOfSync with no conditions means the repo rendered fine and it’s a real diff. Layer 2, argocd app diff: find the exact field that differs. If a field is present in live but not desired and I didn’t write it, a controller owns it. Layer 3, kubectl get <kind> <name> --show-managed-fields -o yaml: read the field-manager — kube-controller-manager/subresource: scale means an HPA. Then the highest fitting ladder rung: Server-Side Diff for webhooks/defaults, managedFieldsManagers for a whole controller, a scoped jsonPointers for one field. Never selfHeal: false.

Q: What is the precise difference between OutOfSync and Unknown, and why does it change where you look? A: OutOfSync means Argo CD ran the comparison and found a difference — so the repo-server successfully cloned and rendered, and the fault is a real diff (drift or a field owner). Unknown means Argo CD couldn’t run the comparison — a ComparisonError — so the repo-server failed to clone/render/reach the source. OutOfSync sends you to argocd app diff; Unknown sends you to the ComparisonError string and the repo-server logs. They’re opposite halves of the system.

Q: An app is Synced but Degraded. Whose problem is it and where do you look? A: It’s your problem, not Argo CD’s. Synced proves Argo CD applied exactly what Git contains; Degraded means a live resource failed its own health check. Open the resource tree in argocd app get, find the Degraded child, read its MESSAGE, then kubectl describe/logs it. Common causes: a typo’d image tag (ImagePullBackOff), a crash loop, a failing readiness probe, or ProgressDeadlineExceeded. You never touch the repo-server or controller — health problems don’t live there.

Q: You see ComparisonError: failed to generate manifests in <path>: app path does not exist. What failed, and what’s the fix? A: The repo-server cloned the repo but couldn’t find the directory named in spec.source.path at targetRevision — the path is wrong, or the repo layout changed at that revision. Confirm in the repo-server logs (GenerateManifest), fix the path or targetRevision, and hard-refresh. It has nothing to do with the target cluster or your pods.

Q: Three apps go Unknown simultaneously. How does that change your hypothesis versus one app going Unknown? A: One app Unknown is almost always that app’s source — a bad path, revision, chart, or values. Many apps Unknown at once points at shared infrastructure: the repo-server (OOMKilled, crashing) or the Git host (outage), or a credential shared across repos that just expired. So one-app-Unknown → check that app’s config; all-apps-Unknown → kubectl -n argocd get pod for repo-server restarts and check the Git host. Same state, completely different root cause and fix.

Q: What does --hard-refresh do that a normal --refresh doesn’t, and when do you need it? A: --refresh re-compares against the latest Git but may reuse the cached render; --hard-refresh invalidates the manifest cache and re-renders from scratch. You need --hard-refresh when you changed something the render depends on — a Helm chart, a plugin, or argocd-cm (a health check or ignoreDifferences rule) — and the app still shows the pre-change diff. It’s also the first thing to try when a state looks impossible, because that’s usually a stale cache.

Q: An app was healthy yesterday; today the controller can’t reach its cluster with dial tcp <ip>:443: i/o timeout. Is that auth or network, and how do you know? A: Network. i/o timeout means there’s no path to the API server — commonly a private control plane (private AKS/EKS/GKE) that lost its route, or a firewall/peering change. An auth failure would say Unauthorized/Unauthenticated or “asked for credentials.” The error code discriminates: timeout → open the network path (peering, VPN, authorized networks); unauthorized → refresh the cluster credential or fix the IAM mapping.

Q: Why is selfHeal: false the wrong response to a flapping app, and what’s the right one? A: The flap means the diff is wrong about which fields Argo CD owns — a controller (usually an HPA) rewrites a field every reconcile, and self-heal reverts it, so they war. selfHeal: false doesn’t make the app Synced (it’s still OutOfSync) and it disables drift correction for every field, so real prod drift now goes uncorrected and unnoticed. The right fix scopes the diff: ServerSideDiff for webhooks/defaults, managedFieldsManagers: [kube-controller-manager] for the HPA, or a scoped jsonPointers — leaving self-heal protecting everything else.

Q: A resource shows blank health and the app reads Healthy, but you’re not sure the resource works. What’s happening? A: Argo CD has no health check for that resource kind (a custom CRD), so it forms no opinion and excludes the resource from the app’s health rollup — the app looks Healthy because the resource is invisible to the calculation, not because it’s confirmed working. Add a Lua health check under resource.customizations.health.<group>_<kind> in argocd-cm, and test it offline with argocd admin settings resource-overrides health. If the Lua itself errors, that resource goes health Unknown — a different failure from a genuine Degraded.

Q: Health status Unknown versus sync status Unknown — same thing? A: No, and conflating them wastes time. Sync Unknown is a ComparisonError — the repo-server couldn’t render/compare, so look at the source and repo-server. Health Unknown means a per-kind health check ran and errored — almost always a custom Lua script indexing a nil field — so look at that health check in argocd-cm, not the repo. Same word, opposite components.

Q (scenario): A junior engineer says “Argo CD broke our deploy, it’s showing red, I’m going to reinstall it.” The app is Synced/Degraded. How do you coach them? A: Reframe the two axes. Synced proves Argo CD did its job — the cluster matches Git — so it didn’t “break” anything and reinstalling it changes nothing. The red is on the health axis: a resource is failing because of what’s in Git. Walk them to the resource tree in argocd app get, read the failing child’s MESSAGE, then kubectl logs/describe. The fix is in their manifest or image; Argo CD is correctly reporting reality. Reinstalling the controller is the reflex the method exists to prevent.

Q (scenario): You’ve been paged for a Degraded app at 2am. What are the first two commands, and what would make you escalate to reading Argo CD component logs? A: argocd app get <app> to find the Degraded child and its MESSAGE, then kubectl describe/logs that resource for the Kubernetes-level reason. For a Degraded app I would not read Argo CD component logs at all — health failures are the workload’s, and the repo-server/controller are innocent. I’d only descend to kubectl -n argocd logs if the symptom were Unknown (repo-server) or a stuck/non-reconciling app (controller). Reading component logs for a Degraded app is the classic wrong turn.

---

Practice challenges

Six graded exercises, beginner to advanced. They need no live cluster — troubleshooting is a reading skill, so each hands you the representative output and asks for the axis, the confirming command, the component, and the fix. Several build on the edge cases from Going deeper. Work each before opening the solution.

Challenge 1 (Beginner) — Is OutOfSync an outage? A teammate pages the channel: “shop-web is OutOfSync / Healthy — the app is down, roll it back!” Are they right, and what’s your first command?

<details> <summary>Solution</summary>

No. OutOfSync only means live ≠ rendered Git; Healthy means every resource passed its own health check — the app is running and serving traffic normally. Nothing is “down.” First command: argocd app diff shop-web to see what differs, then decide whether to sync (adopt Git) or revert (if the drift was wanted). Why: sync and health are independent axes — a perfectly healthy app is routinely OutOfSync in the gap between a commit and its sync. </details>

Challenge 2 (Beginner) — The UI is down; read the truth anyway. You’re paged that payments-api “looks red,” but the Argo CD UI and argocd CLI both time out — argocd-server is down. How do you read the app’s real sync and health?

<details> <summary>Solution</summary>

Read it straight off the custom resource, which the controller writes independently of the API:

kubectl -n argocd get application payments-api \
  -o jsonpath='{.status.sync.status} / {.status.health.status}{"\n"}'
kubectl -n argocd get application payments-api -o yaml | yq '.status.conditions'

Why: argocd-server only renders state; the application-controller stores it on .status, so kubectl sees the ground truth even when the API/UI is dark. </details>

Challenge 3 (Intermediate) — Unknown that isn’t the repo-server. checkout is Sync Status: Unknown, but the condition reads InvalidSpecError: application destination … namespace prod do not match any of the allowed destinations in project 'team-a'. You confirm the same repo renders fine for another app. Which component, and where’s the fix?

<details> <summary>Solution</summary>

Not the repo-server — the application-controller rejected the spec during validation, before any render. The fix is the AppProject team-a (its destinations/sourceRepos) or the app’s destination, not the source. Confirm by the condition type: InvalidSpecError, not ComparisonError. Why: Unknown has more than one cause — an invalid spec blocks the comparison just like a render failure does, but it sends you to the project, not the repo-server. </details>

Challenge 4 (Intermediate) — Ghost Unknown across the fleet. After onboarding ~200 apps that all point at one GitHub org, apps flip to Unknown intermittently and in groups with failed to list refs: 429 Too Many Requests, then clear, then return. Any single app tested alone is fine. Diagnose and fix.

<details> <summary>Solution</summary>

Git-provider rate limiting, not a per-app bug: hundreds of apps each running git ls-remote on the reconciliation timer exhaust the provider’s API limit. Fix: drive refreshes with a push webhook and raise timeout.reconciliation so polling drops, and use a higher-limit credential (a GitHub App token beats a PAT). Why: one app Unknown is that app’s source; many together is shared infrastructure — here the shared Git host’s throttle. </details>

Challenge 5 (Advanced) — A parent app-of-apps goes red. platform-apps (an app-of-apps whose source is a folder of Application YAMLs) reads OutOfSync/Degraded, but nobody changed that folder. Where do you look?

<details> <summary>Solution</summary>

Into the tree, not the parent’s manifests. A parent app-of-apps manages child Application resources, and Argo CD’s built-in health check for the Application kind rolls each child’s sync/health up into the parent — so the parent’s red is a child that went red. Find the unhealthy child Application in argocd app get platform-apps, then run the full four-layer method on that child (it has its own axes). Why: a parent’s verdict is a pointer to a child, not a diagnosis of the parent — fixing the parent’s YAML changes nothing. </details>

Challenge 6 (Advanced) — A flap that Server-Side Diff won’t fix. web and web-canary both keep going OutOfSync on the same ConfigMap; a SharedResourceWarning names both apps, and each sync makes the other drift. You try ServerSideDiff=true and managedFieldsManagers — no change. Why, and what’s the real fix?

<details> <summary>Solution</summary>

Two Applications own the same live resource and overwrite each other every reconcile — a governance problem, not a diffing one, so the diff-scoping ladder can’t touch it (both writers are Argo CD). The SharedResourceWarning condition is the tell. Fix: give the ConfigMap a single owner — remove it from one app, or split ownership cleanly along AppProject/namespace boundaries. Why: ServerSideDiff/managedFieldsManagers/ignoreDifferences scope one app’s diff; they can’t arbitrate two apps contending for one object. </details>

Common beginner mistakes

These are mental-model errors, not method slips — the wrong idea a beginner brings to the dashboard, and the model that replaces it. (The three method failures — trusting the label, reading the wrong component’s logs, and reaching for selfHeal: false — live in Common mistakes and troubleshooting above.)

“OutOfSync means the app is broken.” It doesn’t. OutOfSync is a diff — live differs from rendered Git — and says nothing about whether the app works. A healthy app serving production traffic is OutOfSync for the whole gap between a commit and its sync. Right model: read the health axis to learn whether it works; read the sync axis to learn whether it matches Git. Two separate questions.

“Synced means the app is working.” Also no. Synced proves only that live matches Git. Commit a typo’d image tag and you get Synced / Degraded — Argo CD faithfully applied a broken manifest. Right model: Synced is a statement about fidelity to Git, Healthy a statement about the resources working; you need both green.

“A red app means Argo CD failed / broke my deploy.” Argo CD is a reporter, not the culprit. Degraded means a resource failed its own health check; Synced/Degraded in particular proves Argo CD did exactly what Git said. Right model: Argo CD is the thermometer, not the fever — treat a red badge as evidence about your workload, and reinstalling the controller changes nothing.

“Unknown means Argo CD (or my cluster) is down.” Unknown is a narrow, specific failure to compare — usually the repo-server couldn’t render or reach the source, sometimes an invalid spec. Your running pods are untouched and the cluster is usually fine. Right model: Unknown points at the source/left side (Git, render, spec), never at your live workloads.

“When it looks weird, delete and recreate the Application to reset it.” There is no reset button. Recreating doesn’t fix a source or workload bug — and if the app has a finalizer and prune enabled, deletion can cascade-delete your live resources. Right model: diagnose first with the four-layer read; deleting an Application is a destructive action, not a refresh.

“The badge is the truth — if it’s red, something just changed.” Git and the live cluster are authoritative; the badge is a cached view that can lag reality. An impossible-looking state (OutOfSync on a field you reverted, a diff that contradicts kubectl get) is usually a stale Redis cache. Right model: argocd app get <app> --hard-refresh and re-read before you believe — or debug — a state that shouldn’t exist.

Glossary

• Sync status — the axis comparing live cluster state to the manifests rendered from Git: Synced, OutOfSync, or Unknown. Says nothing about whether the app works.
• Health status — the axis reporting whether each resource actually works: Healthy, Progressing, Degraded, Missing, Suspended, or Unknown. Independent of sync.
• Synced — live matches rendered Git. A statement of fidelity, not of function.
• OutOfSync — live differs from rendered Git; a real, computed diff exists. Not an outage.
• Unknown (sync) — Argo CD couldn’t run the comparison at all; recorded as a condition (usually ComparisonError, sometimes InvalidSpecError).
• Degraded — a resource failed its own per-kind health check (a crash loop, a failed probe, ProgressDeadlineExceeded). Your workload, not Argo CD.
• Progressing — a resource is converging toward ready, or stuck trying. Normal right after a sync.
• Missing — a resource declared in Git isn’t present live (a blocked apply: RBAC, admission, missing namespace, pending wave).
• Suspended — a resource is intentionally paused (a suspended CronJob, a paused Rollout).
• Unknown (health) — a health check ran and errored, almost always a custom Lua script indexing a nil field. Different from a resource being Degraded.
• Condition — an entry in an app’s Conditions: block describing a problem; the type is the routing key.
• ComparisonError — the condition behind a sync Unknown: the repo-server couldn’t clone, render, or reach the source.
• InvalidSpecError — a condition where the application-controller rejected the spec (bad AppProject destination/repo, unregistered cluster, missing project) before any render. Also shows Unknown — but the fix is the spec/project, not the repo-server.
• SharedResourceWarning — a condition warning that two Applications track the same live resource; they will fight over it.
• application-controller — the component that reconciles apps: computes the diff, runs health checks, applies syncs. Blame it for stuck reconciliation or spec rejection.
• repo-server — the component that clones repos and renders manifests (Helm/Kustomize/plugins). Blame it for Unknown/ComparisonError/render errors.
• argocd-server — the API/UI component. When it’s down, the CLI and dashboard go dark but reconciliation continues; read .status off the CR with kubectl.
• applicationset-controller — generates Application objects from an ApplicationSet. Blame it when an expected app never appears.
• Redis — Argo CD’s cache for rendered manifests and live state. Holds no source of truth, so flushing it is safe; stale entries cause impossible-looking states.
• Managed fields / field manager — Kubernetes’ record of which actor last wrote each field (--show-managed-fields). Names the controller behind perpetual OutOfSync.
• Server-Side Diff (SSD) — a diff computed by an API-server dry-run, so webhook- and default-injected fields match on both sides; the first rung of the drift fix ladder.
• ignoreDifferences / managedFieldsManagers — Application settings that scope which fields (or which field-owner) the diff ignores — the surgical and by-owner rungs of the ladder.
• selfHeal — the sync-policy option that auto-reverts drift. Never disable it to silence a flap; fix the diff instead.
• --refresh / --hard-refresh — re-compare against latest Git / re-render from scratch ignoring the manifest cache. --hard-refresh is the fix after editing a chart, plugin, or argocd-cm.
• app-of-apps — a pattern where a parent Application manages child Application resources; the parent’s status rolls up the children’s, so its red points one level down.
• operationState — .status.operationState on the CR: how the last sync operation finished (phase, message) — the kubectl-readable record of the most recent sync.

Key takeaways

• Method beats memorisation: read the state, read the events, read the component logs, isolate the layer — in that order. Most incidents are solved at layer 1 (argocd app get) or layer 2 (argocd app diff / the ComparisonError); you descend to component logs only when the layers above don’t explain it.
• Name the red axis first, because it names the possible causes. Sync OutOfSync = a real diff (drift or a field owner); sync Unknown = the repo-server couldn’t compare; health Degraded = your workload failed. Each lane has its own probe, component, and fix, and they never cross.
• Synced + Degraded is your bug, not Argo CD’s. Synced proves Argo CD applied exactly what Git said, so a broken result is in the commit — go straight to the resource tree and kubectl logs, never to the repo-server or controller.
• Unknown is a ComparisonError, and the string is the diagnosis. Bad path, bad revision, a Helm/Kustomize render error, repo auth, or an OOMKilled repo-server — one app Unknown is that app’s source; many apps Unknown at once is the repo-server or Git host.
• Perpetual OutOfSync means another controller owns a field you never set. Confirm the field with argocd app diff and the owner with --show-managed-fields, then climb the ladder — ServerSideDiff → managedFieldsManagers → ignoreDifferences — and never reach for selfHeal: false.
• The cloud edge is the target-cluster connection, and the error code discriminates it: i/o timeout is a network problem (a private control plane with no path — AKS/EKS/GKE alike), Unauthorized/Unauthenticated is a credentials problem (expired token, unmapped IAM identity).
• When a state looks impossible, suspect the cache before a bug. Redis holds no source of truth, so --hard-refresh (or restarting repo-server/Redis) is safe and resolves most “Argo CD is showing me something untrue” reports.
• Read the right component’s logs: repo-server for Unknown/render, application-controller for a stuck reconcile, argocd-server for UI/login, Dex for SSO — and for Degraded, not Argo CD at all. Sync-time failures (SyncFailed, hooks, pruning, stuck operations) are the domain of Part 2.