Argo CD Architecture: API Server, Repo Server, Application Controller, Redis & Dex

In a nutshell

Argo CD isn’t one program — it’s a small team of five specialist processes sharing one namespace, and this lesson introduces each teammate by name and job. Once you know who does what, Argo CD stops being a black box: every error message and every stuck app points straight at the one component responsible, so troubleshooting turns into a lookup instead of a guessing game.

A restaurant kitchen is the picture to hold. Git is the recipe book — what the food should be. The repo-server is the prep cook who reads a recipe and turns it into ready-to-use ingredients; it “renders” your Helm or Kustomize into plain Kubernetes YAML. The application-controller is the expediter at the pass, endlessly comparing the tickets (Git) against the plates actually going out (your live cluster) and calling for a fix on any mismatch. Redis is the counter where prepped ingredients sit so nobody re-chops the same onion every order. The API server is the host out front who takes your order and shows you the bill — the UI and the CLI. And Dex is the ID-checker at the door (optional single sign-on). Learn these five and the rest of the course falls into place.

Level: Beginner · Time: ~39 min Prerequisites: comfort with basic kubectl and the pull-based GitOps model from GitOps From First Principles. After this you can: name all seven Argo CD workloads and the five core ones; say exactly what the repo-server, controller, Redis, API server, and Dex each do; trace one argocd app sync from your laptop to the cluster and back; and map any symptom to the pod whose logs you should open.

---

Most people meet Argo CD as a web page: a graph of green hexagons, a big blue SYNC button, an app that says Synced and Healthy. That UI is honest, but it hides the machine. Behind the graph sit five cooperating processes, each with one job, talking to each other over gRPC and to a Redis cache in the middle. Learn those five and Argo CD stops being magic. You will know why an app is stuck, which pod’s logs to open, and what a given error message is actually complaining about — before you have read a single line of the docs.

This lesson opens the box. We will name every component, say exactly what each does, and follow one change all the way through: from argocd app sync on your laptop, through the repo-server turning your Git repo into Kubernetes YAML, into the application-controller’s reconcile loop where desired state meets live state, and out to the cluster. Then we will do the thing that makes this knowledge pay: build the debugging map — the table that tells you, for any symptom, which component is to blame.

Everything here is cloud-neutral. Argo CD’s internals are byte-for-byte identical on AKS, EKS, GKE, a laptop kind cluster, or bare metal — the same seven pods, the same reconcile loop. Only the edges differ (how you expose the UI to the internet, which identity provider Dex federates to), and where those edges appear we cover all three clouds in one line each. The engine is universal.

---

Why this matters

Here is the trap. You install Argo CD, point it at a repo, and for a week everything is green. Then one morning an app is OutOfSync and won’t move, or the UI throws ComparisonError, or a login that worked yesterday returns rpc error: code = Unauthenticated. If Argo CD is a single black box in your head, you are now stuck — you restart pods at random and hope. If instead you hold the component model, the same symptoms name their own culprit: ComparisonError is the repo-server failing to render; a stuck sync is the application-controller; a broken login is the API server or Dex. The map turns panic into a two-minute diagnosis.

The second reason is that every later lesson assumes this picture. When we talk about sync waves, we mean the order the controller applies resources. When we talk about private repos, we mean credentials the repo-server uses to clone. When we talk about SSO, we mean the API server delegating to Dex. When we talk about scaling to hundreds of clusters, we mean sharding the controller. Without the architecture, those lessons are vocabulary you memorise; with it, they are obvious consequences of who-does-what.

The mental model to anchor everything: Argo CD is a control loop, not a pipeline. A CI pipeline runs once, top to bottom, and stops. Argo CD’s application-controller runs forever, in a loop, continuously asking one question — “does the cluster match Git?” — and closing the gap when the answer is no. Git is the desired state; the cluster is the actual state; the controller is the thermostat that drives one toward the other. Hold that and the rest is detail. (If the pull-based, continuously-reconciled model itself is new, the sibling lesson GitOps From First Principles is the place to start; this lesson assumes you accept the model and asks how it is built.)

---

The component map at a glance

Argo CD is not one program. A default install drops seven workloads into one namespace, of which five are the core you must know and two are important-but-optional controllers. Every one is an ordinary Kubernetes Deployment or StatefulSet — there is no hidden daemon, no host agent. Here is the whole cast in one table; the rest of the lesson is just this table, expanded.

Three facts to notice immediately, because they drive everything below:

1. Only the application-controller is a StatefulSet. The rest are stateless Deployments. That single design choice is why its pod is named argocd-application-controller-0 (a StatefulSet ordinal) and why “scaling the controller” means sharding across ordinals -0, -1, -2 — each owning a slice of your clusters — rather than plain replicas. (Sharding is a Tier-4 scaling topic; just file the shape away.)
2. Redis sits in the middle. The server, repo-server, and controller do not cache in their own memory; they share Redis. That is why Redis being unhealthy degrades all three at once, and why — because it is only a cache — losing it costs performance, never data.
3. The two extra controllers are genuinely separable. ApplicationSet and Notifications are their own binaries with their own jobs; you can uninstall either and core sync still works. We forward-reference them here and give each its own lesson later.

Now, each component in turn.

---

The API server: the front door

The argocd-server process is the only component you talk to directly. The web UI is served by it; the argocd CLI is a gRPC/REST client of it; every automation token authenticates against it. Think of it as the front door and the reception desk combined — it does not do the deployment work itself, but nothing reaches the work without passing through it first.

It handles a specific, bounded set of responsibilities:

One protocol detail worth internalising because it explains a whole class of connection errors: argocd-server multiplexes gRPC and HTTP on the same port (container port 8080, exposed by the argocd-server Service as 80/443). The CLI speaks gRPC; the browser speaks HTTP; both hit the same endpoint. When you see the CLI hang or throw a TLS/gRPC handshake error, it is almost always about how you reached this port (plain vs TLS, a proxy stripping gRPC, --grpc-web needed behind an L7 load balancer) — not about the controller or your manifests.

Because the API server is the internet-facing piece, exposing it is the one genuinely cloud-specific edge in this lesson. Out of the box its Service is ClusterIP (reachable only inside the cluster); to reach the UI from your laptop you either kubectl port-forward, or put a cloud load balancer / ingress in front. The mechanics differ per cloud:

The install lesson walks through each of these end to end (including HA and first login); here the point is only that the API server is the component that owns “can I reach the UI?” If login or the UI is the problem, this pod — or the load balancer in front of it — is where you look. If apps are the problem, the API server is almost never the cause, because — and this is the fact that surprises people — you can delete the API server entirely and your apps keep reconciling. The controller does not need it.

---

The repository server: turning Git into manifests

The argocd-repo-server has the most underestimated job in the system: it is the one that clones your Git repositories and turns whatever is in them into plain Kubernetes manifests. Argo CD does not deploy “your repo”; it deploys the rendered output of your repo. The repo-server is the renderer.

Give it a source (a repo URL, a revision, a path) and it:

1. Clones (or reuses a cached clone of) the repo at the requested targetRevision.
2. Looks at the path and decides which tool to run — plain YAML, Kustomize, Helm, or a config-management plugin.
3. Runs that tool to produce a flat stream of Kubernetes manifests.
4. Returns those manifests to the caller (the controller during reconcile, or the API server for the UI diff), and caches them in Redis keyed by repo + revision + parameters.

The tool it picks is automatic, based on what the path contains:

Two things here routinely bite newcomers, so say them plainly. First, Helm under Argo CD is helm template, not helm install. The repo-server renders the chart to YAML locally and hands the YAML to the controller; Tiller does not exist, and there is no Helm release stored in the cluster. Your “release” is the Application. That is why helm list shows nothing and why Helm hooks behave differently — the manifests are just applied by Argo CD like any other. Second, the repo-server is where the words “render” and “generate” always point. Any error that mentions manifest generation, a kustomize build failure, a missing Helm value, or ComparisonError originates here, because rendering is exclusively this component’s job.

Caching is the repo-server’s other half, and it is why big installs stay fast:

The cache is also the source of a classic gotcha: after a Git change the repo-server may briefly serve a cached render, so a --hard-refresh (which bypasses the manifest cache and forces a fresh helm template/kustomize build) is the fix when an app looks stale even though you know Git moved. And because the repo-server runs your tooling with your credentials, it is also the component that needs repo access — SSH keys, HTTPS tokens, Helm/OCI registry creds all live as Secrets the repo-server consumes (covered in the repositories lesson).

---

The application controller: the reconcile loop

This is the heart of Argo CD, so we go slowly. The argocd-application-controller is the process that actually makes your cluster match Git. It is a control loop: for every Application, over and over, it compares desired state (the manifests the repo-server renders from Git) against live state (what is actually running in the target cluster), decides whether they match, judges whether the running app is healthy, and — if you have asked it to — applies the difference. Everything the UI shows you (Synced/OutOfSync, Healthy/Degraded) is this loop reporting its findings.

One full pass of the loop, for one Application, looks like this:

Steps 4 and 5 produce two independent answers, and conflating them is the single most common beginner confusion. Sync status answers “does live match Git?” Health status answers “is the running thing actually working?” They are orthogonal — an app can be any combination:

(The full state machine — including Progressing, Missing, and Unknown — is the whole subject of the sibling lesson Sync Status & Health Assessment. Here the load-bearing idea is just that the controller computes both, separately.)

What actually triggers a reconcile

The loop does not run continuously at full tilt; it is triggered. Knowing the triggers is what lets you answer “why is Argo CD slow to notice my commit?”

The default 3-minute poll is the number to remember. Out of the box, a commit to Git can take up to three minutes to be noticed, because the controller only re-examines each app every timeout.reconciliation. That interval lives in the argocd-cm ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  # How often the controller re-checks every app even with no event. Default 180s.
  timeout.reconciliation: 180s

You could lower it, but the right fix for “GitOps feels slow” is webhooks, not a tighter poll. A webhook makes a push near-instant while the 3-minute poll stays as a safety net for missed webhooks. Cranking the poll down to, say, 10s instead just multiplies load on the repo-server and every Git host you talk to — a self-inflicted rate-limit. Reach for a webhook first.

A subtle but important point about soft vs hard refresh: a normal (soft) refresh re-runs the diff using cached manifests — it re-reads live cluster state but trusts the repo-server’s cache for desired state. A hard refresh additionally throws away the manifest cache and forces the repo-server to re-clone and re-render. So when an app is stuck showing an old revision even though Git moved, a hard refresh — not a pod restart — is the targeted fix, because it invalidates exactly the cache that is stale.

---

Redis, Dex, and the two extra controllers

Redis — the cache in the middle

argocd-redis is a plain Redis instance, and its role is easy to state and easy to get wrong: it is an ephemeral cache, never a database. The server, repo-server, and controller all read and write it instead of caching in their own memory, which is what lets them stay fast and (in HA) stay stateless.

The crucial property: because it is only a cache, Redis is safe to lose. The source of truth for desired state is Git; the source of truth for live state is the cluster itself. If Redis is wiped or restarted, Argo CD simply rebuilds the cache — it re-renders manifests and re-lists cluster resources. The cost is performance, not correctness: while the cache is cold, reconciles are slower and the UI may momentarily show stale or errored panels. If Redis is fully down (not just wiped), you will see degraded behaviour across the UI and slow or failing reconciles, with cache-connection errors in the logs — but the instant Redis is back, everything recovers with no data lost. This is exactly why production installs run argocd-redis-ha (Redis with Sentinel) — not to protect data, but to keep the cache continuously available so performance never craters.

Dex — optional SSO federation

argocd-dex-server is an embedded Dex instance: a small OIDC provider that Argo CD bundles so you can wire up single sign-on without standing up your own identity broker. It is optional in two senses — you can run Argo CD with just the local admin user and no Dex at all, and even for SSO you can bypass Dex by pointing Argo CD at an OIDC issuer directly. Dex exists for the cases where your identity provider isn’t a clean OIDC issuer, or you want SAML/LDAP/GitHub-org federation normalised into OIDC for Argo CD.

The flow is: the API server delegates login to Dex; Dex federates to your real IdP; the token comes back and the API server enforces RBAC on the groups inside it. Which IdP you federate to is a cloud/identity edge:

Dex’s config lives under dex.config in argocd-cm, with client secrets in argocd-secret. The one thing to know here is the blast radius: misconfigure Dex and only SSO login breaks — the local admin login and already-running apps are untouched. The full setup (per cloud) is the subject of the SSO lesson; for architecture, Dex is “the optional auth sidecar the API server talks to.”

ApplicationSet and Notifications controllers

Two more controllers ship by default and each earns its own lesson later, but you should recognise them in get pods:

Both are strictly additive: core sync works without either. The ApplicationSet controller is how you go from hand-writing one Application per app-per-cluster to generating hundreds — the scaling story the App-of-Apps and ApplicationSets lessons pick up in Tier 2.

---

How a change flows end to end

Now assemble the pieces into one motion. Say you run argocd app sync guestbook from your laptop. Read the diagram left to right, then the walkthrough — the numbered badges mark exactly where each component’s job (and its failure mode) lives.

The path, step by step:

1. CLI → API server. The argocd CLI opens a gRPC call to argocd-server. The server validates your token (local, API-key, or SSO via Dex) and checks RBAC: may you sync this app? If login fails, you never leave this box — the error is Unauthenticated, and the culprit is the API server or Dex, never the controller.
2. API server records intent. The server writes a sync operation onto the Application resource. It does not deploy anything itself — it just records that a sync was requested. (This is why declarative and imperative management converge: the CLI and a Git edit both end up mutating the same CRD.)
3. Controller wakes. The application-controller, watching its Application CRDs, sees the requested operation and begins a reconcile.
4. Controller → repo-server. It asks the repo-server for the desired manifests. The repo-server clones the repo at targetRevision, runs kustomize build/helm template/plain YAML, and returns rendered manifests — caching them in Redis. A broken chart or kustomization surfaces here as ComparisonError.
5. Controller diffs and applies. It compares desired against the live cluster state (kept warm in its Redis-backed cache), computes the diff, then applies the changes to the target cluster in sync-wave order, running any hooks.
6. Cluster → status → you. The controller watches the applied objects, assesses health, writes Synced/Healthy back to the Application status, and the API server serves that back to your CLI and the UI.

Notice what depends on what. The API server is only needed at the ends (to accept your request and to show you the result); the actual reconcile — steps 3 through 6 — runs whether or not any human is watching. That is the control-loop nature made concrete: kill the UI and your apps still self-heal; kill the controller and everything freezes.

---

Namespaces, CRDs, and the pods you’ll see

One namespace, by convention

By default the whole control plane lives in the argocd namespace. Nothing forces that name — you can install elsewhere — but argocd is the universal convention, and every command in this course uses -n argocd. The workloads there are the seven above plus their Services, ConfigMaps (argocd-cm, argocd-cmd-params-cm, argocd-rbac-cm), and Secrets (argocd-secret, plus repo/cluster Secrets).

Where your apps run is a separate question. Argo CD can deploy into the same cluster it runs in, into other namespaces, or into entirely different clusters — the Application’s destination decides. The control plane and the workloads it manages are decoupled.

Three CRDs are the whole API surface

Argo CD extends Kubernetes with a small set of Custom Resource Definitions, all in the argoproj.io/v1alpha1 API group. These are Argo CD’s declarative interface — the UI and CLI are just ergonomic ways to edit them:

A minimal Application is worth seeing now, because it is the object every other lesson manipulates and the thing the controller reconciles:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: guestbook
  namespace: argocd                     # the Application lives in the control-plane namespace
spec:
  project: default
  source:
    repoURL: https://github.com/argoproj/argocd-example-apps.git
    targetRevision: HEAD
    path: guestbook                     # the repo-server renders whatever is here
  destination:
    server: https://kubernetes.default.svc   # deploy into THIS cluster
    namespace: guestbook
  syncPolicy:
    automated:                          # let the controller sync without a human click
      prune: true
      selfHeal: true

You do not need to understand every field yet — the first-Application lesson is entirely about this manifest. What matters architecturally is the mapping: source is the repo-server’s input, destination is the controller’s target, syncPolicy tells the controller whether to act automatically, and the whole object is what the API server writes and the controller reads.

The pods you’ll actually see

On a fresh non-HA install, kubectl -n argocd get pods shows the cast in the flesh (representative output — names have random suffixes, and an HA install adds Redis-HA and multiple server/repo replicas):

kubectl -n argocd get pods
# NAME                                                READY   STATUS    RESTARTS   AGE
# argocd-application-controller-0                     1/1     Running   0          6m
# argocd-applicationset-controller-6d8f4b5c9d-abcde   1/1     Running   0          6m
# argocd-dex-server-7b9c8f6d5b-fghij                  1/1     Running   0          6m
# argocd-notifications-controller-5c7d9f8b6c-klmno    1/1     Running   0          6m
# argocd-redis-6fd9c8b7f5-pqrst                       1/1     Running   0          6m
# argocd-repo-server-84b7d9c6f8-uvwxy                 1/1     Running   0          6m
# argocd-server-6c9f7d8b5c-z1234                      1/1     Running   0          6m

The -0 on the controller (versus the ReplicaSet-hash suffixes on the others) is the StatefulSet tell from earlier. And the matching Services expose the internal ports the components use to talk to each other:

kubectl -n argocd get svc
# NAME                               TYPE        PORT(S)                     (what it is)
# argocd-server                      ClusterIP   80/TCP,443/TCP              UI + API (→ 8080)
# argocd-repo-server                 ClusterIP   8081/TCP,8084/TCP           render gRPC + metrics
# argocd-redis                       ClusterIP   6379/TCP                    the cache
# argocd-dex-server                  ClusterIP   5556/TCP,5557/TCP,5558/TCP  OIDC http/grpc/metrics
# argocd-applicationset-controller   ClusterIP   7000/TCP,8080/TCP           webhook + metrics
# argocd-metrics                     ClusterIP   8082/TCP                    controller metrics
# argocd-server-metrics              ClusterIP   8083/TCP                    server metrics

Two reading tips: the application-controller has no serving Service other than argocd-metrics (it is a controller, not a server — it dials out to clusters and the repo-server, nothing dials in), and argocd-server is the only one whose Service you would ever expose beyond the cluster.

---

The debugging map: which component owns which symptom

This is the table the whole lesson exists to give you. When Argo CD misbehaves, do not guess — read the symptom, find the component, open that pod’s logs. Memorise the shape of it and you have turned every future incident into a lookup.

The pattern to internalise: rendering problems are the repo-server, sync/status problems are the controller, login/UI problems are the API server, “slow/stale” is Redis, and “SSO only” is Dex. Five rules cover the vast majority of real incidents.

---

Hands-on lab

This lab is pure exploration — you will not change anything, just see the architecture running and practise the mapping. It assumes you already have Argo CD installed (the install lesson in this tier covers that on AKS, EKS, GKE, or a free local cluster); if you don’t yet, do that first — a kind or minikube cluster is perfect and none of this bills. Everything below is read-only kubectl/argocd.

Step 1 — Meet the whole cast.

kubectl -n argocd get pods,svc
# You should see the 7 workloads and their Services from the section above.

What just happened: you are looking at the entire Argo CD control plane. Find each of the five core components by name. Confirm the controller ends in -0 (StatefulSet) while the rest carry ReplicaSet hashes (Deployments).

Step 2 — Confirm the CRDs are installed.

kubectl get crd | grep argoproj.io
# applications.argoproj.io
# applicationsets.argoproj.io
# appprojects.argoproj.io

What just happened: these three CRDs are Argo CD’s entire declarative API. If this command comes back empty, your install is incomplete — that is the “kind not found” row in the debug map, seen from the other side.

Step 3 — Watch the application-controller think.

kubectl -n argocd logs statefulset/argocd-application-controller --tail=40 -f
# time="..." level=info msg="Refreshing app status (controller refresh requested)" application=argocd/guestbook
# time="..." level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: guestbook)" application=argocd/guestbook
# time="..." level=info msg="Reconciliation completed" application=argocd/guestbook time_ms=137

What just happened (representative logs): you are watching the reconcile loop from the last section, live. Each Comparing app state line is step 4 (the diff); Reconciliation completed closes one pass. Leave this streaming and move to Step 4.

Step 4 — Trigger a reconcile and watch it land. In a second terminal:

argocd app get guestbook --refresh    # force a soft refresh now, don't wait for the 3-min poll

What just happened: the --refresh flag made the controller reconcile immediately instead of at the next timeout.reconciliation tick. Flip back to your Step 3 terminal — you should see a fresh Refreshing app status → Comparing app state → Reconciliation completed burst appear the moment you ran it. You just caused a trip around the loop and observed it in the logs. That connection — command in one terminal, loop reaction in the other — is the whole mental model made tangible.

Step 5 — Watch the repo-server render.

kubectl -n argocd logs deploy/argocd-repo-server --tail=30
# time="..." level=info msg="manifest cache miss" ...
# time="..." level=info msg="Generating manifests with no cache" ...

What just happened (representative): a cache miss followed by Generating manifests is the repo-server doing its one job — running your kustomize/helm/plain-YAML render. A cache hit on a repeat means Redis served the previous render. This is exactly where ComparisonError would show up if a chart were broken.

Step 6 — Prove the API server is separable (optional, conceptual).

# Read-only observation: note the controller reconciles on its own timer.
kubectl -n argocd logs statefulset/argocd-application-controller --tail=5
# Even with nobody logged into the UI, reconciles keep happening.

What just happened: you confirmed the claim from earlier — the reconcile loop runs without any client attached. The API server is the door you use, not the engine that does the work.

Step 7 — Map a symptom yourself. Without scrolling up, answer: an app shows ComparisonError. Which pod’s logs do you open? (Answer: argocd-repo-server — rendering is its job.) Now: argocd login returns Unauthenticated. Which pod? (argocd-server, and argocd-dex-server if you use SSO.) If both came instantly, you have the map.

Teardown: nothing to tear down — this lab created no resources. If you spun up a local cluster only for this, delete it when done (kind delete cluster / minikube delete) so it isn’t left running.

---

Common mistakes and troubleshooting

The debugging map above is your symptom→component index. This section adds the causes and fixes for the mistakes beginners actually make, plus prose on the three nastiest.

Three gotchas cost the most hours, so spell them out:

1. Blaming the cluster for a render error. ComparisonError feels like a deployment failure, so people go poke at their pods and Deployments. But ComparisonError fires in step 4 of the reconcile — the controller could not even get the desired manifests, because the repo-server failed to render them. Nothing was ever applied to the cluster. The fix is always in the repo-server’s world: a bad Helm value, a kustomization.yaml that doesn’t build, a path pointing at the wrong directory. Reproduce it by running helm template or kustomize build on that path yourself; the error will be identical, and you never touched the cluster.

2. Fighting the 3-minute poll the wrong way. When “GitOps feels slow,” the instinct is to shrink timeout.reconciliation. That is a trap: a low poll multiplies load on the repo-server and hammers your Git host (GitHub/GitLab will rate-limit you), and it still isn’t instant. The correct fix is a webhook, which makes a push near-instant while the 3-minute poll remains a cheap safety net for the rare missed webhook. Latency is a trigger problem, not a poll-interval problem.

3. Treating Redis like it holds your data. Because “database-shaped things” usually hold state, people panic when Redis restarts and assume apps are lost. They are not. Redis is only a cache; desired state is in Git and live state is in the cluster. A Redis blip degrades performance and can throw transient UI errors, and then everything recovers on its own when the cache warms back up. Run argocd-redis-ha in production not to protect data but to keep the cache continuously available so you never even notice the blip.

---

Going deeper

You now have the five components and the reconcile loop. This section opens the two hot spots that decide whether Argo CD stays fast at scale — the repo-server and the controller — plus the one internal detail (how a live object is tied back to its Application) that the loop quietly depends on and most people never see. None of it is needed for a first install; all of it is what separates “it works on my kind cluster” from “it holds 300 apps across 40 clusters.”

Inside the repo-server: the CPU/memory hot spot you scale first

Rendering is not free. When the repo-server picks a tool (step 2 of the render), it forks a real subprocess — helm template, kustomize build, a Jsonnet VM, or your plugin — and those subprocesses are CPU- and memory-hungry. On a busy instance (a big monorepo, or a refresh storm after many commits) the repo-server is the component that saturates first. Watch its pod under load and you will see CPU spike while everything else idles.

Two knobs bound that cost:

• Concurrency cap. --parallelismlimit (param reposerver.parallelism.limit in the argocd-cmd-params-cm ConfigMap; 0 means unlimited) caps how many manifest generations run at once. Set it so a burst of refreshes queues instead of forking a hundred helm templates and OOM-killing the pod.
• Render cache TTL. The rendered-manifest cache in Redis expires on --repo-cache-expiration (param reposerver.repo.cache.expiration, default 24h). A --hard-refresh busts exactly this entry; a routine reconcile reuses it.

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cmd-params-cm       # component flags live here, not in argocd-cm
  namespace: argocd
data:
  reposerver.parallelism.limit: "10"    # --parallelismlimit: max concurrent manifest generations
  reposerver.repo.cache.expiration: "24h"

The important structural fact: the repo-server is a stateless Deployment, so you scale it horizontally — just raise replicas. Requests fan out across the pods with no coordination needed. That makes “renders are slow / the repo-server is pegged” the easy scaling problem: add replicas and (if needed) raise the parallelism limit and CPU. Contrast that with the controller, which cannot be scaled by plain replicas at all. The dedicated Scaling & Sharding the Repo-Server lesson takes this all the way for monorepos.

Tuning the controller: processors, resync, and sharding

The controller’s throughput is governed by two worker pools, and knowing them explains “why is my reconcile queue backed up?”:

On an instance with hundreds of apps, raising both keeps the reconcile queue from lagging behind the timeout.reconciliation tick (the resync interval you already met; its legacy flag was --app-resync). But processors only help until you hit the real ceiling: memory. The controller keeps a live, in-memory cache of every object in every managed cluster so its diffs are fast — which is precisely why it is a StatefulSet and cannot be scaled with interchangeable replicas.

To grow past one pod’s memory you shard clusters across ordinals. Set the StatefulSet replicas to N and tell every pod the total with the ARGOCD_CONTROLLER_REPLICAS env var (each pod derives which shard it owns from its -0/-1/-2 ordinal). The mapping of clusters to shards is chosen by the sharding algorithm:

The nuance that bites in production: sharding balances cluster count, not object count. One giant cluster (say, a 5,000-Pod prod fleet) makes its shard hot and OOMKilled even while the other shards sit half-empty — that is the real story behind the “controller OOMing” row in the debugging map. The fix is more memory on that shard, fewer clusters per shard, or consistent-hashing plus manual pinning; the sharding lesson goes deeper.

How the controller ties a live object back to its app

Step 4 of the loop — the diff — has a hidden prerequisite: given a random live object in the cluster, Argo CD must know which Application owns it. That is the resource-tracking method, and it is worth understanding because getting it wrong causes apps to “adopt” each other’s resources or orphan their own.

• label (the default). Argo CD stamps app.kubernetes.io/instance: <app-name> on everything it applies (the key is configurable via application.instanceLabelKey in argocd-cm). Its weakness: a label value is capped at 63 characters, and the same value can collide across namespaces or clusters — so two apps named frontend in different namespaces can confuse ownership.
• annotation (sturdier). Set application.resourceTrackingMethod: annotation and Argo CD tracks via the argocd.argoproj.io/tracking-id annotation instead. Annotations have no length limit and encode app name plus group/kind, so ownership is unambiguous. annotation+label keeps the label too, for external tooling that reads it.

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  application.resourceTrackingMethod: annotation   # label | annotation | annotation+label

Once ownership is known, the controller builds the resource tree you see in the UI: it starts from the objects it applied, then walks Kubernetes ownerReferences downward — Deployment → ReplicaSet → Pod, Rollout → ReplicaSet → Pod, and so on — which is how the graph shows Pods the manifests never mention. Anything the app manages but Git no longer declares is a prune candidate (with prune: true); anything in the namespace it does not own can be surfaced as an orphaned resource (an off-by-default AppProject hint). The comparison and drift mechanics that sit on top of this are the whole subject of the Diffing, Drift & ignoreDifferences lesson.

API server statelessness, RBAC, and a 3.0 caveat

The API server holds no application state of its own — everything lives in the CRDs and in Redis — which is why you can run several argocd-server replicas behind one load balancer and lose any of them freely. On every gRPC/REST call it evaluates the RBAC model in argocd-rbac-cm: the built-in role:admin and role:readonly, any custom roles, and the g, <group>, <role> bindings that map SSO groups to roles.

One Argo CD 3.0-GA caveat to file away: fine-grained RBAC on application sub-resources got stricter by default. A grant to update/delete an Application no longer silently extends to the resources that app manages — actions like viewing a managed Pod’s logs or deleting a managed resource from the UI now want their own explicit policy. If you upgraded from 2.x and some UI actions suddenly return permission-denied, this is usually why; the RBAC, Local Users & Policies lesson has the exact policy lines.

Finally, Dex is skippable when your IdP already speaks clean OIDC. Entra ID, Google, Okta, and Cognito user pools can be wired directly via oidc.config in argocd-cm, dropping the argocd-dex-server pod entirely — one fewer thing to run and patch. Keep Dex only when you need SAML, LDAP, or GitHub-org federation normalised into OIDC. The trade-off and per-cloud setup live in SSO with OIDC, Dex, Entra, Cognito & Google.

---

Common beginner mistakes

The troubleshooting table above maps symptoms to fixes. This section is different: it catches the mental models that quietly send beginners in the wrong direction long before an error appears. Each row is a belief to unlearn, paired with the model that replaces it.

The thread running through all six: separate the door from the engine, and separate “matches Git” from “actually works.” Almost every early confusion is one of those two distinctions collapsing. When something looks wrong, first ask which component’s job is this? (render → repo-server, reconcile/status → controller, login/UI → API server) and which question is failing? (sync = matches Git, health = actually works). Those two questions resolve the vast majority of “wait, why is it doing that?” moments before you ever open a log.

---

Practice challenges

Six exercises, easy to hard. They build on the component model, not on memorising commands. Most are read-only; where one mutates state, it says so and tells you how to undo it — run those only on a throwaway/lab cluster. Try each before opening its solution.

Challenge 1 (Beginner) — Spot the StatefulSet. From memory, in a default kubectl -n argocd get pods, which one core component is a StatefulSet rather than a Deployment, and how does the pod name reveal it?

<details> <summary>Solution</summary>

The application-controller. Its pod is argocd-application-controller-0 — the -0 is a StatefulSet ordinal, whereas the Deployments carry a ReplicaSet hash (e.g. argocd-repo-server-84b7d9c6f8-uvwxy).

Why: only the controller holds per-cluster state (its live-object cache), so it needs stable identity and is scaled by sharding, not plain replicas. </details>

Challenge 2 (Beginner) — Route a render error. An app shows ComparisonError with a helm template message in the detail. Which pod’s logs do you open, and what single local command reproduces the failure without touching the cluster?

<details> <summary>Solution</summary>

Open the repo-server: kubectl -n argocd logs deploy/argocd-repo-server. Reproduce locally by running the same render on the same path — helm template <release> <chart> -f values.yaml (or kustomize build <path> for Kustomize).

Why: rendering is exclusively the repo-server’s job; ComparisonError fires before anything is applied, so the cluster is never the culprit. </details>

Challenge 3 (Intermediate) — Make a push near-instant, the right way. Out of the box a commit can take up to three minutes to deploy. Name the correct fix and the tempting-but-wrong one, and say why the wrong one backfires.

<details> <summary>Solution</summary>

Right: add a Git provider webhook to POST your Argo CD’s /api/webhook, which triggers a refresh on push (near-instant), while timeout.reconciliation: 180s stays as a safety net for missed webhooks. Wrong: shrinking timeout.reconciliation to something like 10s.

Why: a tight poll multiplies repo-server load and hammers your Git host until it rate-limits you (403 secondary limit) — latency is a trigger problem, not a poll-interval one. </details>

Challenge 4 (Intermediate) — Prove the API server is separable. Design a reversible experiment (lab cluster) that shows apps keep reconciling with the UI gone.

<details> <summary>Solution</summary>

Scale the API server to zero and watch the loop continue:

kubectl -n argocd scale deploy/argocd-server --replicas=0
kubectl -n argocd logs statefulset/argocd-application-controller -f   # reconciles keep flowing
kubectl -n argocd get applications                                    # status still updates
kubectl -n argocd scale deploy/argocd-server --replicas=1             # restore (HA: original count)

Why: the controller reconciles independently of argocd-server — the API server is the door you use, not the engine that does the work. </details>

Challenge 5 (Advanced) — Tame an OOMing controller. A single argocd-application-controller-0 is OOMKilled running 300 apps across 40 clusters. Give a two-part fix — immediate and structural — and the caveat that can defeat it.

<details> <summary>Solution</summary>

Immediate: raise the controller’s memory request/limit (and --status-processors/--operation-processors if the reconcile queue is also backing up). Structural: shard — set the StatefulSet replicas: 3, ARGOCD_CONTROLLER_REPLICAS=3, and controller.sharding.algorithm: consistent-hashing so clusters spread across ordinals -0/-1/-2.

Why: the controller scales by sharding clusters, not by plain replicas — but sharding balances cluster count, so one huge cluster can still make its shard hot; that shard needs more memory or its own pinning. </details>

Challenge 6 (Advanced) — Fix ownership collisions. In a shared cluster, two teams run apps whose names collide across namespaces, and resources are being mis-adopted or orphaned. Which single argocd-cm setting fixes it, and why does it work?

<details> <summary>Solution</summary>

Switch the tracking method: set application.resourceTrackingMethod: annotation (or annotation+label) in argocd-cm. Argo CD then tracks ownership via the argocd.argoproj.io/tracking-id annotation instead of the app.kubernetes.io/instance label.

Why: the default label method caps values at 63 chars and collides on duplicate names; the annotation has no length limit and encodes app name plus group/kind, so ownership is unambiguous. </details>

---

Cheat-sheet

The commands and mappings you will reach for constantly.

Component → job → failure signature, in one glance:

Ports worth knowing: server 443/80→8080, repo-server 8081, Redis 6379, Dex 5556, controller metrics 8082.

---

Interview and exam questions

Q: Name the five core components of Argo CD and give each a one-line job. A: API server — serves the UI/CLI and handles auth, RBAC, and app management. Repository server — clones Git and renders manifests (kustomize/helm/plain/plugins). Application controller — runs the reconcile loop: diff desired vs live, assess health, sync. Redis — an ephemeral cache of rendered manifests and cluster state. Dex — an optional bundled OIDC provider for SSO federation.

Q: Which component renders your Helm chart, and does Argo CD run helm install? A: The repo-server, and no — it runs helm template to render the chart to plain manifests locally, then hands them to the controller to apply. There is no Tiller and no Helm release stored in the cluster; the Application is the release. That’s why helm list shows nothing.

Q: An app shows ComparisonError. Which component is at fault and why? A: The repo-server. ComparisonError means the controller couldn’t obtain the desired manifests because rendering failed — a broken chart/kustomization, a missing value, or a wrong path. Nothing was applied to the cluster; the fix is in the repo-server’s render, reproducible with helm template/kustomize build on that path.

Q: What is the reconcile loop, and what two independent answers does it produce? A: It’s the application-controller’s continuous cycle: get desired state (repo-server), get live state (cluster), diff, assess health, and — if automated — apply. It produces sync status (Synced/OutOfSync: does live match Git?) and health status (Healthy/Progressing/Degraded: is the app actually working?). They’re orthogonal — an app can be Synced but Degraded, or OutOfSync but Healthy.

Q: What is timeout.reconciliation, its default, and the right way to make syncs faster? A: It’s the interval in argocd-cm at which the controller re-checks every app even with no event; default 180s (3 minutes). The right fix for slow syncs is a Git webhook to /api/webhook (near-instant on push), keeping the poll as a safety net — not shrinking the poll, which just loads the repo-server and rate-limits your Git host.

Q: If Redis goes down, do you lose data? What breaks? A: No data loss — Redis is only a cache; desired state lives in Git, live state in the cluster. What breaks is performance: slow reconciles, stale or errored UI panels, cache-connection errors in the logs. When Redis returns, Argo CD rebuilds the cache and recovers. Prod runs argocd-redis-ha for cache availability, not data safety.

Q: You delete the argocd-server pod. Do your applications stop syncing? A: No. The application-controller reconciles independently of the API server. You lose the UI, the CLI, and SSO/webhook reception, but existing apps keep diffing and self-healing. Kill the controller, though, and all reconciliation freezes — that’s the component whose outage actually stops deployments.

Q: Why is the application-controller a StatefulSet while the others are Deployments, and what does that imply for scale? A: It holds per-cluster state (watched-resource caches) and is scaled by sharding — each ordinal replica (-0, -1, …) owns a subset of clusters — rather than by interchangeable replicas. That’s why its pod is ...-controller-0 and why “scale the controller” means configure sharding, not just bump replicas.

Q: What are Argo CD’s three CRDs and what does each declare? A: Application (one deployable unit: source → destination + sync policy), ApplicationSet (a generator + template that produces many Applications), and AppProject (a guardrail restricting which repos, clusters, namespaces, and kinds a group of apps may use). All are in argoproj.io/v1alpha1.

Q: Is Dex required? What breaks if it’s misconfigured? A: No — you can run with just the local admin user, or use a direct OIDC issuer without Dex. Dex is the bundled OIDC broker for SSO federation (Entra ID, Google, IAM Identity Center, GitHub org, LDAP…). Misconfigure it and only SSO login breaks; local and API-token auth, and all running apps, are unaffected.

Q (scenario): The UI is down but kubectl get applications -n argocd shows everything Synced/Healthy. What’s going on and is it urgent? A: The API server (or its load balancer/ingress) is down, but the controller is fine — apps are still reconciling, which is why the CRDs report healthy. It’s a visibility/access outage, not a delivery outage: fix argocd-server or the LB in front of it, but deployments are not at risk in the meantime.

Q (scenario): A teammate lowered timeout.reconciliation to 10s and now Git pushes sometimes fail with 403. Why? A: The tight poll makes the repo-server re-fetch/re-render far more often, hammering the Git host until it rate-limits (403 secondary rate limit on GitHub). Restore the default 180s and add a webhook for latency instead — you get near-instant syncs without the load.

---

Glossary

Every term this lesson uses, in one place and in plain language.

---

Key takeaways

• Argo CD is seven pods in one namespace, five of them core. API server (front door), repo-server (renders Git → manifests), application-controller (the reconcile loop), Redis (cache), Dex (optional SSO) — plus the ApplicationSet and Notifications controllers.
• It’s a control loop, not a pipeline. The controller runs forever, continuously diffing desired (Git, via the repo-server) against live (the cluster) and closing the gap. Sync status and health status are two separate answers it computes each pass.
• The repo-server owns rendering. Anything about helm template, kustomize build, or ComparisonError is this component — never the cluster. Helm is templated, not installed; there’s no Tiller and no release object.
• Redis is a cache, never a database. Losing it costs performance, not data — Git and the cluster remain the source of truth. argocd-redis-ha keeps the cache available, not “safe.”
• The debugging map is the payoff: rendering → repo-server, sync/status → controller, login/UI → API server, slow/stale → Redis, SSO-only → Dex. Read the symptom, open that pod’s logs.
• Latency is a trigger problem. The default 3-minute timeout.reconciliation poll is meant to be backstopped by a webhook, not shrunk. A tight poll just rate-limits your Git host.
• The API server is separable; the controller is not. Kill the UI and apps still reconcile; kill the controller and delivery stops. That asymmetry tells you where the real engine is.