Multi-Cluster Argo CD: Registering Clusters, Cluster Secrets & Targeting Deployments

In a nutshell

Picture your platform team as the front desk of a hotel chain’s head office. Head office doesn’t run each hotel itself — it keeps one keycard per property in a locked drawer, and every keycard is printed with the hotel’s street address and a way to prove you’re allowed through the door. To send an instruction to one hotel, the front desk pulls that hotel’s keycard by name. Opening a new property just means cutting a fresh keycard and dropping it in the drawer.

That drawer is Argo CD’s multi-cluster model. One Argo CD — the hub — manages many Kubernetes clusters (the spokes), and it remembers each one as a single Kubernetes Secret: the cluster Secret, holding the cluster’s API address (server), a human name (name), and how to prove who you are (config). Register a cluster and you add a keycard; target a deployment and you pick a keycard by name. That is the entire idea — everything else in this lesson is the fine print of how you prove who you are, which is genuinely different on Azure, AWS and Google Cloud.

If you are new to this, hold three ideas and the rest follows: a cluster is just a labelled Secret; the hub is a client of every spoke’s API server (its own cluster is only the client’s home); and “registered” is not the same as “reachable and allowed” — those are three separate things that fail three separate ways.

Level: Advanced · Time: ~47 min

Prerequisites: Argo CD installed and you can log in (Installing Argo CD), at least one repository connected (Connecting Repositories), and comfort with Kubernetes Secrets and RBAC. After this you can: register a spoke both imperatively and declaratively; read and hand-write a cluster Secret field by field; wire per-cloud exec auth for AKS, EKS and GKE; scope a cluster by namespace, project and RBAC; and target a deployment at one spoke by name.

---

Sooner or later a single cluster stops being the story. You have a dev cluster and a prod cluster; then a second region; then a compliance boundary that forces payments onto its own cluster; then someone acquires a company running on a different cloud entirely. The question that decides your next two years of operations is deceptively small: do you run one Argo CD per cluster, or one Argo CD that reaches them all?

This lesson is about the second answer — the hub-and-spoke model — and the single mechanism that makes it work: the cluster Secret. Everything Argo CD knows about a remote cluster (where its API server lives, and how to prove who you are to it) is stored in one Kubernetes Secret in the hub’s argocd namespace. Learn the shape of that Secret and you can register any cluster, on any cloud, by hand or with one command — and, crucially, debug the ones that refuse to connect.

The hard part is never the Secret’s YAML. It is authentication: how does an Argo CD pod running in one cluster prove its identity to a different cluster’s API server, when that cluster is guarded by Entra ID, or AWS IAM, or Google Cloud IAM? That edge is different on AKS, EKS and GKE, and it is where most real multi-cluster setups break. So we cover all three, with real execProviderConfig blocks for each. By the end you will register a remote cluster two ways, wire up per-cloud auth, and deploy an application to a spoke by name.

This lesson assumes you have Argo CD installed and can log in — if not, start with Installing Argo CD — and that you have connected at least one Git repository, covered in Connecting Repositories.

---

Why this matters

A cluster you cannot deploy to is just expensive idle compute. The whole promise of GitOps — the state in Git is the state in the cluster — only scales if one control plane can hold that promise across a fleet. The alternative, an Argo CD instance babysitting every cluster, means N upgrade cycles, N sets of credentials, N dashboards, and no single place to answer “is checkout at version 4.2 everywhere?”

Multi-cluster is also where the friendly single-cluster mental model quietly breaks. On one cluster, Argo CD talks to its own API server over https://kubernetes.default.svc using the ServiceAccount token mounted into its pods — you never think about authentication, because Kubernetes hands it to you for free. The moment the target is a different cluster, that free ride ends. The hub’s pods have no automatic identity on the spoke. You have to establish one, store it, keep it fresh, and make sure the network even allows the connection. Four separate things, each with its own failure mode.

The mental model to hold for the whole lesson: Argo CD is a client of many Kubernetes API servers. Its own cluster is just the client’s home. Every other cluster is a remote server it must be handed a URL for and credentials to. A “cluster” in Argo CD is not a mystical binding — it is a rest.Config (the same struct any Kubernetes client library builds from a kubeconfig), serialized into a Secret. Once you see it that way, registration stops being magic and becomes plumbing you fully control.

---

The hub-and-spoke model

In the hub-and-spoke topology, one Argo CD installation — the hub — holds all your Application and ApplicationSet objects and reconciles them onto many target clusters — the spokes. The hub runs in exactly one cluster (often a small, dedicated “management” or “tooling” cluster). Everything else is a spoke it reaches over the network.

The contrast is with standalone Argo CD, one instance per cluster, each managing only itself. Both are valid; the trade-offs are real and worth a table before you commit.

Most enterprises land on hub-and-spoke, because the operational savings dominate and the two downsides — blast radius and credential concentration — are manageable with HA and least-privilege RBAC. A common refinement is a small number of hubs (say one per region or per security tier) rather than one global hub, which caps blast radius without returning to per-cluster sprawl.

The vocabulary is worth pinning down, because the docs and the CLI use these words precisely:

The in-cluster default

Before you register anything, every Argo CD already knows about exactly one cluster: itself. This destination is called in-cluster, and its server URL is the well-known internal address https://kubernetes.default.svc. There is no Secret for it — the hub authenticates to its own API server with the ServiceAccount token that Kubernetes mounts into the argocd-application-controller pods, the same way any in-cluster workload does.

# List clusters on a fresh install — only the hub's own cluster is present
argocd cluster list
# SERVER                          NAME        VERSION  STATUS      MESSAGE  PROJECT
# https://kubernetes.default.svc  in-cluster  1.29     Successful

That single line is the baseline you fan out from. A couple of facts about in-cluster that trip people up:

• Its name is literally in-cluster and its server is literally https://kubernetes.default.svc. An Application that targets the hub itself uses one of those two values in spec.destination.
• It has no cluster labels, which is why an ApplicationSet cluster generator with a matchLabels selector excludes the hub automatically (nothing to match) — a behaviour we lean on later.
• You can disable it (kubectl -n argocd patch configmap argocd-cm ... clusterResources/cluster.inClusterEnabled: "false") if the hub must never deploy to its own cluster — a hardening step for setups where the management cluster runs only Argo CD and nothing else.

Here is the whole hub-and-spoke picture end to end. Read it left to right: the hub holds its own in-cluster destination, plus one cluster Secret per spoke that carries the API URL and the credentials; the hub uses those to authenticate across the network to each managed cluster’s API server; and an Application selects exactly one spoke through its destination.

The badges mark the six things that actually matter: the hub always manages itself (1); a spoke is nothing but a labelled Secret (2); the config JSON picks how you authenticate (3); the network path to a private API server is a separate problem from credentials (4); each cloud authenticates its own way (5); and an Application targets one spoke by destination (6). If you internalise only this diagram, you can reason about any multi-cluster Argo CD topology.

---

The cluster Secret: how Argo CD stores a cluster

Everything the hub knows about a spoke lives in one Kubernetes Secret in the argocd namespace, marked with a specific label. This is the single most important object in the lesson, so we take it apart field by field.

# A cluster Secret — the complete, minimal shape (bearer-token variant)
apiVersion: v1
kind: Secret
metadata:
  name: prod-eks-secret            # any name; NOT the cluster name Argo CD uses
  namespace: argocd                # must be Argo CD's namespace
  labels:
    argocd.argoproj.io/secret-type: cluster   # THIS is what makes it a cluster
    # your own labels below drive the ApplicationSet cluster generator
    cloud: aws
    environment: prod
    region: us-east-1
type: Opaque
stringData:
  name: prod-eks                   # the cluster name used by destination.name
  server: https://ABCD1234.gr7.us-east-1.eks.amazonaws.com   # the API server URL
  config: |
    {
      "bearerToken": "<ServiceAccount-token>",
      "tlsClientConfig": {
        "insecure": false,
        "caData": "<base64-encoded CA PEM>"
      }
    }

Two things make this Secret special, and both are mandatory:

1. The label argocd.argoproj.io/secret-type: cluster. Without it, Argo CD ignores the Secret entirely — it is how the controller finds clusters (and how repo credentials, which use secret-type: repository, are kept separate).
2. It lives in the argocd namespace (or wherever your Argo CD runs). A cluster Secret in another namespace does nothing.

The top-level data keys are a small, fixed set. Learn them once:

Note that config is a JSON string stored inside YAML, not nested YAML. That mismatch is a top-three source of malformed cluster Secrets — a stray unescaped quote or a missing brace and the whole cluster silently fails to load. Using stringData (as above) lets you write the values in plaintext and lets Kubernetes base64-encode them into data for you; if you write data directly, every value must already be base64.

The config JSON, field by field

The config string deserializes into Argo CD’s cluster configuration — effectively a Kubernetes rest.Config. These are its top-level fields; you fill in exactly the subset your auth method needs.

You pick one primary credential path — bearerToken, or tlsClientConfig client cert, or awsAuthConfig, or execProviderConfig — and almost always also supply tlsClientConfig.caData so the hub trusts the spoke’s API server certificate. The tlsClientConfig sub-fields:

And the two cloud-auth objects, which are the reason this lesson exists:

That is the entire vocabulary of a cluster Secret. Every registration method below — imperative or declarative, any cloud — produces a Secret with exactly these fields. The rest of the lesson is just which fields, filled with what.

---

Registering a cluster the imperative way: argocd cluster add

The fast path is one command. You point the argocd CLI at a kubeconfig context that can already reach the target cluster as an admin, and it does the rest:

# Your local kubeconfig already has a context for the target cluster
kubectl config get-contexts
# CURRENT   NAME              CLUSTER           AUTHINFO
# *         hub-mgmt          hub-mgmt          hub-mgmt
#           prod-eks          prod-eks          prod-eks

# Register the prod-eks context as a spoke of this Argo CD
argocd cluster add prod-eks --name prod-eks
# WARNING: This will create a service account `argocd-manager` on the cluster
# referenced by context `prod-eks` with full cluster level privileges. Do you
# want to continue [y/N]? y
# INFO[0002] ServiceAccount "argocd-manager" created in namespace "kube-system"
# INFO[0002] ClusterRole "argocd-manager-role" created
# INFO[0002] ClusterRoleBinding "argocd-manager-role-binding" created
# Cluster 'https://ABCD1234.gr7.us-east-1.eks.amazonaws.com' added

What it actually does under the hood

That prompt is not boilerplate — it is telling you exactly what is about to be created on the target cluster. Step by step:

The default argocd-manager-role is deliberately broad — it grants full access so Argo CD can deploy anything:

# The ClusterRole argocd cluster add creates on the spoke (default: full access)
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: argocd-manager-role
rules:
  - apiGroups: ["*"]
    resources: ["*"]
    verbs: ["*"]
  - nonResourceURLs: ["*"]
    verbs: ["*"]

The credential it grabs is the argocd-manager ServiceAccount’s token, and this is the crucial security point: it is a static, long-lived bearer token written verbatim into the cluster Secret’s config.bearerToken on the hub. On Kubernetes 1.24+ ServiceAccounts no longer auto-generate token Secrets, so argocd cluster add explicitly creates one (a Secret annotated kubernetes.io/service-account.name: argocd-manager) to obtain a non-expiring token. Convenient — and a liability, because you cannot rotate it without re-registering, and if the hub is compromised, that token is cluster-admin on the spoke. For cloud clusters, the exec-provider approach in the next sections avoids this entirely.

The most useful flags on argocd cluster add:

# Register with labels the cluster generator can select on later
argocd cluster add prod-eks \
  --name prod-eks \
  --label cloud=aws --label environment=prod --label region=us-east-1

argocd cluster add is perfect for a quick spoke, a lab, or bootstrapping. But notice what it did: it created a static token and mutated the target cluster imperatively. Neither of those is GitOps. The credential now lives only in a Secret nobody reviewed, and the RBAC on the spoke was created by a laptop. For anything you want reproducible and auditable, hand-write the Secret instead.

---

Registering a cluster the declarative way

The GitOps way to register a spoke is to author the cluster Secret yourself and kubectl apply it (ideally through Argo CD itself, managed like any other resource). You get review, version history, and — the big win for cloud — you can use exec providers so no static token ever exists.

The three credential shapes you will actually write:

Variant A — static ServiceAccount bearer token (portable, but rotate-averse)

This is the declarative equivalent of what argocd cluster add produces. You create the argocd-manager SA and RBAC on the spoke (also in Git), read its token, and reference it. It works on any Kubernetes cluster, cloud or not.

apiVersion: v1
kind: Secret
metadata:
  name: onprem-prod-secret
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: cluster
    cloud: onprem
    environment: prod
type: Opaque
stringData:
  name: onprem-prod
  server: https://k8s-api.internal.example.com:6443
  config: |
    {
      "bearerToken": "<argocd-manager ServiceAccount token>",
      "tlsClientConfig": {
        "insecure": false,
        "caData": "<base64 of the API server CA certificate>"
      }
    }

Never commit the real bearerToken or caData to Git in plaintext. Use Sealed Secrets, the External Secrets Operator, or SOPS to encrypt them, and let Argo CD (or your bootstrap) materialise the decrypted Secret. The placeholders above are exactly that — placeholders. This is the same secrets discipline covered for repositories in Connecting Repositories.

Variant B — TLS client certificate (mTLS)

Some clusters (kubeadm defaults, on-prem, kOps) authenticate with a client certificate rather than a token. Fill certData/keyData instead of bearerToken:

stringData:
  name: onprem-mtls
  server: https://k8s-api.internal.example.com:6443
  config: |
    {
      "tlsClientConfig": {
        "insecure": false,
        "caData": "<base64 API server CA PEM>",
        "certData": "<base64 client cert PEM>",
        "keyData": "<base64 client key PEM>"
      }
    }

Variant C — exec provider (the cloud way, no static token)

For AKS, EKS and GKE you almost always want an execProviderConfig: instead of storing a token that never changes, Argo CD runs a small plugin every time it needs to talk to the spoke, and the plugin mints a fresh, short-lived token from the cloud’s identity system. The token in flight is valid for minutes, there is nothing long-lived to leak, and access is governed by real cloud IAM.

# EKS via the aws CLI exec plugin (one of three cloud variants — see next section)
apiVersion: v1
kind: Secret
metadata:
  name: prod-eks-secret
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: cluster
    cloud: aws
    environment: prod
type: Opaque
stringData:
  name: prod-eks
  server: https://ABCD1234.gr7.us-east-1.eks.amazonaws.com
  config: |
    {
      "execProviderConfig": {
        "apiVersion": "client.authentication.k8s.io/v1beta1",
        "command": "aws",
        "args": ["--region", "us-east-1", "eks", "get-token",
                 "--cluster-name", "prod-eks", "--output", "json"]
      },
      "tlsClientConfig": {
        "insecure": false,
        "caData": "<base64 EKS API server CA>"
      }
    }

# Apply it — the GitOps way, or straight to the API for a lab
kubectl apply -f prod-eks-secret.yaml
# secret/prod-eks-secret created

argocd cluster list
# SERVER                                                    NAME       VERSION  STATUS       MESSAGE  PROJECT
# https://kubernetes.default.svc                            in-cluster 1.29     Successful
# https://ABCD1234.gr7.us-east-1.eks.amazonaws.com          prod-eks   1.29     Successful

The imperative-vs-declarative choice, summarised:

The exec-provider Secret is the shape you want in production on every cloud. What differs per cloud is the command, the args, the env, and the identity the plugin uses — which is the multi-cloud edge, and the core of this lesson.

---

The multi-cloud auth edge

Here is the crux. An Argo CD pod on the hub has to prove its identity to a foreign cluster’s API server. Kubernetes itself does not care which cloud that cluster runs on — a bearer token is a bearer token — but how you obtain a valid token differs completely across AKS, EKS and GKE, because each cloud gates its API server behind its own identity system. Get this right per cloud, or the cluster shows Failed with an auth error no amount of RBAC will fix.

The one table to memorise:

The pattern is identical in all three: the hub has a cloud identity → the exec plugin turns that identity into a short-lived Kubernetes token → the spoke’s API server trusts it because you granted that identity RBAC on the spoke. Two grants are always in play — the cloud IAM grant (so the plugin can mint a token) and the Kubernetes RBAC grant (so the token is authorized). Miss either and you get a different error; the troubleshooting table later separates them.

Now the concrete execProviderConfig for each. These are the config JSON bodies; wrap each in the same cluster Secret shell from the previous section.

AKS — Entra ID via kubelogin

AKS clusters with Entra (AAD) integration authenticate through kubelogin, which exchanges an Azure identity for a Kubernetes token. az aks get-credentials writes a kubeconfig whose exec block calls kubelogin get-token; you mirror that in the cluster Secret. For a hub running in Azure, use --login workloadidentity (Azure Workload Identity) or --login msi (a managed identity).

# Fetch admin/user credentials, then convert the kubeconfig to kubelogin (exec) form
az aks get-credentials --resource-group rg-prod --name prod-aks --overwrite-existing
kubelogin convert-kubeconfig -l workloadidentity   # or: -l azurecli (for a laptop)

{
  "execProviderConfig": {
    "apiVersion": "client.authentication.k8s.io/v1beta1",
    "command": "kubelogin",
    "args": [
      "get-token",
      "--login", "workloadidentity",
      "--server-id", "6dae42f8-4368-4678-94ff-3960e28e3630",
      "--environment", "AzurePublicCloud"
    ],
    "env": {
      "AZURE_CLIENT_ID": "<hub-managed-identity-client-id>",
      "AZURE_TENANT_ID": "<tenant-id>",
      "AZURE_FEDERATED_TOKEN_FILE": "/var/run/secrets/azure/tokens/azure-identity-token",
      "AZURE_AUTHORITY_HOST": "https://login.microsoftonline.com/"
    }
  },
  "tlsClientConfig": { "insecure": false, "caData": "<base64 AKS API CA>" }
}

The --server-id value 6dae42f8-4368-4678-94ff-3960e28e3630 is the well-known first-party application ID of the Azure Kubernetes Service AAD Server — it is the same across all AKS clusters and is the audience the token is minted for. The AZURE_FEDERATED_TOKEN_FILE path is where the Azure Workload Identity webhook projects the hub’s federated token; kubelogin reads it and exchanges it for an AKS token. On the spoke you must bind that identity’s object ID to Kubernetes RBAC (a ClusterRoleBinding referencing the group/OID), or the token authenticates but is denied.

EKS — AWS IAM via aws eks get-token or native awsAuthConfig

EKS has two clean paths, and this is where Argo CD is nicest: it can call AWS STS itself through awsAuthConfig, so you do not even need the aws CLI in the image.

# Write a kubeconfig (exec plugin form) for reference / laptop use
aws eks update-kubeconfig --region us-east-1 --name prod-eks

Native awsAuthConfig — preferred, no exec binary required:

{
  "awsAuthConfig": {
    "clusterName": "prod-eks",
    "roleARN": "arn:aws:iam::111122223333:role/argocd-hub-eks-access"
  },
  "tlsClientConfig": { "insecure": false, "caData": "<base64 EKS API CA>" }
}

Exec-plugin form — if you prefer to call the AWS CLI explicitly:

{
  "execProviderConfig": {
    "apiVersion": "client.authentication.k8s.io/v1beta1",
    "command": "aws",
    "args": ["--region", "us-east-1", "eks", "get-token",
             "--cluster-name", "prod-eks", "--output", "json"],
    "env": { "AWS_STS_REGIONAL_ENDPOINTS": "regional" }
  },
  "tlsClientConfig": { "insecure": false, "caData": "<base64 EKS API CA>" }
}

The hub’s identity here is an IAM role, granted to the Argo CD pods via IRSA (IAM Roles for Service Accounts) or the newer EKS Pod Identity. Then — the step everyone forgets — that IAM role (or the roleARN it assumes) must be authorised on the spoke: either an entry in the legacy aws-auth ConfigMap, or a modern EKS access entry mapping the role to Kubernetes groups. The token is an STS presigned URL valid ~15 minutes, minted fresh every request, so there is nothing long-lived to leak.

GKE — Google Cloud IAM via gke-gcloud-auth-plugin

GKE moved to an external auth plugin (gke-gcloud-auth-plugin) for kubectl 1.26+. It authenticates using Application Default Credentials (ADC) — for a hub in GKE, that means Workload Identity mapping the Argo CD Kubernetes ServiceAccount to a Google service account.

# Install the plugin and fetch credentials
gcloud components install gke-gcloud-auth-plugin
gcloud container clusters get-credentials prod-gke --region us-central1 --project my-proj

{
  "execProviderConfig": {
    "apiVersion": "client.authentication.k8s.io/v1beta1",
    "command": "gke-gcloud-auth-plugin",
    "installHint": "Install gke-gcloud-auth-plugin: gcloud components install gke-gcloud-auth-plugin"
  },
  "tlsClientConfig": { "insecure": false, "caData": "<base64 GKE API CA>" }
}

The plugin reads ADC from the environment — with Workload Identity, the Argo CD SA’s projected token is exchanged for a Google OAuth2 access token (~1 hour) that GKE’s API server trusts. On the spoke, bind the Google service account to Kubernetes RBAC. GKE is the fiddliest of the three because the plugin relies on ambient ADC rather than explicit args, so the hub’s environment (Workload Identity annotations, GOOGLE_APPLICATION_CREDENTIALS, or metadata server access) has to be correct — a valid ADC but missing RBAC yields a token that is accepted and then denied.

The per-cloud “how do I even get the kubeconfig” commands, side by side, because you will run these constantly:

And the hub-side identity you attach so the plugin has something to exchange:

One more honesty note the brief demands: you do not have to use the cloud IAM path. All three clouds are ordinary Kubernetes underneath, so a static argocd-manager ServiceAccount token (Variant A) works everywhere and sidesteps every plugin. Teams choose it for simplicity and accept the rotation cost; teams that must avoid long-lived credentials choose the exec/native path. Know both — reach for exec/IAM on anything production, fall back to the SA token when the plugin plumbing is not worth it.

---

Scoped clusters, labels & the ApplicationSet cluster generator

A registered cluster does not have to be all-or-nothing. Two scoping mechanisms and one selection mechanism turn a flat list of clusters into a governed fleet.

Cluster labels (and how the generator uses them)

Any labels on the cluster Secret’s metadata.labels (beyond the mandatory secret-type) become selectable attributes. Label every cluster consistently at registration and you never hand-write a per-cluster Application again — the ApplicationSet cluster generator fans out by selector. Common label conventions:

The cluster generator reads these directly. This ties straight into the ApplicationSets and generators lesson, where generators are covered in full — here is just the join:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: monitoring-everywhere
  namespace: argocd
spec:
  goTemplate: true
  generators:
    - clusters:
        selector:
          matchLabels:
            environment: prod       # every prod cluster, any cloud
  template:
    metadata:
      name: 'monitoring-{{.name}}'
    spec:
      project: platform
      source:
        repoURL: https://github.com/acme/platform-gitops.git
        targetRevision: main
        path: monitoring
      destination:
        server: '{{.server}}'       # the generator supplies each cluster's URL
        namespace: monitoring

Add a prod cluster Secret with environment: prod and a monitoring-<name> Application appears automatically. Because the selector is non-empty, the hub’s own in-cluster (which has no labels) is excluded — exactly what you want when a workload should hit only registered spokes.

Namespaced (scoped) clusters

By default a registered cluster is cluster-scoped: apps on it can create resources in any namespace and manage cluster-scoped resources (CRDs, ClusterRoles). Set the namespaces key to restrict it to a list, turning it into a namespaced cluster:

stringData:
  name: shared-tenant-cluster
  server: https://shared.example.com
  namespaces: "team-a,team-a-jobs,team-a-ci"   # apps may only touch these
  clusterResources: "false"                      # and no cluster-scoped resources
  config: '{"bearerToken":"...","tlsClientConfig":{"caData":"..."}}'

This is how one physical cluster is safely shared between tenants without letting a stray Application create a ClusterRole or land in kube-system.

Project-scoped clusters

Set the project key and the cluster Secret is bound to a single AppProject — only that project’s Applications may target it. This is a control-plane guardrail rather than a target-side one, and it pairs with the destination allow-lists covered in AppProjects and multi-tenancy boundaries:

stringData:
  name: payments-prod
  server: https://payments.example.com
  project: team-payments      # only Applications in this project can use this cluster
  config: '{"bearerToken":"...","tlsClientConfig":{"caData":"..."}}'

---

Targeting deployments

With clusters registered, an Application chooses one through spec.destination. There are two ways to name the target, and picking the right one avoids a whole class of bugs.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: web-prod-eks
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/acme/app-config.git
    targetRevision: main
    path: web/overlays/prod
  destination:
    name: prod-eks          # by cluster NAME (the Secret's `name` field)
    namespace: web
  syncPolicy:
    automated: { prune: true, selfHeal: true }

or, equivalently, by API URL:

  destination:
    server: https://ABCD1234.gr7.us-east-1.eks.amazonaws.com   # by SERVER url
    namespace: web

The rules:

Set exactly one of server or name, never both — specifying both is a validation error. Prefer name for anything humans edit: it is readable, and if a cluster’s API endpoint ever changes (a rebuild, a private-endpoint migration), the Applications keep working because they never hard-coded the URL. Use server in generated Applications where the ApplicationSet already has the URL in hand ({{.server}}).

To deploy the same app to many clusters, do not copy the Application N times — use the ApplicationSet cluster generator from the previous section, which stamps one Application per matching cluster and fills destination.server with each cluster’s URL. That is the whole multi-cluster fan-out story: label clusters once, select them with a generator, and adding a spoke makes apps appear with no new manifests.

Connectivity: the part credentials do not solve

A perfectly valid cluster Secret still fails if the hub cannot reach the spoke’s API server. Registration is authentication; it is not networking. The hub’s application-controller (and repo-server for some operations) opens a TCP connection to the server URL, and that has to succeed:

Private-cluster connectivity — VNet peering for private AKS, VPC peering or PrivateLink for private EKS, authorized networks and Private Service Connect for GKE — is a substantial topic in its own right and a design decision that belongs with your networking team. For this lesson, the thing to internalise is the split: a Failed cluster with an auth message is a credentials problem; a Failed cluster with dial tcp ... i/o timeout is a connectivity problem. They live in different teams’ backlogs.

Verifying a registration

# List every cluster and its connection state
argocd cluster list
# SERVER                                            NAME        VERSION  STATUS       MESSAGE
# https://kubernetes.default.svc                    in-cluster  1.29     Successful
# https://prod-aks-dns-xxxx.hcp.westeurope...       prod-aks    1.29     Successful
# https://ABCD1234.gr7.us-east-1.eks.amazonaws.com  prod-eks    1.29     Successful
# https://34.72.x.x                                 prod-gke    1.29     Successful

# Full detail for one cluster, including the connection state and config
argocd cluster get prod-eks

# The underlying Secrets (labels drive the cluster generator)
kubectl get secrets -n argocd -l argocd.argoproj.io/secret-type=cluster \
  -o custom-columns=NAME:.metadata.name,SERVER:.data.server

STATUS has three values worth knowing cold:

Unknown surprises people: Argo CD does not eagerly probe a cluster no Application targets, so a freshly added spoke can sit at Unknown until you point an app at it. That is not an error — it is laziness by design.

---

Hands-on lab

You will register a remote cluster both ways, inspect exactly what each produced, deploy an app to the spoke by name, and tear it all down. Because this machine has no clusters attached, the commands below show the real invocations and representative output (labelled as such) — run them against your own hub plus one spoke to see the live equivalents. Nothing here invents a field or a flag.

⚠️ Cloud spokes bill. A managed cluster’s control plane and any load balancers you create cost money for as long as they exist. Do the lab against clusters you already run, and complete the teardown at the end. Never commit a real bearerToken or caData to Git.

Step 1 — Confirm your starting point (only the hub).

argocd cluster list
# (representative)
# SERVER                          NAME        VERSION  STATUS       MESSAGE
# https://kubernetes.default.svc  in-cluster  1.29     Successful

What just happened: a fresh hub knows only itself. Everything below adds a spoke.

Step 2 — Register imperatively with argocd cluster add. Point at a kubeconfig context that can already reach the spoke as admin.

kubectl config get-contexts             # find the spoke's context name
argocd cluster add prod-eks --name prod-eks \
  --label cloud=aws --label environment=prod
# WARNING: This will create a service account `argocd-manager` ... [y/N]? y
# INFO ServiceAccount "argocd-manager" created in namespace "kube-system"
# INFO ClusterRole "argocd-manager-role" created
# INFO ClusterRoleBinding "argocd-manager-role-binding" created
# Cluster 'https://ABCD1234.gr7.us-east-1.eks.amazonaws.com' added

What just happened: Argo CD created the argocd-manager ServiceAccount + full-access ClusterRole/binding on the spoke, grabbed that SA’s token, and wrote a cluster Secret on the hub. Verify:

argocd cluster list        # prod-eks now appears, STATUS Successful once an app probes it
# Inspect the hub-side Secret this created:
kubectl get secret -n argocd -l argocd.argoproj.io/secret-type=cluster

Step 3 — Inspect what it wrote (the static token).

kubectl get secret -n argocd <cluster-secret-name> -o jsonpath='{.data.config}' | base64 -d
# (representative — note the long-lived bearerToken)
# {"bearerToken":"eyJhbGciOi...","tlsClientConfig":{"insecure":false,"caData":"LS0t..."}}

What just happened: you are looking at the liability — a static bearerToken that is cluster-admin on the spoke and cannot be rotated without re-registering. Fine for a lab; not what you want in prod. So do it the other way too.

Step 4 — Remove it, then register declaratively.

argocd cluster rm https://ABCD1234.gr7.us-east-1.eks.amazonaws.com
# Cluster 'https://ABCD1234.gr7.us-east-1.eks.amazonaws.com' removed

Now author the cluster Secret with an exec provider (EKS shown; the AKS and GKE config bodies from the multi-cloud section drop straight in):

# prod-eks-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: prod-eks-secret
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: cluster
    cloud: aws
    environment: prod
type: Opaque
stringData:
  name: prod-eks
  server: https://ABCD1234.gr7.us-east-1.eks.amazonaws.com
  config: |
    {
      "execProviderConfig": {
        "apiVersion": "client.authentication.k8s.io/v1beta1",
        "command": "aws",
        "args": ["--region", "us-east-1", "eks", "get-token",
                 "--cluster-name", "prod-eks", "--output", "json"]
      },
      "tlsClientConfig": { "insecure": false, "caData": "<base64 EKS API CA>" }
    }

Paired AKS and GKE config bodies (same Secret shell, swap the config):

// AKS
{
  "execProviderConfig": {
    "apiVersion": "client.authentication.k8s.io/v1beta1",
    "command": "kubelogin",
    "args": ["get-token", "--login", "workloadidentity",
             "--server-id", "6dae42f8-4368-4678-94ff-3960e28e3630",
             "--environment", "AzurePublicCloud"],
    "env": { "AZURE_CLIENT_ID": "<mi-client-id>", "AZURE_TENANT_ID": "<tenant-id>",
             "AZURE_FEDERATED_TOKEN_FILE": "/var/run/secrets/azure/tokens/azure-identity-token",
             "AZURE_AUTHORITY_HOST": "https://login.microsoftonline.com/" }
  },
  "tlsClientConfig": { "insecure": false, "caData": "<base64 AKS API CA>" }
}

// GKE
{
  "execProviderConfig": {
    "apiVersion": "client.authentication.k8s.io/v1beta1",
    "command": "gke-gcloud-auth-plugin",
    "installHint": "gcloud components install gke-gcloud-auth-plugin"
  },
  "tlsClientConfig": { "insecure": false, "caData": "<base64 GKE API CA>" }
}

kubectl apply -f prod-eks-secret.yaml
# secret/prod-eks-secret created
argocd cluster list
# (representative) prod-eks  1.29  Successful

What just happened: the spoke is registered again — but now with no static token. Argo CD runs the exec plugin on every API call to mint a fresh, short-lived token from AWS IAM. The Secret is a reviewable file you can commit (encrypted) to Git.

Step 5 — Deploy an app to the spoke by name.

argocd app create web-prod-eks \
  --repo https://github.com/acme/app-config.git \
  --path web/overlays/prod \
  --dest-name prod-eks \
  --dest-namespace web \
  --sync-policy automated
# application 'web-prod-eks' created

argocd app get web-prod-eks
# (representative)
# Name:        argocd/web-prod-eks
# Cluster:     prod-eks (https://ABCD1234.gr7.us-east-1.eks.amazonaws.com)
# Namespace:   web
# Health Status:  Healthy
# Sync Status:    Synced to main (a1b2c3d)

What just happened: --dest-name prod-eks matched the cluster Secret’s name field, and Argo CD authenticated with the exec provider, cloned the repo, rendered, and synced onto the spoke. The app runs on EKS while its Application object lives on the hub.

Step 6 — Teardown (leave no static tokens, no orphan SAs).

# 1. Delete the app (prune removes its workloads from the spoke)
argocd app delete web-prod-eks --cascade
# 2. Remove the declaratively-registered cluster
kubectl delete secret -n argocd prod-eks-secret
# 3. If you kept the imperative registration at any point, also remove the
#    argocd-manager identity it created ON THE SPOKE:
kubectl --context prod-eks -n kube-system delete serviceaccount argocd-manager
kubectl --context prod-eks delete clusterrole argocd-manager-role
kubectl --context prod-eks delete clusterrolebinding argocd-manager-role-binding
argocd cluster list      # back to just in-cluster

What just happened: you deleted the app and its resources on the spoke, removed the hub-side cluster Secret, and — critically — cleaned up the argocd-manager ServiceAccount and RBAC that argocd cluster add had left behind on the spoke. argocd cluster rm removes those for you; a hand-deleted Secret does not, so you do it explicitly.

---

Common mistakes and troubleshooting

Multi-cluster failures cluster (pun intended) into credentials, RBAC, networking, and plain typos. The table separates them by the symptom you actually see; the prose covers the three that cost the most hours.

Three gotchas deserve extra words.

1. The exec plugin has to exist inside Argo CD’s pods. This is the single most common surprise. Your laptop has kubelogin, aws, and gke-gcloud-auth-plugin, so kubectl works there — but the Argo CD application-controller and repo-server pods do not ship those binaries. An execProviderConfig referencing kubelogin fails with executable file not found in $PATH until you put the binary in the image (a custom image FROM the Argo CD image with the plugin added, or an init container that copies it onto a shared PATH volume). The static-token path (Variant A) has no such requirement, which is one reason teams reach for it despite the rotation cost.

2. Authentication and authorization are two different failures. A valid token that reaches the API server can still be denied. The plugin minting a token proves the hub’s cloud identity; it does not grant that identity any Kubernetes permissions. Unauthenticated means the token is bad (auth failed); forbidden / cannot list ... in the namespace means the token is fine but no RBAC binds your identity (authorization failed). On EKS the authorization grant is an aws-auth entry or EKS access entry; on AKS it is a ClusterRoleBinding to the Entra object ID; on GKE it is a binding to the Google service account. Reading the exact word in the error tells you which of the two to fix.

3. name vs server mismatches route apps into the void. destination.server must match the registered server byte-for-byte — a trailing slash, :443 present on one side and absent on the other, or an IP vs a DNS name are all “no match,” and the app reports the cluster as not found or sits Unknown. destination.name is safer for hand-written Applications because it matches a stable, human-chosen string. And do not confuse the cluster Secret’s metadata.name (the Kubernetes object name, irrelevant to targeting) with its stringData.name (the cluster name destination.name matches) — they are different fields and only the latter matters for routing.

---

Going deeper

The core lesson gets you a working spoke. Production adds three questions the happy path skips: how little power can the hub hold on a spoke, how does one hub carry hundreds of spokes, and how do you take a cluster out without stranding the apps that lived on it.

Least-privileging the argocd-manager identity

argocd cluster add grants the hub a ClusterRole of */*/* — every verb on every resource, cluster-wide. That is convenient, and it is also the single largest credential in your platform: if the hub is compromised, that token is cluster-admin on every spoke. The namespaces and clusterResources: "false" keys from earlier are Argo CD-side guardrails — the application-controller refuses to act outside them — but they do not shrink the Kubernetes RBAC the token actually carries. Defense in depth means tightening both layers.

The Kubernetes-side fix is to stop using the default role. Pre-create your own ServiceAccount and a scoped Role/ClusterRole on the spoke (in Git, of course), then register against it with --service-account so Argo CD never creates its full-access one:

# On the SPOKE — a hub identity scoped to two namespaces, no cluster-scope write
apiVersion: v1
kind: ServiceAccount
metadata: { name: argocd-manager, namespace: kube-system }
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata: { name: argocd-manager-read }
rules:
  # read-only cluster-wide so Argo CD can diff and keep its resource cache warm
  - apiGroups: ["*"]
    resources: ["*"]
    verbs: ["get", "list", "watch"]
---
# full write, but ONLY inside an app namespace (one Role + RoleBinding per ns)
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: { name: argocd-manager-write, namespace: team-a }
rules:
  - apiGroups: ["*"]
    resources: ["*"]
    verbs: ["*"]

# Bind the read ClusterRole and the per-namespace write Roles, then register
kubectl create clusterrolebinding argocd-manager-read \
  --clusterrole=argocd-manager-read \
  --serviceaccount=kube-system:argocd-manager
kubectl -n team-a create rolebinding argocd-manager-write \
  --role=argocd-manager-write --serviceaccount=kube-system:argocd-manager

argocd cluster add prod-eks --name prod-eks \
  --service-account argocd-manager --system-namespace kube-system \
  --namespace team-a --namespace team-b

Broad read is close to a floor you cannot go under: Argo CD must get/list/watch widely to compute diffs and build its cluster cache, so a role with no read is unusable. Grant write narrowly and you cap the blast radius without blinding the controller. Combine this with the Secret’s namespaces/clusterResources keys and a project binding, and the hub’s reach over that spoke is now defined in three independent places — remove any one and it still cannot run wild.

How one hub carries a large fleet: controller sharding

The unit that costs the hub is the cluster, not the Application. For every registered spoke the argocd-application-controller opens watch streams and holds a live in-memory cache of that cluster’s resources — that is how it detects drift without re-listing on every reconcile. Cache plus watches means each spoke has a real memory and connection cost, so a hub reconciling a hundred clusters is doing a hundred clusters’ worth of caching in one process. Past a few dozen busy clusters, you shard.

Sharding splits clusters across multiple controller replicas. The controller runs as a StatefulSet; you scale its replicas and tell it how many shards exist:

# Run 3 controller shards — replicas MUST match the env var
kubectl -n argocd scale statefulset argocd-application-controller --replicas=3
kubectl -n argocd set env statefulset/argocd-application-controller \
  ARGOCD_CONTROLLER_REPLICAS=3

Each cluster is assigned to exactly one shard. The assignment algorithm is set via controller.sharding.algorithm (or ARGOCD_CONTROLLER_SHARDING_ALGORITHM):

For a spoke that is disproportionately heavy — a huge prod cluster — pin it to its own shard with the Secret’s shard field (the same key from the anatomy table) so it never shares a replica:

stringData:
  name: prod-eks
  shard: "2"      # this cluster always lands on shard index 2
  # ...server and config as usual

Newer Argo CD also offers dynamic (heartbeat-based) cluster distribution, which lets you add or remove controller replicas and have clusters rebalance without a full restart — valuable at fleet scale, and worth enabling deliberately rather than assuming it is on. Repo-server throughput and API fan-out scale separately: the scaling and sharding lesson covers the whole performance picture, and the multi-cloud fleet lesson puts this to work across AKS, EKS and GKE at once.

Removing a cluster safely

Deleting a cluster Secret is not the same as decommissioning a spoke, and doing it in the wrong order strands resources. An Application whose destination cluster has vanished cannot be reconciled — the controller reports a ComparisonError (it can no longer reach the cluster to diff), and, crucially, it will not prune the app’s workloads, because pruning requires talking to the very API server you just cut off. The live resources keep running on the spoke with nothing managing them.

The safe order is apps first, cluster second:

# 1. Delete (or migrate) every Application targeting the spoke FIRST, so their
#    resources are pruned from the spoke while the credentials still work
argocd app delete web-prod-eks --cascade

# 2. THEN remove the registration. Prefer `argocd cluster rm`, which also
#    deletes the argocd-manager SA/ClusterRole/binding it created on the spoke
argocd cluster rm prod-eks

If you registered declaratively, deleting the Secret removes the hub-side record but leaves any argocd-manager RBAC you created on the spoke — clean that up yourself (the lab’s teardown does exactly this). And if an Application is stuck deleting on a finalizer while the cluster is already unreachable, remove the resources-finalizer.argocd.argoproj.io finalizer so the Application object can be deleted from the hub even though its workloads can no longer be pruned remotely.

---

Common beginner mistakes

These are model-level misunderstandings, not symptoms — the troubleshooting table above tells you what to do when something breaks; this tells you what to believe so it breaks less.

• “The Secret’s metadata.name is the cluster name.” It is not. metadata.name is just the Kubernetes object’s name (call it whatever you like); the name Argo CD shows and that destination.name matches is the stringData.name field inside the Secret. Two different fields, and only the inner one routes deployments.

• “Once a cluster is registered, Argo CD can deploy to it.” Registration only creates a record and an identity. Deploying also needs the hub to reach the spoke’s API server over the network, and needs that identity to be authorized by Kubernetes RBAC on the spoke. Record, reachability, authorization — three independent things, each with its own failure (config error, i/o timeout, forbidden). Private endpoints in particular are a whole networking topic, covered in Private clusters and connectivity.

• “The exec plugin works because it’s installed on my machine.” Your laptop’s kubelogin/aws/gke-gcloud-auth-plugin is irrelevant. The binary must live inside the Argo CD pods (application-controller and repo-server), because that is where the plugin actually runs. A perfect execProviderConfig fails with executable file not found in $PATH until you bake the binary into the image.

• “A short-lived token is risky — it might expire mid-sync.” Backwards. Argo CD re-runs the exec plugin and mints a fresh token whenever it needs one, so there is no window where an expired token blocks a sync. Short-lived is more secure precisely because nothing durable exists to steal — the static ServiceAccount token is the risky one.

• “namespaces: a,b,c on the Secret is real isolation.” It is an Argo CD-side guardrail — the controller refuses to act outside those namespaces — but the underlying credential may still be cluster-admin. Anyone holding the token (or a bug that bypasses the controller) is not stopped by it. Real isolation also needs least-privilege RBAC on the spoke (see Going deeper). A guardrail and a permission boundary are not the same thing.

• “Cloud clusters require an exec provider.” They do not. EKS, AKS and GKE are ordinary Kubernetes underneath, so a static argocd-manager ServiceAccount token (Variant A) authenticates to all three. Exec/native IAM is preferred in production to avoid long-lived credentials — but it is a choice, not a requirement, and reaching for it in a five-minute lab is exactly how beginners get stuck on a missing plugin binary.

• “Adding more spokes to one hub is free.” Each registered cluster costs the hub a live resource cache and watch connections in the application-controller. Dozens of busy clusters on one unsharded controller is how a hub runs out of memory. Plan sharding before you feel it, not after.

---

Practice challenges

Work these against your own hub plus one spoke. They escalate from reading state to locking a spoke down; each solution is real and schema-correct, with placeholders where a real value would go.

1. (Beginner) Prove the hub already knows one cluster before you register anything. List the clusters and identify which entry is the hub itself, its server URL, and whether it has a backing Secret.

<details> <summary>Solution</summary>

argocd cluster list
# SERVER                          NAME        VERSION  STATUS       MESSAGE
# https://kubernetes.default.svc  in-cluster  1.29     Successful

The hub is the in-cluster entry, server https://kubernetes.default.svc. It has no cluster Secret — the controller uses its own mounted ServiceAccount token. Why it matters: this is the baseline you fan out from; every other row will be a Secret you can inspect. </details>

2. (Beginner) Find the Secret behind a registered spoke, and name the one label without which Argo CD ignores it.

<details> <summary>Solution</summary>

kubectl get secrets -n argocd -l argocd.argoproj.io/secret-type=cluster

The mandatory label is argocd.argoproj.io/secret-type: cluster. Remove or misspell it and the Secret is invisible to Argo CD — it is what marks the Secret as a cluster (versus a repository). Why it matters: a cluster is just a labelled Secret, and the label is the whole registration trigger. </details>

3. (Intermediate) Hand-write a declarative cluster Secret for an on-prem spoke using a static ServiceAccount token, and add two labels an ApplicationSet cluster generator can select on.

<details> <summary>Solution</summary>

apiVersion: v1
kind: Secret
metadata:
  name: onprem-prod-secret
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: cluster
    environment: prod
    cloud: onprem
type: Opaque
stringData:
  name: onprem-prod
  server: https://k8s-api.internal.example.com:6443
  config: |
    {
      "bearerToken": "REDACTED",
      "tlsClientConfig": { "insecure": false, "caData": "REDACTED-base64-CA" }
    }

Why it matters: this is the portable Variant A that works on any cluster; the environment/cloud labels are what the cluster generator’s matchLabels fans out on. Encrypt the token before it ever touches Git. </details>

4. (Intermediate) A spoke was added with argocd cluster add and has no labels, so your matchLabels: {environment: prod} generator skips it. Add the label without re-registering.

<details> <summary>Solution</summary>

# Labels live on the cluster Secret's metadata — patch the existing Secret
kubectl label secret -n argocd <cluster-secret-name> \
  environment=prod cloud=aws

Why it matters: the cluster generator reads metadata.labels on the Secret, not anything in config; labelling the existing Secret is enough — no re-registration, no new token. </details>

5. (Advanced) Migrate a spoke from its static argocd cluster add token to an EKS exec provider without re-registering — and name the two things that must already be in place.

<details> <summary>Solution</summary>

Edit the existing cluster Secret’s config to drop the bearerToken and use execProviderConfig:

stringData:
  config: |
    {
      "execProviderConfig": {
        "apiVersion": "client.authentication.k8s.io/v1beta1",
        "command": "aws",
        "args": ["--region", "us-east-1", "eks", "get-token",
                 "--cluster-name", "prod-eks", "--output", "json"]
      },
      "tlsClientConfig": { "insecure": false, "caData": "REDACTED-base64-CA" }
    }

Two things must already exist or the switch fails: (1) the aws binary baked into the controller/repo-server image, and (2) the hub’s IAM role (IRSA / Pod Identity) mapped on the spoke via an EKS access entry or aws-auth. Why it matters: exec auth removes the long-lived token, but only works when both the plugin and the cloud-to-RBAC mapping are present — authentication and authorization are separate. </details>

6. (Advanced) Least-privilege a spoke so the hub can write only in team-a and team-b and never touch cluster-scoped resources — at BOTH the Argo CD layer and the Kubernetes RBAC layer.

<details> <summary>Solution</summary>

Argo CD layer — scope the cluster Secret:

stringData:
  name: shared-tenant
  server: https://shared.example.com
  namespaces: "team-a,team-b"
  clusterResources: "false"
  config: '{"bearerToken":"REDACTED","tlsClientConfig":{"caData":"REDACTED"}}'

Kubernetes layer — register against a pre-created SA (read-only cluster-wide, write only in those namespaces, as in Going deeper) so the token itself is not cluster-admin:

argocd cluster add shared-tenant --name shared-tenant \
  --service-account argocd-manager --system-namespace kube-system \
  --namespace team-a --namespace team-b

Why it matters: the namespaces key is a controller guardrail, not a permission boundary; only scoped RBAC on the spoke actually limits what the credential can do. Do both — defense in depth. </details>

---

Cheat-sheet

Everything in one place — the cluster Secret shape, the argocd cluster verbs, and the per-cloud exec config.

Cluster Secret — the shape:

argocd cluster verbs:

Per-cloud exec config (drop into stringData.config):

Destination — target selection:

---

Interview and exam questions

Q: What is the hub-and-spoke model in Argo CD, and why do enterprises prefer it? A: One Argo CD installation (the hub) holds all Applications and reconciles them onto many target clusters (spokes) over the network, instead of running a separate Argo CD per cluster. Enterprises prefer it for a single pane of glass across the fleet, one control plane to upgrade and secure, and fleet-wide policy — at the cost of a larger blast radius if the hub goes down and a concentration of credentials the hub must protect.

Q: How does Argo CD store the information about a remote cluster? A: As a Kubernetes Secret in the argocd namespace labelled argocd.argoproj.io/secret-type: cluster. Its data holds name, server (the API URL) and config (a JSON string describing how to authenticate), plus optional namespaces, clusterResources, and project for scoping. That one Secret is the registration.

Q: What does argocd cluster add actually create, and where? A: On the target cluster it creates a argocd-manager ServiceAccount in kube-system, a full-access argocd-manager-role ClusterRole, and a ClusterRoleBinding; on Kubernetes 1.24+ it also creates a token Secret to obtain a long-lived bearer token. On the hub it writes a cluster Secret containing that token and the API server CA. The identity created on the spoke is cluster-admin by default.

Q: Why is a static ServiceAccount bearer token a security concern, and what is the alternative? A: It is long-lived and cannot be rotated without re-registering, and it is cluster-admin on the spoke — if the hub is compromised, the token is a full key to the managed cluster. The alternative is an execProviderConfig (or native awsAuthConfig for EKS): a plugin mints a short-lived token per request from the cloud’s IAM, so nothing long-lived is stored and access is governed by cloud identity.

Q: Walk through how the hub authenticates to an EKS spoke. A: The hub’s pods carry an AWS IAM identity via IRSA or EKS Pod Identity. Either Argo CD’s native awsAuthConfig (with clusterName and optional roleARN) calls STS directly, or an execProviderConfig runs aws eks get-token, producing a presigned STS token valid ~15 minutes. The EKS API server validates it, and authorization comes from mapping that IAM role in the aws-auth ConfigMap or an EKS access entry to Kubernetes groups.

Q: How does AKS authentication differ from EKS? A: AKS uses Microsoft Entra ID (AAD), not IAM. The exec plugin is kubelogin, which exchanges an Azure identity (Workload Identity or a managed identity via --login workloadidentity/msi) for an AKS token, with --server-id set to the well-known AKS AAD server app ID. Authorization is Kubernetes RBAC bound to the Entra object ID. Tokens last ~1 hour versus EKS’s ~15 minutes.

Q: What is special about GKE’s auth, and what is the common failure? A: GKE uses gke-gcloud-auth-plugin, which authenticates via Application Default Credentials — typically GKE Workload Identity mapping the Argo CD KSA to a Google service account. The common failure is a valid ADC token that is then denied, because the Google service account was never bound to Kubernetes RBAC on the spoke — authentication succeeded but authorization was never granted.

Q: An exec-provider cluster shows executable file not found in $PATH. Why, and how do you fix it? A: The plugin binary (kubelogin, aws, or gke-gcloud-auth-plugin) is not present in the Argo CD application-controller/repo-server pods — your laptop has it, the pods do not. Fix it by baking the binary into a custom Argo CD image, or using an init container that copies it onto a shared PATH volume.

Q: What is the difference between destination.name and destination.server, and can you set both? A: destination.server matches the cluster Secret’s server (the API URL) exactly; destination.name matches the Secret’s name field. You set exactly one — specifying both is a validation error. Prefer name for hand-written apps (readable and survives an API-URL change) and server in generated Applications where the URL is already in hand.

Q: A newly registered cluster shows STATUS Unknown. Is that a problem? A: Not by itself. Argo CD only probes a cluster when an Application targets it, so a spoke with no apps sits at Unknown. Point an app at it (or run argocd cluster get <server>) to force a connection; if it then goes Successful you are fine, and if it goes Failed you read the MESSAGE to tell a credentials problem from a network timeout.

Q: How do cluster labels connect to multi-cluster deployment at scale? A: Labels on the cluster Secret’s metadata are read by the ApplicationSet cluster generator’s matchLabels selector. Label every cluster consistently at registration (environment, cloud, region), and one ApplicationSet fans an app to every matching cluster automatically — adding a spoke with the right labels makes the apps appear with no new manifests.

Q: You registered a spoke but sync fails with dial tcp: i/o timeout. Credentials or networking? A: Networking. An i/o timeout means the hub could not open a TCP connection to the spoke’s API server — the credential was never even tried. The cluster likely has a private API endpoint or restricted authorized networks; you need a network path (VNet/VPC peering, Private Link, authorized-networks entry, or a proxyUrl). An auth problem instead shows Unauthenticated or forbidden.

---

Glossary

• Hub — the one cluster where Argo CD’s control plane (API server, repo-server, application-controller, Redis) runs. It holds every Application and reconciles them outward.
• Spoke — a remote/managed cluster the hub deploys to, represented entirely by one cluster Secret.
• Cluster Secret — the Kubernetes Secret in the argocd namespace, labelled argocd.argoproj.io/secret-type: cluster, that is a registered cluster: its name, server, and config.
• argocd.argoproj.io/secret-type: cluster — the mandatory label that makes Argo CD treat a Secret as a cluster (the sibling value repository marks repo credentials). No label, no cluster.
• config JSON — the string field inside the cluster Secret that describes how to authenticate to the spoke; it deserializes to a Kubernetes rest.Config.
• bearerToken — a static token (usually a ServiceAccount token) in config; long-lived and non-rotating, the simplest and least secure credential.
• tlsClientConfig — the config block holding TLS settings: caData (CA to trust), optional certData/keyData (client cert), and insecure.
• execProviderConfig — a config block that runs an external plugin to mint a short-lived token on every request; the cloud-native, no-static-token path.
• awsAuthConfig — EKS-only config block letting Argo CD call AWS STS itself (with clusterName and optional roleARN) — no exec binary needed.
• in-cluster — the always-present built-in destination for the hub’s own cluster, server https://kubernetes.default.svc, backed by no Secret.
• Managed cluster — any registered spoke Argo CD can target.
• Destination — an Application’s spec.destination; selects the target cluster by name or server (exactly one) plus a namespace.
• argocd-manager — the ServiceAccount (default in kube-system) that argocd cluster add creates on the spoke as the hub’s identity, bound to a full-access ClusterRole by default.
• kubelogin — the exec plugin that exchanges a Microsoft Entra ID identity for an AKS Kubernetes token.
• aws eks get-token — the AWS CLI subcommand (an exec plugin) that returns a presigned STS token EKS accepts, valid ~15 minutes.
• gke-gcloud-auth-plugin — GKE’s exec plugin (kubectl 1.26+) that turns Application Default Credentials into a Google OAuth2 token.
• IRSA — IAM Roles for Service Accounts; binds an AWS IAM role to a Kubernetes ServiceAccount via OIDC, giving the hub pods an AWS identity.
• EKS Pod Identity — the newer AWS mechanism (an alternative to IRSA) for attaching an IAM role to pods.
• Workload Identity — the equivalent federation on Azure (Entra Workload ID) and GKE (KSA to GSA) that gives hub pods a cloud identity with no stored keys.
• ADC (Application Default Credentials) — the ambient Google credential chain gke-gcloud-auth-plugin reads to authenticate.
• aws-auth / access entry — the two ways to map an AWS IAM identity to Kubernetes RBAC groups on an EKS spoke (the legacy ConfigMap versus the modern access entry).
• ApplicationSet cluster generator — the generator that reads cluster Secret labels and stamps one Application per matching cluster, filling {{.name}}/{{.server}}.
• Namespaced (scoped) cluster — a registered cluster restricted by the Secret’s namespaces key to a fixed list of namespaces.
• Project-scoped cluster — a cluster bound by the Secret’s project key so only one AppProject’s Applications may target it.
• Controller sharding — splitting registered clusters across multiple application-controller replicas so one hub can carry a large fleet; the cluster is the unit of sharding.

---

Key takeaways

• Hub-and-spoke is the standard enterprise topology: one Argo CD reconciles Applications onto many clusters, giving one control plane and one pane of glass — at the cost of blast radius and concentrated credentials.
• A cluster is just a Secret. Every spoke is a Secret in the argocd namespace labelled argocd.argoproj.io/secret-type: cluster, holding name, server, and a config JSON that says how to authenticate. The hub always has the built-in in-cluster destination for itself.
• Two ways to register. argocd cluster add <context> is fast but imperative — it creates a full-access argocd-manager ServiceAccount on the spoke and stores its static token. Hand-writing the cluster Secret is the GitOps way: reviewable, reproducible, and able to use exec providers.
• Static tokens are a liability; exec providers are the cloud answer. An execProviderConfig (or native awsAuthConfig) mints a short-lived token per request from the cloud’s IAM, so nothing long-lived is stored and access follows cloud identity.
• The multi-cloud edge is authentication. AKS uses Entra ID via kubelogin; EKS uses IAM via aws eks get-token or native awsAuthConfig; GKE uses Google IAM via gke-gcloud-auth-plugin. Each needs the hub’s own identity (Workload Identity / IRSA / Pod Identity) granted RBAC on the spoke — authentication and authorization are separate grants and separate failures.
• Target with destination. Set name (the Secret’s name) or server (the API URL), never both. Fan one app to many clusters with the ApplicationSet cluster generator selecting on the labels you set at registration.
• Credentials are not connectivity. A valid Secret still fails if the hub cannot reach the spoke’s API server; private endpoints need peering/Private Link/authorized networks. Unauthenticated/forbidden is auth; i/o timeout is the network.
• Put the exec plugin in the image. kubelogin, aws, and gke-gcloud-auth-plugin must exist inside the Argo CD pods, not just on your laptop, or every exec-provider cluster fails with executable file not found.