Argo CD on Azure AKS: Entra ID SSO, Key Vault Secrets, ACR, Workload Identity & Application Gateway Ingress

In a nutshell

Argo CD is the robot that keeps your Kubernetes cluster matching what is written in Git. This lesson is about the plumbing that connects that robot to Azure specifically — how it proves who it is, where it reads its passwords, which registry it pulls from, and how a human opens its web page. The robot is the same on every cloud; the plumbing is what changes, and Azure’s plumbing has a few pipes that trip up almost everyone.

Think of it like moving a team into a serviced office building. The tenant (Argo CD) is identical no matter which building you rent. But each building has its own key-card system (Azure Workload Identity — you get a badge that is re-issued every hour, never a key you can lose), its own locked supply cupboard (Key Vault, opened by that badge), its own loading dock (Azure Container Registry), and its own front desk that checks staff IDs (Entra ID SSO). Get one badge wrong and nothing else works — which is exactly why this lesson spends most of its time on identity.

Level: Advanced · Time: ~52 min

Before you start, be comfortable with core Argo CD (the Application CRD and the sync loop), basic Kubernetes (ServiceAccounts, Secrets, Ingress) and the az CLI. After this lesson you will be able to turn on Workload Identity, federate a ServiceAccount so no password is ever stored, read Key Vault through External Secrets Operator, pull images and OCI charts from ACR, log humans in with Entra groups, expose the UI through Application Gateway, and — most valuably — read the specific error each of those throws when it is wired wrong.

---

Argo CD itself is cloud-agnostic — the same Application CRD, the same reconcile loop, the same argocd app sync on AKS, EKS and GKE. What is not agnostic is everything the control plane touches at its edges: how it proves its identity to the cloud, where its secrets live, which registry serves its charts and images, and how a human reaches the UI. Those edges are where a “works on my kind cluster” install turns into a production platform — and where most of the failures live.

This lesson wires Argo CD into Azure end to end. The spine is Azure Workload Identity: a federation that lets Argo CD’s components obtain Microsoft Entra tokens with no stored secret at all — the modern replacement for aad-pod-identity and for pasting a service-principal password into a Secret. Hang everything else off that spine: Entra ID for SSO, Azure Key Vault for secrets (via External Secrets Operator), Azure Container Registry (ACR) for images and OCI Helm charts, and the Application Gateway Ingress Controller (AGIC) for the UI. Because you will meet the same shapes on the other two clouds, each edge carries a short AKS vs EKS vs GKE contrast so the mapping is one glance away. The companion Argo CD on Amazon EKS lesson does the AWS side in the same depth.

No live cluster is assumed. Every manifest, az command and Terraform block below is schema-correct with real fields and flags, outputs are representative (labelled as such), and every secret is a placeholder. Pin versions to what you actually run; the shapes do not change.

---

Why this matters

You can kubectl apply the Argo CD install manifest onto AKS in two minutes and log in with the bootstrap admin password. That is a demo, not a platform. A demo has a shared admin password (no audit trail), plaintext secrets in Git or in Secret YAML, imagePullSecrets copied between namespaces, a kubectl port-forward for access, and a service principal whose client secret is one screenshot away from a breach. Each of those is a cloud edge you have not wired yet.

The Azure “grown-up” version replaces every one of those with a managed primitive. Instead of a shared password, Entra ID SSO with your real corporate groups. Instead of secrets in Git, Key Vault read through a workload identity. Instead of pull secrets, the kubelet identity’s AcrPull on ACR. Instead of port-forward, an Application Gateway ingress with TLS. And underpinning the lot, Workload Identity so none of those integrations needs a stored credential. Here is the before/after that frames the whole lesson:

The through-line is remove the stored credential. Once Workload Identity is in place, Key Vault, ACR and cluster registration all stop needing passwords, and your blast radius on a Git leak drops to “references, no values.” That is the mental model: identity first, then everything the identity unlocks.

This is the Azure sibling of the multi-cloud “wire Argo CD into the platform” family. If you have already read the EKS lesson, you will recognise the structure exactly — only the service names and three key annotations change. The contrast tables in each section make that swap explicit.

---

The Azure wiring at a glance

Before the details, hold the whole picture in your head. Read it left to right: the AKS cluster runs Argo CD; you annotate a ServiceAccount so a federated credential trusts it; Entra mints a short-lived token; with that token the repo-server reaches ACR and External Secrets Operator reaches Key Vault; AGIC exposes the UI and Entra is also the SSO identity provider. Not one client secret sits anywhere in that chain.

The numbered points are the ones that cost people the most hours: the ServiceAccount annotation and pod label that make Workload Identity actually inject a token (1), the federated credential whose subject must match the SA exactly (2), the Entra groups claim that arrives as GUIDs (3), the read-only Key Vault role ESO needs (4), the split between image pulls and chart pulls on ACR (5), and the App Gateway that fronts and TLS-terminates the UI (6). Everything below is those six boxes in depth.

---

Installing Argo CD on AKS

First you need a kubeconfig context that points at the cluster. az aks get-credentials merges cluster credentials into ~/.kube/config. The three clouds each have their own credential-fetch command, and this is the first place a script written for one cloud breaks on another:

On an Entra-integrated AKS cluster the context uses an exec plugin (kubelogin) rather than a static token, so the first kubectl call opens a browser or device-code prompt. This matters later: when this same cluster is registered as a spoke into a hub Argo CD, that exec-based kubeconfig is exactly what has to be translated into a cluster Secret.

# Point kubectl at the AKS cluster (Entra-integrated; uses kubelogin under the hood)
az aks get-credentials --resource-group rg-platform --name aks-platform
# Merged "aks-platform" as current context in ~/.kube/config

kubectl get nodes
# NAME                              STATUS   ROLES   AGE   VERSION
# aks-sys-31840americ-vmss000000    Ready    <none>  9d    v1.30.4

High availability is not optional in production

Argo CD is not one process. The install lays down several components, and the difference between the demo and an HA deployment is how many replicas each runs and whether Redis is highly available. The pieces you will scale:

You have three realistic install routes. Pick one and stay declarative:

The HA raw-manifest path, pinned to the version you run:

kubectl create namespace argocd

# Pin to the exact version you run — never "stable"
kubectl apply -n argocd \
  -f https://raw.githubusercontent.com/argoproj/argo-cd/v2.13.0/manifests/ha/install.yaml
# customresourcedefinition.apiextensions.k8s.io/applications.argoproj.io created
# statefulset.apps/argocd-application-controller created
# deployment.apps/argocd-repo-server created
# ... (HA manifests set redis-ha and multiple replicas)

kubectl -n argocd rollout status deploy/argocd-server
# deployment "argocd-server" successfully rolled out

Or the Helm route, where HA and replica counts are values:

helm repo add argo https://argoproj.github.io/argo-helm
helm upgrade --install argocd argo/argo-cd \
  --namespace argocd --create-namespace \
  --set redis-ha.enabled=true \
  --set controller.replicas=1 \
  --set server.replicas=2 \
  --set repoServer.replicas=2 \
  --set applicationSet.replicas=2 \
  --set configs.params."server\.insecure"=true   # TLS terminates at the ingress

The server.insecure=true there is deliberate and worth flagging now: argocd-server normally serves TLS and does its own HTTP→HTTPS redirect, which fights an ingress that also terminates TLS. Running it --insecure (plain HTTP inside the cluster, TLS at the gateway) is the standard pattern behind AGIC or nginx. We return to it under ingress.

First login uses a generated bootstrap password stored in a Secret — your very first act should be to wire SSO and delete it:

# The bootstrap admin password
argocd admin initial-password -n argocd
# ph7Fk2mQx9LtV0Zn
#  (equivalently: kubectl -n argocd get secret argocd-initial-admin-secret \
#     -o jsonpath="{.data.password}" | base64 -d)

argocd login argocd.example.com --username admin --password 'ph7Fk2mQx9LtV0Zn'

The full install matrix — Redis HA, resource requests, TLS options and the first-login walkthrough — lives in Installing Argo CD: Helm, manifests, HA & first login. Here we assume it is up and move to the Azure edges.

---

Azure Workload Identity: secretless tokens for Argo CD

This is the spine, so we go slowly. The problem it solves: several Argo CD components need to call Azure APIs. External Secrets Operator must read Key Vault. The repo-server may need an Entra token to pull from ACR. The old answers were all bad — a service-principal client secret in a Kubernetes Secret (long-lived, leakable, a rotation chore), or aad-pod-identity (a per-node daemon intercepting IMDS, now deprecated). Azure Workload Identity replaces both with an OIDC federation and no stored credential.

Here is the mechanism in one paragraph. Every AKS cluster can expose an OIDC issuer — a public HTTPS endpoint publishing the cluster’s token-signing keys. Kubernetes already projects a signed, short-lived token into each pod for its ServiceAccount. Workload Identity teaches Entra to trust those SA tokens for a specific ServiceAccount: you create a federated identity credential on a managed identity that says “a token from this cluster’s issuer, for this exact ServiceAccount, may be exchanged for one of my Azure tokens.” A pod presents its projected SA token, Entra validates the federation and returns an Azure access token. Nothing is stored; the trust is the credential.

Four moving parts must line up, and every Workload Identity failure is one of them being wrong:

Step one — enable the issuer and the webhook on the cluster. These flags turn on the OIDC issuer and install the mutating admission webhook that injects tokens:

az aks update --resource-group rg-platform --name aks-platform \
  --enable-oidc-issuer \
  --enable-workload-identity

# Capture the issuer URL — the federation's "issuer" field
export OIDC_ISSUER="$(az aks show -g rg-platform -n aks-platform \
  --query oidcIssuerProfile.issuerUrl -o tsv)"
echo "$OIDC_ISSUER"
# https://westeurope.oic.prod-aks.azure.com/<tenant-guid>/<cluster-guid>/

Step two — create the managed identity that External Secrets Operator (and, if you go that route, the repo-server) will assume:

az identity create --resource-group rg-platform --name id-argocd-eso

export MI_CLIENT_ID="$(az identity show -g rg-platform -n id-argocd-eso --query clientId -o tsv)"
export MI_PRINCIPAL_ID="$(az identity show -g rg-platform -n id-argocd-eso --query principalId -o tsv)"
# clientId    -> goes on the ServiceAccount annotation
# principalId -> gets the Azure role assignments (Key Vault, ACR)

Step three — create the federated credential, the trust itself. Note the subject: it names the exact namespace and ServiceAccount, and it must match byte-for-byte or Entra refuses the exchange:

az identity federated-credential create \
  --name fic-eso \
  --identity-name id-argocd-eso \
  --resource-group rg-platform \
  --issuer "$OIDC_ISSUER" \
  --subject "system:serviceaccount:external-secrets:external-secrets" \
  --audience "api://AzureADTokenExchange"

Those three fields are the whole trust contract. Get any one wrong and no token is issued:

Step four — annotate the ServiceAccount and label the pod. The annotation carries the MI’s client-id; the pod label is what tells the admission webhook to actually inject the token file and env vars:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: external-secrets
  namespace: external-secrets
  annotations:
    azure.workload.identity/client-id: "00000000-1111-2222-3333-444444444444"  # MI clientId
---
# On the workload's pod template (ESO Helm exposes this as a value):
# metadata.labels:
#   azure.workload.identity/use: "true"

When both are present, the webhook mutates the pod to add a projected token and four environment variables — this is what the Azure SDK inside ESO reads:

Which Argo CD components actually need an Azure identity

A useful clarifier before you over-federate: most of Argo CD needs no Azure identity at all. The server and controller only ever talk to the Kubernetes API. Only two Argo pods (plus the node kubelet, which is not an Argo pod) reach into Azure:

So you federate exactly one identity for ESO, wire one repository Secret for the repo-server, and run attach-acr once for the kubelet. That is the entire Azure-identity surface — do not annotate the server or controller.

The same pattern on EKS and GKE

The concept — federate a Kubernetes ServiceAccount to a cloud identity, no stored key — is identical on all three clouds. Only three things change: the feature’s name, the SA annotation, and one pod requirement. This table is the whole translation:

If you internalise one thing, make it this: on Azure the identity is expressed as an annotation client-id plus a pod label plus a federated credential whose subject is the SA. Miss the label and the SDK silently gets no token; miss the subject match and Entra returns AADSTS70021. Those two are the top Workload Identity tickets.

---

Entra ID SSO: App Registration, groups, RBAC

With identity solved for the machines, solve it for the humans. You never want a shared admin password in production — you want engineers logging in with their corporate Entra accounts, MFA enforced by Entra, and their group membership deciding what they can do in Argo CD. Argo CD is an OIDC client; Entra is the OIDC provider.

The Azure side is an App Registration. Its fields:

On the Argo CD side, oidc.config in argocd-cm points at the Entra v2 issuer. The client secret is $-referenced from argocd-secret — it never sits in the ConfigMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  url: "https://argocd.example.com"
  oidc.config: |
    name: Entra ID
    issuer: https://login.microsoftonline.com/<TENANT_ID>/v2.0
    clientID: <APP_CLIENT_ID>
    clientSecret: $oidc.entra.clientSecret        # key in argocd-secret
    requestedIDTokenClaims:
      groups:
        essential: true                            # ask Entra to include groups
    requestedScopes:
      - openid
      - profile
      - email

apiVersion: v1
kind: Secret
metadata:
  name: argocd-secret
  namespace: argocd
type: Opaque
stringData:
  oidc.entra.clientSecret: "<APP_CLIENT_SECRET>"   # placeholder — inject via ESO/CSI in real life

The claims Argo CD actually reads out of the Entra ID token, and where each comes from:

Now the part that generates the most “SSO works but I have no permissions” tickets: Entra emits groups as directory object-ID GUIDs, not names. So your RBAC must bind the GUID, not Platform-Admins:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-rbac-cm
  namespace: argocd
data:
  scopes: '[groups]'                # read roles from the groups claim
  policy.default: role:readonly
  policy.csv: |
    # g, <ENTRA GROUP OBJECT-ID GUID>, <role>
    g, 6c8b7f1e-9a2d-4f3b-8e21-0d5c7a1b2e34, role:admin
    g, 9f4a2c88-1b7e-4d6a-b0c3-2e8f5a9d1c40, proj:team-payments:developer

Two Entra-specific gotchas dominate here:

The per-cloud SSO shapes differ mostly in how you get groups into the token — the single most fragile part on every cloud:

The full three-cloud SSO walkthrough — Dex vs direct OIDC, decoding the ID token to confirm the claim, the Google domain-wide-delegation dance — is in Argo CD SSO: OIDC, Dex, Entra, Cognito & Google. For Azure, the summary is: v2 issuer, groups as GUIDs, prefer app-assigned groups, and confirm the claim before you ever blame RBAC.

---

Key Vault secrets with External Secrets Operator

Argo CD’s own rule is absolute: never commit a plaintext secret to Git. On Azure the clean answer is Key Vault as the backend and something in-cluster that reads it. There are two “somethings,” and choosing between them is a real decision.

External Secrets Operator (ESO) is the modern default. You commit an ExternalSecret — a reference, not a value — Argo CD syncs it, and ESO authenticates to Key Vault (using the Workload Identity you just built) and materialises a normal Kubernetes Secret. The plaintext never enters Git; a repo leak exposes only pointers.

The Secrets Store CSI driver is the alternative: it mounts secrets straight into a pod’s filesystem at runtime and can optionally sync them to a Secret. Different delivery, same identity.

ESO wired to Key Vault via Workload Identity

The ESO ServiceAccount is the one you federated above (external-secrets/external-secrets). Grant its managed identity read access on the vault — nothing more:

# RBAC-authorization vault: assign the read-only built-in role to the MI principal
az role assignment create \
  --assignee-object-id "$MI_PRINCIPAL_ID" \
  --assignee-principal-type ServicePrincipal \
  --role "Key Vault Secrets User" \
  --scope "$(az keyvault show -n kv-platform --query id -o tsv)"

Two vault authorization models exist, and picking the wrong grant is a common 403:

The SecretStore names the vault and says “authenticate with Workload Identity, using this SA”:

apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: azure-keyvault
  namespace: external-secrets
spec:
  provider:
    azurekv:
      authType: WorkloadIdentity
      vaultUrl: "https://kv-platform.vault.azure.net"
      serviceAccountRef:
        name: external-secrets          # the federated, annotated SA

The ExternalSecret declares which vault secret to pull and which Secret to create:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: db-credentials
  namespace: team-payments
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: azure-keyvault
    kind: SecretStore
  target:
    name: db-credentials                # the k8s Secret ESO will create + own
    creationPolicy: Owner
  data:
    - secretKey: password               # key inside the k8s Secret
      remoteRef:
        key: db-password                # the secret NAME in Key Vault

Argo CD and ESO can fight over the same Secret. ESO owns and rewrites the materialised Secret, so if Argo also tracks it, Argo sees the data change and flags OutOfSync forever. The fix: keep only the ExternalSecret under Argo’s management (that is the Git object), and either let ESO own the Secret entirely or add it to ignoreDifferences on /data. Never put both the ExternalSecret and a hand-written Secret for the same name in Git.

The API group is external-secrets.io/v1 on current ESO (v0.10+, GA). Older installs use external-secrets.io/v1beta1 with identical field names — check kubectl get crd externalsecrets.external-secrets.io -o jsonpath='{.spec.versions[*].name}' and match it.

The CSI alternative, briefly

If a single pod needs the secret mounted as a file, the CSI driver is lighter. Enable the add-on, then a SecretProviderClass (still using Workload Identity — clientID is the MI client-id) describes what to fetch:

az aks enable-addons --addons azure-keyvault-secrets-provider \
  -g rg-platform -n aks-platform

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: kv-db
  namespace: team-payments
spec:
  provider: azure
  parameters:
    usePodIdentity: "false"
    useVMManagedIdentity: "false"
    clientID: "00000000-1111-2222-3333-444444444444"   # MI client-id, Workload Identity
    keyvaultName: "kv-platform"
    tenantId: "<TENANT_ID>"
    objects: |
      array:
        - |
          objectName: db-password
          objectType: secret
  secretObjects:                          # optional: also sync to a k8s Secret
    - secretName: db-credentials
      type: Opaque
      data:
        - objectName: db-password
          key: password

The per-cloud secret-store mapping — same ESO, different provider block:

The complete secrets story — ESO vs Sealed Secrets vs SOPS vs Vault, refresh, rotation and the ownership war above — is in Secrets in Argo CD: Sealed Secrets, ESO, SOPS & Vault.

---

Pulling from ACR: images and OCI Helm charts

Azure Container Registry serves two very different things to Argo CD, over two different auth paths, and conflating them is a classic incident. Be precise:

1. Container images are pulled by the kubelet on the nodes when a Pod starts. This is a cluster-level concern, solved once with attach-acr.
2. OCI Helm charts are pulled by the repo-server when it renders an Application. This is a repository-credential concern, solved with a repository Secret.

Remember what Argo CD is not: it is not a CI system and it does not build images. Your pipeline builds and pushes to ACR; Argo CD only pulls. The registry auth is purely about read access.

Images: attach ACR to the cluster

attach-acr grants the cluster’s kubelet managed identity the AcrPull role on the registry. After it, Pods pull from ACR with no imagePullSecrets anywhere:

az aks update --resource-group rg-platform --name aks-platform \
  --attach-acr acrplatform
# Grants AcrPull to the kubelet identity on registry "acrplatform"

OCI Helm charts: a repository Secret for the repo-server

The repo-server does not use the kubelet identity — it is a different Pod with a different (or no) identity. To let it pull an OCI chart from ACR, register the registry as a repository. Create a scoped ACR token (never the admin user) and reference it:

# A repository-scoped, pull-only token — least privilege for the repo-server
az acr token create --registry acrplatform --name argocd-pull \
  --scope-map _repositories_pull
# credentials.passwords[].value -> use as the repository password (rotate on schedule)

apiVersion: v1
kind: Secret
metadata:
  name: acr-oci-charts
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: repository
stringData:
  name: acr-charts
  type: helm
  enableOCI: "true"
  url: acrplatform.azurecr.io
  username: argocd-pull                 # the ACR token name
  password: "<ACR_TOKEN_PASSWORD>"      # placeholder — inject via ESO from Key Vault

An Application then sources the chart by OCI URL:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: payments
  namespace: argocd
spec:
  project: team-payments
  source:
    repoURL: acrplatform.azurecr.io      # OCI registry host
    chart: payments                      # chart repo path in ACR
    targetRevision: 1.4.2                # chart version (pin it)
    helm:
      valueFiles:
        - values-prod.yaml
  destination:
    server: https://kubernetes.default.svc
    namespace: payments
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

The two ACR paths, side by side, so you never mix them up:

Per-cloud registry mapping:

---

Exposing the UI: AGIC, ingress-nginx, and internal load balancers

Now put a real front door on argocd-server. Three options on AKS, and the choice is about WAF, cost and whether the UI should be public at all.

AGIC

Enable the add-on (pointing at an existing Application Gateway or letting AKS create one), then an Ingress with AGIC annotations. Because argocd-server does its own redirect and gRPC, run it --insecure and let the gateway own TLS:

az aks enable-addons --addons ingress-appgw \
  -g rg-platform -n aks-platform \
  --appgw-id "$(az network application-gateway show -g rg-platform -n appgw-platform --query id -o tsv)"

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: argocd-server
  namespace: argocd
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
    appgw.ingress.kubernetes.io/backend-protocol: "http"        # server runs --insecure
    appgw.ingress.kubernetes.io/ssl-redirect: "true"
    appgw.ingress.kubernetes.io/appgw-ssl-certificate: "argocd-cert"  # cert on the gateway (from Key Vault)
spec:
  rules:
    - host: argocd.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: argocd-server
                port:
                  number: 80

The AGIC annotations you will actually reach for:

TLS from Key Vault. App Gateway can reference a certificate stored in Key Vault directly — you load it onto the gateway once and reference it by name in the annotation:

# Put the Key Vault cert on the Application Gateway (gateway's identity needs Key Vault access)
az network application-gateway ssl-cert create \
  --resource-group rg-platform --gateway-name appgw-platform \
  --name argocd-cert \
  --key-vault-secret-id "https://kv-platform.vault.azure.net/secrets/argocd-tls"

Internal LoadBalancer (private UI)

If Argo CD should stay off the internet, skip public ingress entirely and give argocd-server an internal LB — a private IP reachable only inside the VNet (and whatever peers into it):

apiVersion: v1
kind: Service
metadata:
  name: argocd-server-internal
  namespace: argocd
  annotations:
    service.beta.kubernetes.io/azure-load-balancer-internal: "true"
spec:
  type: LoadBalancer
  selector:
    app.kubernetes.io/name: argocd-server
  ports:
    - name: https
      port: 443
      targetPort: 8080

Private AKS and the hub-connectivity implication

If you go further and make the API server private (--enable-private-cluster), the control plane has no public endpoint — it is reachable only via a private FQDN in a private DNS zone. That is great for the cluster’s own security, but it changes cluster registration: a hub Argo CD sitting in another VNet cannot dial a private API server without VNet peering, Private Link, or private-DNS resolution into that network. The symptom is a spoke cluster stuck Failed with dial tcp ... i/o timeout even though the cluster Secret is perfect — the creds are right, the network path is missing. Planning that connectivity (peering, Private Link, authorized IP ranges) is a topic of its own; a dedicated lesson on private clusters and hub reachability covers it. For this lesson, know that private ingress (internal LB) and a private API server are two separate switches, and the second one is what breaks hub-and-spoke.

Per-cloud ingress mapping:

⚠️ Billing. An Application Gateway bills per hour plus per capacity unit, an internal or public Standard Load Balancer has an hourly + rule cost, and Private Link / private endpoints bill per hour and per GB. None of these are free-tier. For a lab, an internal LB or ingress-nginx behind a single Standard LB is the cheapest route; reserve AGIC for when you genuinely want Azure L7 + WAF, and tear the gateway down when you are done.

---

Terraform: AKS + federated identity in one place

Clicking this together in the portal is how you end up unable to rebuild it. The whole spine — cluster with OIDC issuer and Workload Identity on, the managed identity, the federated credential, and the Key Vault / ACR role assignments — is a few azurerm (v4) resources. This is the skeleton; the bootstrapping lesson covers installing Argo CD itself via helm_release and the app-of-apps root.

terraform {
  required_providers {
    azurerm = { source = "hashicorp/azurerm", version = "~> 4.0" }
  }
}

provider "azurerm" {
  features {}
}

resource "azurerm_kubernetes_cluster" "this" {
  name                = "aks-platform"
  location            = "westeurope"
  resource_group_name = "rg-platform"
  dns_prefix          = "aksplatform"

  oidc_issuer_enabled       = true      # publish the OIDC issuer
  workload_identity_enabled = true      # install the mutating webhook

  default_node_pool {
    name       = "system"
    node_count = 3
    vm_size    = "Standard_D4s_v5"
  }

  identity {
    type = "SystemAssigned"
  }
}

# The identity External Secrets Operator will assume
resource "azurerm_user_assigned_identity" "eso" {
  name                = "id-argocd-eso"
  location            = azurerm_kubernetes_cluster.this.location
  resource_group_name = "rg-platform"
}

# The trust: this SA on this cluster may become the identity above
resource "azurerm_federated_identity_credential" "eso" {
  name                = "fic-eso"
  resource_group_name = "rg-platform"
  audience            = ["api://AzureADTokenExchange"]
  issuer              = azurerm_kubernetes_cluster.this.oidc_issuer_url
  parent_id           = azurerm_user_assigned_identity.eso.id
  subject             = "system:serviceaccount:external-secrets:external-secrets"
}

# ESO reads Key Vault — read-only
resource "azurerm_role_assignment" "eso_kv" {
  scope                = azurerm_key_vault.this.id
  role_definition_name = "Key Vault Secrets User"
  principal_id         = azurerm_user_assigned_identity.eso.principal_id
}

# The kubelet identity pulls images from ACR
resource "azurerm_role_assignment" "kubelet_acr" {
  scope                = azurerm_container_registry.this.id
  role_definition_name = "AcrPull"
  principal_id         = azurerm_kubernetes_cluster.this.kubelet_identity[0].object_id
}

The resources and what each replaces:

Feed the outputs forward: azurerm_user_assigned_identity.eso.client_id becomes the ServiceAccount annotation, and azurerm_kubernetes_cluster.this.oidc_issuer_url is what the federated credential trusts. Installing Argo CD on top — the helm_release, the root app-of-apps, ordering the bootstrap so CRDs exist before the Applications — is the subject of the dedicated bootstrapping lesson.

---

Hands-on lab: wire it end to end

No live cluster is required — this lab is config-level: you produce every manifest and command in the correct order, exactly as you would apply them, and you verify shape and cross-references rather than a running sync. That is the honest way to learn the wiring without an Azure bill. Where a step would bill on a real cluster, it is flagged. Follow it as a build sheet.

Before you start, know what would cost money if this were live, so you reach for the cheap option in a lab:

Scenario. One AKS cluster runs Argo CD. You will: (1) turn on Workload Identity, (2) federate an identity for ESO, (3) deploy a SecretStore + ExternalSecret pulling a Key Vault secret, (4) configure Entra SSO with a group→role mapping, and (5) expose the UI via AGIC.

Step 1 — Turn on the OIDC issuer and Workload Identity.

az aks update -g rg-lab -n aks-lab --enable-oidc-issuer --enable-workload-identity
export OIDC_ISSUER="$(az aks show -g rg-lab -n aks-lab --query oidcIssuerProfile.issuerUrl -o tsv)"

What just happened: the cluster now publishes an OIDC issuer and runs the mutating webhook. Nothing bills yet — these are cluster feature flags. $OIDC_ISSUER is the string the federation will trust.

Step 2 — Create the identity and federate it to the ESO ServiceAccount.

az identity create -g rg-lab -n id-eso-lab
export MI_CLIENT_ID="$(az identity show -g rg-lab -n id-eso-lab --query clientId -o tsv)"
export MI_PRINCIPAL_ID="$(az identity show -g rg-lab -n id-eso-lab --query principalId -o tsv)"

az identity federated-credential create \
  --name fic-eso-lab --identity-name id-eso-lab -g rg-lab \
  --issuer "$OIDC_ISSUER" \
  --subject "system:serviceaccount:external-secrets:external-secrets" \
  --audience "api://AzureADTokenExchange"

What just happened: Entra now trusts the external-secrets ServiceAccount in the external-secrets namespace. The subject string is the load-bearing part — it must equal the SA’s system:serviceaccount:<ns>:<name> exactly.

Step 3 — Grant read on Key Vault and seed a test secret.

az keyvault secret set --vault-name kv-lab --name db-password --value 'S3cr3t-in-Vault'   # ⚠️ demo value
az role assignment create \
  --assignee-object-id "$MI_PRINCIPAL_ID" --assignee-principal-type ServicePrincipal \
  --role "Key Vault Secrets User" \
  --scope "$(az keyvault show -n kv-lab --query id -o tsv)"

What just happened: the managed identity can now read secrets from kv-lab and nothing else. The secret value lives in Key Vault — it will never appear in Git.

Step 4 — Annotate the ESO ServiceAccount (the manifest ESO’s Helm chart renders).

apiVersion: v1
kind: ServiceAccount
metadata:
  name: external-secrets
  namespace: external-secrets
  annotations:
    azure.workload.identity/client-id: "REPLACE_WITH_MI_CLIENT_ID"
# The ESO pod template must also carry the label:
#   azure.workload.identity/use: "true"

What just happened: the annotation ties the SA to the managed identity; the pod label makes the webhook inject the token. Install ESO with its Helm chart and set both via values (serviceAccount.annotations and podLabels).

Step 5 — Deploy the SecretStore and ExternalSecret.

apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: azure-keyvault
  namespace: external-secrets
spec:
  provider:
    azurekv:
      authType: WorkloadIdentity
      vaultUrl: "https://kv-lab.vault.azure.net"
      serviceAccountRef:
        name: external-secrets
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: db-credentials
  namespace: external-secrets
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: azure-keyvault
    kind: SecretStore
  target:
    name: db-credentials
    creationPolicy: Owner
  data:
    - secretKey: password
      remoteRef:
        key: db-password

What just happened: on a live cluster, ESO would authenticate with the federated token, read db-password from Key Vault, and create a db-credentials Secret. Representative verification output:

kubectl -n external-secrets get externalsecret db-credentials
# NAME             STORE            REFRESH INTERVAL   STATUS         READY
# db-credentials   azure-keyvault   1h                 SecretSynced   True

kubectl -n external-secrets get secret db-credentials -o jsonpath='{.data.password}' | base64 -d
# S3cr3t-in-Vault   (materialized from Key Vault; never in Git)

Step 6 — Configure Entra SSO with a group→role mapping. Apply the argocd-cm oidc.config, the argocd-secret client-secret, and the RBAC binding shown earlier. The load-bearing line:

# argocd-rbac-cm data.policy.csv
g, 6c8b7f1e-9a2d-4f3b-8e21-0d5c7a1b2e34, role:admin   # your Platform-Admins group OBJECT-ID

What just happened: members of that Entra group become Argo CD admins on next login; everyone else falls to role:readonly. Confirm the group object-id with az ad group show --group Platform-Admins --query id -o tsv.

Step 7 — Expose the UI via AGIC. Apply the Ingress from the ingress section (server --insecure, TLS cert on the gateway). Representative result:

kubectl -n argocd get ingress argocd-server
# NAME            CLASS   HOSTS                ADDRESS         PORTS   AGE
# argocd-server   <none>  argocd.example.com   20.50.x.x       80      2m

What just happened: AGIC programmed the Application Gateway with a listener for argocd.example.com and a backend pool of argocd-server pods. ⚠️ This step bills on a real cluster — the Application Gateway is live from the moment the add-on provisions it.

Verify (the shape you expect):

argocd app list          # Applications visible after SSO login
kubectl -n external-secrets get secretstore azure-keyvault -o jsonpath='{.status.conditions[0].reason}'
# ValidationSucceeded      <- Workload Identity auth to Key Vault worked

Teardown. Reverse order, and delete the billing resources first:

kubectl -n argocd delete ingress argocd-server                        # stop AGIC programming
az aks disable-addons --addons ingress-appgw -g rg-lab -n aks-lab      # ⚠️ stop App Gateway billing
az network application-gateway delete -g rg-lab -n appgw-lab           # if AKS created it
kubectl delete -n external-secrets externalsecret db-credentials secretstore azure-keyvault
az identity federated-credential delete --name fic-eso-lab --identity-name id-eso-lab -g rg-lab
az identity delete -g rg-lab -n id-eso-lab
az keyvault secret delete --vault-name kv-lab --name db-password
# Delete the resource group last if the lab is disposable: az group delete -n rg-lab

What just happened: you removed the billing surfaces (App Gateway, LB) first, then the identity federation and the test secret. The cluster feature flags (OIDC issuer, Workload Identity) cost nothing and can stay.

---

Going deeper

The body above wires the happy path. This section is the layer underneath — the facts that decide whether the design survives a second cluster, a sovereign region, a rotated secret, or a 300-group user. None of it re-teaches the setup; each item is a gap the walkthrough deliberately left for here.

aad-pod-identity is not “deprecated,” it is gone

The lesson calls the old approach deprecated; be blunter in a design review. The open-source aad-pod-identity project is archived — read-only, no fixes — and the AKS pod-managed identities add-on that wrapped it is on the retirement path. There is no supported migration target except Workload Identity. If you inherit a cluster still running the aad-pod-identity NMI DaemonSet intercepting IMDS on every node, treat that as a live risk, not a working system: it can wedge node IMDS traffic and has no security backports. The migration is exactly the four moving parts from the Workload Identity section — issuer, identity, federated credential, SA annotation + pod label — after which you delete the NMI DaemonSet and the AzureIdentity / AzureIdentityBinding CRs.

App roles beat the groups claim for Argo RBAC

The body’s “groups arrive as GUIDs, and vanish past ~200 groups” pain has a cleaner fix than juggling GUIDs: Entra app roles. Instead of the groups claim, define app roles on the App Registration (values you choose, e.g. argocd-admin, payments-dev), assign users or groups to them, and Entra emits a roles claim of those string values — never GUIDs, and not subject to the groups-overage limit. Point Argo at that claim instead of groups:

# argocd-rbac-cm
data:
  scopes: '[roles]'                     # match on the roles claim, not groups
  policy.default: role:readonly
  policy.csv: |
    g, argocd-admin, role:admin         # a role VALUE — human-readable, stable
    g, payments-dev, proj:team-payments:developer

You trade a little Entra admin (defining and assigning the roles) for RBAC lines a human can actually read and review, and you sidestep the overage cliff entirely. Groups are still right when you must reuse existing corporate groups verbatim; app roles are better when Argo authorization is its own concern. The full role-vs-group trade-off is in Argo CD RBAC: local users & policies.

The third ACR consumer: Argo CD Image Updater

The body names two ACR consumers — the kubelet (images) and the repo-server (OCI charts). If you run Argo CD Image Updater to bump tags automatically, that is a third pod reaching ACR, and it needs its own read path: it polls the registry’s tag list, which neither the kubelet identity nor the repo-server’s repository Secret grants it. Give Image Updater either its own scoped ACR token (a pull secret it references in registries.conf) or its own federated ServiceAccount — do not assume it inherits the repo-server’s credential. Miss this and syncs keep working while tag detection silently stalls. The registry-by-registry setup is in Argo CD Image Updater on ACR, ECR & Artifact Registry.

App Gateway for Containers: the Gateway-API successor to AGIC

AGIC programs a classic Application Gateway from the old Ingress API. Its strategic successor is Application Gateway for Containers (AGfC), which implements the Kubernetes Gateway API (gateway.networking.k8s.io) through the ALB Controller rather than Ingress annotations. The same argocd-server front door, expressed as Gateway API:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: argocd
  namespace: argocd
spec:
  gatewayClassName: azure-alb-external      # AGfC's gateway class (azure-alb-internal for private)
  listeners:
    - name: https
      protocol: HTTPS
      port: 443
      tls:
        mode: Terminate
        certificateRefs:
          - kind: Secret
            name: argocd-tls
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: argocd-server
  namespace: argocd
spec:
  parentRefs:
    - name: argocd
  hostnames:
    - argocd.example.com
  rules:
    - backendRefs:
        - name: argocd-server
          port: 80                          # server still runs --insecure

Why care: Gateway API cleanly separates the infra owner’s Gateway from the app owner’s HTTPRoute (better for multi-tenant platforms), and the --insecure backend trick from AGIC still applies. AGIC is not dead, but new Azure L7 designs should evaluate AGfC first. Both front the same server; only the routing API changes.

CSI secret rotation is off until you turn it on

The body shows the CSI driver mounting a Key Vault secret, but a mounted secret is a snapshot: rotate the value in Key Vault and the pod keeps the stale copy until it restarts — unless rotation is enabled. On the AKS add-on it is a switch, not a default:

az aks addon update -g rg-platform -n aks-platform \
  -a azure-keyvault-secrets-provider \
  --enable-secret-rotation --rotation-poll-interval 5m

Even then, mind the boundary: the driver refreshes the mounted file and the synced Secret on its poll interval, but a process that read the value into memory at startup will not see the new value until it re-reads or restarts. ESO has the same caveat via refreshInterval. Rotation gets the new bytes onto the node; making the app use them is still the app’s job — often a rolling restart via a checksum annotation.

Multi-cluster, sovereign clouds and the FIC ceiling

Two edges bite once you scale past one cluster in one commercial region:

• Federated-credential ceiling. A single user-assigned managed identity holds up to 20 federated identity credentials. Reuse one identity across many clusters and namespaces and you will hit that ceiling; past it, prefer one identity per cluster (or per team) over one global identity. Each (issuer, subject) pair is one FIC.
• Sovereign clouds. In Azure Government or Azure operated by 21Vianet (China), the authority host the SDK needs is not login.microsoftonline.com. Workload Identity injects AZURE_AUTHORITY_HOST for you, but hard-coded endpoints in your own tooling (login.microsoftonline.us for Gov, login.chinacloudapi.cn for China) are a classic “works in commercial, 401s in Gov” bug. The cluster OIDC issuer host differs by cloud too.

Private API servers add a third scaling edge — a hub Argo CD cannot dial a private spoke without a network path — covered in Private clusters & network connectivity.

---

Common mistakes and troubleshooting

The failures here are almost never in Argo CD itself — they are in the Azure edges. This table is the one to keep open; the prose after it dissects the three nastiest.

1. The Workload Identity token that never arrives. This is the number-one ticket, and it is almost always the pod label, not the credential. People set the SA annotation, create a flawless federated credential, and forget that the pod template also needs azure.workload.identity/use: "true". Without the label the mutating webhook does nothing — no projected token file, no AZURE_* env — so the Azure SDK falls through DefaultAzureCredential and fails as if you configured nothing. Debug it by exec-ing into the pod and checking env | grep AZURE_ and ls /var/run/secrets/azure/tokens/. Empty means the webhook did not fire, which means the label (or the webhook itself, if --enable-workload-identity was skipped) is missing. The credential subject is the second thing to check, and its failure has a distinct signature: AADSTS70021.

2. Images pull but the chart 404s (or vice-versa). ACR serves images to the kubelet and charts to the repo-server over different identities, and the two failure modes look unrelated until you see the split. If Pods run but the Application is ComparisonError/failed to get repo, your images are fine (kubelet has AcrPull) but the repo-server has no repository Secret for the OCI chart. If the Application renders fine but Pods are ImagePullBackOff, it is the reverse — the chart auth is fine but attach-acr never ran. Always ask “which side is failing, the render or the pod?” before touching anything. The fix is on the matching side, never both.

3. SSO succeeds and the user has no rights. A login that works but lands on role:readonly almost always means the groups claim did not match your RBAC. Two Entra-specific causes: the claim carries GUIDs and your policy.csv used a friendly name, or the user is in more than ~200 groups and Entra replaced the groups array with an overage pointer (_claim_sources), sending no groups at all. Decode the ID token (base64-decode the middle segment of the JWT) and look for groups. If it is a list of GUIDs, fix your RBAC lines to use GUIDs. If it is absent with an overage pointer, switch the App Registration to emit only “Groups assigned to the application.” Never debug this from the Argo side first — confirm the claim, then map it.

---

Common beginner mistakes

These are wrong mental models, not wrong commands — the symptom table above fixes symptoms; this fixes the thinking that produces them.

• “Argo CD needs an Azure identity.” Mostly it does not. The server and the application-controller only ever talk to the Kubernetes API — they never call Azure. Only External Secrets Operator (Key Vault) and, if you use them, the repo-server and Image Updater (ACR) need a cloud identity. Federating the whole install “to be safe” grants blast radius for nothing. Right model: identity follows the Azure API call, and most Argo pods make none.

• “Workload Identity means I don’t set up any identity.” The name misleads. “No stored secret” is not “no configuration.” You still create a managed identity, a federated credential, an SA annotation and a pod label. What you skip is the password, not the wiring. Right model: Workload Identity replaces the credential’s value with a trust, and a trust has four parts that must line up.

• “The ServiceAccount annotation is enough.” The azure.workload.identity/client-id annotation names the identity, but the pod only gets a token if its template also carries the label azure.workload.identity/use: "true". Annotation without label is the single most common Workload Identity ticket: everything looks configured and no token appears. Right model: the annotation says who, the label says inject it — you need both.

• “ACR is one integration.” It is two, on two identities. --attach-acr fixes image pulls (kubelet); it does nothing for OCI chart pulls (repo-server), which need a repository Secret. Beginners run attach-acr, see images work, and are baffled when a Helm-OCI Application still 404s. Right model: ask “is the render failing or the pod failing?” — they have different fixes.

• “TLS everywhere is safer, so keep argocd-server on HTTPS behind the gateway.” Double TLS termination plus the server’s own HTTP→HTTPS redirect is what breaks the CLI and UI behind AGIC. The correct production pattern is argocd-server --insecure (plain HTTP inside the cluster) with TLS terminated at the gateway. Right model: terminate TLS once, at the edge — the in-cluster hop is already inside your trust boundary.

• “SSO logged me in, so RBAC is set up.” Authentication (who you are) and authorization (what you may do) are separate. A perfect Entra login still lands you on role:readonly if the groups / roles claim never matches a policy.csv line. Right model: after SSO works, decode the ID token and confirm the claim before you touch RBAC — most “SSO is broken” tickets are really “the claim never matched.”

• “Private ingress makes my cluster private.” An internal LoadBalancer privatises only the UI; the Kubernetes API server is still public unless you also set --enable-private-cluster. The two are independent switches, and only the second affects hub-and-spoke reachability. Right model: UI exposure and API-server exposure are different doors with different keys.

---

Practice challenges

Work these in order — each builds the muscle the next assumes. No live cluster needed; every answer is a manifest, a command, or a diagnosis you can reason to on paper.

1 (Beginner) — Read the identity surface. Without scrolling up, name which of these five make an Azure API call and therefore need an identity: argocd-server, argocd-application-controller, argocd-repo-server, External Secrets Operator, the node kubelet.

<details><summary>Solution</summary>

ESO (Key Vault), the repo-server (ACR OCI charts) and the kubelet (ACR images) call Azure. argocd-server and argocd-application-controller talk only to the Kubernetes API and need no Azure identity.

Why: over-federating the server/controller is a real review smell — identity should follow an actual Azure call. </details>

2 (Beginner) — Spot the missing pod label. A teammate’s ESO pod fails with DefaultAzureCredential ... no credential. The ServiceAccount has azure.workload.identity/client-id set and the federated credential exists. What is almost certainly missing, and how do you confirm it from inside the pod?

<details><summary>Solution</summary>

The pod template is missing the label azure.workload.identity/use: "true", so the webhook injected nothing. Confirm with kubectl exec then env | grep AZURE_ and ls /var/run/secrets/azure/tokens/ — both empty means no injection.

Why: the annotation names the identity; the label triggers injection. Beginners set the first and forget the second. </details>

3 (Intermediate) — Grant ESO the least role. Write the az command that lets ESO’s managed identity read (only read) secrets from an RBAC-enabled Key Vault named kv-platform, given its principal id in $MI_PRINCIPAL_ID.

<details><summary>Solution</summary>

az role assignment create \
  --assignee-object-id "$MI_PRINCIPAL_ID" --assignee-principal-type ServicePrincipal \
  --role "Key Vault Secrets User" \
  --scope "$(az keyvault show -n kv-platform --query id -o tsv)"

Why: Key Vault Secrets User is read-only on secrets; Contributor or Key Vault Administrator would be over-privileged and would show up as such in an audit. </details>

4 (Intermediate) — Fix the OutOfSync loop. Argo CD tracks an ExternalSecret and the Secret ESO creates from it. The app is stuck OutOfSync forever on that Secret. What is happening, and what are the two clean fixes?

<details><summary>Solution</summary>

ESO owns and rewrites the materialised Secret, so Argo sees drift on /data every reconcile. Fix by (a) tracking only the ExternalSecret in Git and letting ESO own the Secret, or (b) adding the Secret’s /data to ignoreDifferences on the Application.

Why: two controllers writing one object always fight; the ExternalSecret is the Git object, the Secret is ESO’s output. </details>

5 (Advanced) — Make groups overage-proof. A director’s account is in 300+ Entra groups and lands on role:readonly because the groups claim was dropped. Redesign the mapping so it is immune to overage, and give the argocd-rbac-cm lines.

<details><summary>Solution</summary>

Switch from the groups claim to app roles: define app roles (e.g. argocd-admin) on the App Registration, assign the group to the role, and match the roles claim:

# argocd-rbac-cm
scopes: '[roles]'
policy.csv: |
  g, argocd-admin, role:admin

Why: app roles emit human-readable string values and are not subject to the >200-group overage that silently strips groups. </details>

6 (Advanced) — Diagnose the perfect-creds hub failure. A private AKS spoke is registered into a hub Argo CD in another VNet. The cluster Secret is correct, yet the cluster shows Failed with dial tcp ... i/o timeout. Creds are verified good. What is wrong, and what class of fix applies?

<details><summary>Solution</summary>

The spoke’s API server is private (--enable-private-cluster) and the hub has no network path to it. Credentials are irrelevant to a routing failure. Fix at the network layer: VNet peering or Private Link plus private-DNS resolution of the API-server FQDN, or authorized IP ranges — not anything in Argo CD. Wiring this handoff is covered in Cluster bootstrapping & the Terraform handoff.

Why: a timeout is a path problem; an auth error is a credential problem. Reading the error class tells you which layer to touch. </details>

---

Cheat-sheet

The Azure-specific wiring, command by command:

The one-line cross-cloud map — the whole lesson compressed:

---

Interview and exam questions

Q: Why is Azure Workload Identity preferred over a service-principal client secret or aad-pod-identity for Argo CD’s Azure access? A: It removes the stored credential entirely. A client secret is long-lived, leakable and must be rotated; aad-pod-identity is a deprecated per-node daemon that intercepts IMDS. Workload Identity federates a Kubernetes ServiceAccount to a managed identity via OIDC — the pod presents its short-lived, cluster-signed SA token and Entra returns an Azure token. Nothing is stored; the trust is a federated credential, so a Git or cluster leak exposes no usable secret.

Q: A pod using Workload Identity gets no token even though the federated credential exists. What are the two things to check first? A: The pod label azure.workload.identity/use: "true" (without it the mutating webhook never injects a token) and the ServiceAccount annotation azure.workload.identity/client-id. Then confirm the federated credential’s subject equals system:serviceaccount:<ns>:<sa> exactly — a mismatch throws AADSTS70021. Check inside the pod with env | grep AZURE_ and ls /var/run/secrets/azure/tokens/.

Q: How does External Secrets Operator authenticate to Key Vault without a stored credential, and what role does it need? A: Its ServiceAccount is federated to a managed identity via Workload Identity; the SecretStore sets provider.azurekv.authType: WorkloadIdentity and references that SA. The identity needs only Key Vault Secrets User (RBAC vaults) or get/list secret access-policy permissions (legacy vaults) — read-only, least privilege.

Q: Explain the difference between how ACR serves container images and OCI Helm charts to Argo CD. A: Two paths. Container images are pulled by the kubelet on the nodes — solved cluster-wide with az aks update --attach-acr, which grants the kubelet identity AcrPull. OCI Helm charts are pulled by the argocd-repo-server, a different pod, which needs its own repository Secret (labelled argocd.argoproj.io/secret-type: repository, type: helm, enableOCI: "true") authenticated with an ACR scope-map token. Confusing the two produces “images pull but the chart 404s” (or the reverse).

Q: Why do you often run argocd-server with --insecure behind AGIC or ingress-nginx? A: argocd-server normally terminates TLS itself and issues its own HTTP→HTTPS redirect, and it serves gRPC for the CLI. Behind an ingress that also terminates TLS, that double TLS/redirect breaks the flow. Running the server --insecure (plain HTTP in-cluster) and terminating TLS at the gateway with backend-protocol: http is the standard, working pattern.

Q: An engineer logs in via Entra SSO but has only read-only access. Walk through diagnosing it. A: The groups claim did not match RBAC. Decode the ID token (base64 its middle segment) and inspect groups. If it lists GUIDs, your policy.csv used a group name — rebind with the object-ID GUID. If groups is absent with a _claim_sources overage pointer, the user is in >200 groups and Entra dropped the claim — switch the App Registration to “Groups assigned to the application.” Confirm the claim before touching Argo RBAC.

Q: What is the difference between a private ingress (internal LB) and a private AKS API server, and why does the second one matter for a hub Argo CD? A: An internal LoadBalancer gives argocd-server a private IP so the UI is not internet-facing — purely about human access. A private API server (--enable-private-cluster) removes the control plane’s public endpoint. That second switch breaks hub-and-spoke: a hub Argo CD in another VNet cannot dial a private API server without VNet peering / Private Link and private-DNS resolution. Symptom: a registered spoke stuck Failed with dial tcp ... i/o timeout despite correct credentials.

Q: Map Azure’s Workload Identity, Key Vault, ACR and AGIC to their EKS and GKE equivalents. A: Workload Identity ≈ EKS IRSA / Pod Identity ≈ GKE Workload Identity. Key Vault ≈ AWS Secrets Manager ≈ Google Secret Manager (ESO providers azurekv / aws / gcpsm). ACR ≈ ECR ≈ Artifact Registry. AGIC (Application Gateway) ≈ AWS Load Balancer Controller (ALB) ≈ GKE Ingress (GCLB). The SA annotation differs on each: azure.workload.identity/client-id, eks.amazonaws.com/role-arn, iam.gke.io/gcp-service-account.

Q: Which azurerm (v4) fields turn on Workload Identity, and how do you express the trust in Terraform? A: On azurerm_kubernetes_cluster, set oidc_issuer_enabled = true and workload_identity_enabled = true. The trust is azurerm_federated_identity_credential with issuer = azurerm_kubernetes_cluster.this.oidc_issuer_url, subject = "system:serviceaccount:<ns>:<sa>", audience = ["api://AzureADTokenExchange"], and parent_id pointing at the azurerm_user_assigned_identity. Role assignments (azurerm_role_assignment) grant that identity Key Vault Secrets User and AcrPull.

Q: When would you choose the Secrets Store CSI driver over External Secrets Operator on AKS? A: Choose CSI when a single pod needs the secret mounted as a file with the tightest possible lifetime — the secret exists only while the pod mounts the volume. Choose ESO (the default) when you want a real Kubernetes Secret that many workloads share, a clean Git object (ExternalSecret) for GitOps, and a refreshInterval. Both use the same Workload Identity; they differ only in delivery.

Q: You register an Entra-integrated AKS cluster as a spoke and sync fails with an auth error even though registration succeeded. Why? A: The cluster’s kubeconfig uses an exec plugin (kubelogin) rather than a static token, and the hub’s cluster Secret must encode that exec/credential path (and the hub’s own identity must have RBAC on the spoke). Registration writing a Secret is necessary but not sufficient — the hub still needs a valid auth mechanism and a network path to the API server.

Q: What is the least-privilege way to let the Argo CD repo-server pull OCI charts from ACR, and why not the admin user? A: Create a repository-scoped ACR token (az acr token create --scope-map _repositories_pull) — pull-only, revocable, rotatable independently — and put it in a repository Secret. The registry admin user is a single all-powerful credential that grants push and delete across the whole registry and cannot be scoped; using it for a read-only pull is a large, unnecessary blast radius.

---

Glossary

• Workload Identity (Entra) — Azure’s mechanism for giving a Kubernetes pod an Entra token with no stored password, by federating a ServiceAccount to a managed identity. The modern replacement for aad-pod-identity and for service-principal secrets.
• OIDC issuer (of a cluster) — a public HTTPS endpoint the AKS cluster publishes, exposing the keys that sign its ServiceAccount tokens. Entra trusts tokens from this issuer.
• Federated identity credential (FIC) — the trust object on a managed identity that says “a token from this issuer, for this subject, may be exchanged for my Azure token.” Its three fields are issuer, subject, audience.
• Managed identity (user-assigned) — an Azure identity you create and attach; it has a clientId (goes on the SA annotation) and a principalId (gets role assignments). No password.
• Subject (of a FIC) — the exact string system:serviceaccount:<namespace>:<sa-name>. A byte mismatch here is the AADSTS70021 error.
• azure.workload.identity/use label — the pod-template label that tells the mutating webhook to inject a token and the AZURE_* env vars. Without it, nothing is injected.
• Entra ID — Microsoft’s identity provider (formerly Azure AD). Here it is both the SSO provider for humans and the token issuer for workloads.
• App Registration — the Entra object representing Argo CD as an OIDC client: redirect URI, client secret, and the groups/roles claim configuration.
• Groups claim vs app roles — two ways Entra tells Argo CD what a user may do. groups arrives as GUIDs and breaks past ~200 groups; app roles arrive as readable strings via the roles claim and do not.
• External Secrets Operator (ESO) — an in-cluster controller that reads a secret backend (here Key Vault) and materialises a normal Kubernetes Secret from an ExternalSecret reference committed to Git.
• SecretStore / ExternalSecret — ESO’s two CRs: the store says where and how to authenticate; the ExternalSecret says which secret to fetch and what Secret to create.
• Secrets Store CSI driver — the alternative to ESO that mounts Key Vault secrets into a pod’s filesystem via a SecretProviderClass, optionally syncing to a Secret.
• ACR — Azure Container Registry. Serves images to the kubelet and OCI Helm charts to the repo-server, over two different auth paths.
• AcrPull — the least-privilege Azure role granting read (pull) on a registry; --attach-acr assigns it to the kubelet identity.
• AGIC / App Gateway for Containers — AGIC programs a classic Application Gateway from an Ingress; App Gateway for Containers (AGfC) is its Gateway-API successor driven by the ALB Controller.
• --insecure (argocd-server) — runs the API/UI as plain HTTP inside the cluster so TLS terminates once, at the gateway; the standard pattern behind AGIC / AGfC / nginx.
• Private cluster — an AKS cluster whose API server has no public endpoint (--enable-private-cluster); distinct from a private UI, and the thing that breaks hub-and-spoke reachability.

---

Key takeaways

• Identity first, then everything it unlocks. Azure Workload Identity — a federated credential trusting a ServiceAccount, plus the SA annotation and the azure.workload.identity/use: "true" pod label — is the spine. It replaces stored client secrets and deprecated aad-pod-identity, and Key Vault, ACR and cluster auth all hang off it with no password anywhere.
• The pod label and the credential subject are the top two failures. No label means the webhook injects nothing; a subject that is not exactly system:serviceaccount:<ns>:<sa> throws AADSTS70021. Check env | grep AZURE_ inside the pod before anything else.
• Key Vault via ESO, read-only. A SecretStore with authType: WorkloadIdentity lets ESO materialise a real Secret; grant only Key Vault Secrets User. The CSI driver is the mount-into-the-pod alternative for single-pod, file-based secrets. Let ESO own the Secret or Argo will fight it as OutOfSync.
• ACR is two paths, not one. --attach-acr gives the kubelet AcrPull for images; the repo-server needs its own repository Secret (a scoped ACR token) for OCI charts. “Images pull but the chart 404s” is the tell that you did one and not the other.
• Entra SSO groups arrive as GUIDs and can vanish past ~200 groups. Bind the object-ID GUID in argocd-rbac-cm, prefer “Groups assigned to the application,” and decode the ID token to confirm the claim before blaming RBAC.
• AGIC fronts the UI; run the server --insecure. Terminate TLS on the Application Gateway with a Key Vault certificate. Prefer an internal LoadBalancer or ingress-nginx when you do not need Azure L7/WAF — App Gateway and private endpoints all bill.
• Private ingress ≠ private API server. An internal LB privatises the UI; --enable-private-cluster privatises the control plane and is what breaks hub-and-spoke reachability. Plan peering/Private Link separately.
• The pattern transfers. Workload Identity ≈ IRSA/Pod Identity ≈ GKE Workload Identity; Key Vault ≈ Secrets Manager ≈ Secret Manager; ACR ≈ ECR ≈ Artifact Registry; AGIC ≈ ALB ≈ GCLB. Learn the Azure edges here and the EKS and GKE versions are a three-annotation swap.