Argo CD on AWS EKS: IRSA & Pod Identity, Secrets Manager, ECR & the AWS Load Balancer Controller

In a nutshell

Argo CD itself is the same everywhere — it reads Git and makes your cluster match what Git says. This lesson is about the four places where running it on Amazon EKS forces a conversation with AWS: proving who a pod is (identity), fetching a password from AWS instead of Git (secrets), pulling images from Amazon’s registry (registry), and putting a safe front door on the UI (networking). Each one has a clean way to do it and a tempting shortcut that quietly leaks a long-lived AWS key. The whole lesson is really one instruction, applied four times: take the clean way.

The analogy to hold: think of EKS as an office building and Argo CD as a contractor who does the same job in every building. But this building has its own badge system (identity), its own locked supply cabinet (secrets), its own parts warehouse (registry), and its own reception desk with an intercom (networking). You are not teaching the contractor a new trade — you are issuing a badge, cabinet access, a warehouse pass, and a desk phone, each scoped to exactly what they need, and none of them a master key that never expires.

Level: Advanced · Time: ~54 min

You should already have: a working Argo CD install and login (Installing Argo CD), a working grasp of IAM roles and policies, and a rough idea of how Argo CD handles secrets in general (Secrets in GitOps). After this you can: grant a pod AWS permissions with IRSA or EKS Pod Identity, read Secrets Manager through the External Secrets Operator, pull ECR images and OCI charts without a token that dies overnight, and expose the UI through an ALB where the CLI actually works.

---

Argo CD does not care which cloud it runs on. It watches Git, renders manifests, and reconciles a cluster — the same Application YAML works on a laptop kind cluster and on a fleet of production clusters. But the instant you run Argo CD on Amazon EKS and ask it to do real work, four cloud-specific edges appear, and each one is a place where teams either wire it correctly or quietly bake a long-lived AWS credential into a pod. Those four edges are identity (how does a pod get AWS permissions?), secrets (how does a pulled Secret’s value come from AWS Secrets Manager, not Git?), registry (how do images and OCI Helm charts come from ECR without a token that expires every twelve hours?), and networking (how is the UI exposed, and why does the CLI break behind a load balancer?).

This lesson wires all four, end to end, at production quality. We go deepest on the one that unlocks the rest — identity — because on EKS there are now two mechanisms (the established IRSA and the newer EKS Pod Identity), and choosing between them shapes everything downstream. Then we point that identity at Secrets Manager through the External Secrets Operator, solve the ECR twelve-hour-token problem for both images and charts, and expose the Argo CD UI through the AWS Load Balancer Controller with the --grpc-web and ACM details that trip up almost everyone. We finish with a Terraform snippet that provisions the whole thing, notes on private EKS, and — because this is a multi-cloud course — a per-edge contrast table that maps every AWS piece to its AKS and GKE equivalent, so the pattern you learn here transfers to the other two clouds you will inevitably also run.

This lesson assumes you can already install Argo CD and log in — if not, start with Installing Argo CD — and that you understand how Argo CD handles secrets in general, covered in Secrets in GitOps. Everything below is schema-correct for Argo CD 2.13+/3.x on Kubernetes 1.29+ with the AWS provider v5 idioms; account IDs, ARNs and hostnames are placeholders you replace.

---

Why this matters

Here is the trap this lesson exists to prevent. A team stands up Argo CD on EKS, needs it to read a database password, and — because it is Friday and IRSA looks like a lot of JSON — creates an IAM user, generates an access key, and drops it into a Kubernetes Secret. It works. It also means a static, long-lived AWS credential now lives in the cluster, is mounted into a pod, and will be there in a year when nobody remembers it, cannot be rotated without downtime, and is one kubectl get secret -o yaml away from an incident. Every one of the four edges has a version of this trap, and every one has a clean answer that uses short-lived, federated credentials instead.

The clean answer rests on a single idea: a pod on EKS can prove its identity to AWS without any stored secret at all. The cluster publishes an OIDC identity, AWS trusts it, and a pod’s ServiceAccount is mapped to an IAM role. The pod receives temporary credentials, scoped to exactly what it needs, refreshed automatically, and revocable by editing one IAM role. Once you have that, the other three edges fall out of it: the External Secrets Operator uses that identity to read Secrets Manager, a refresh job uses it to mint ECR tokens, and the load balancer controller uses it to create ALBs. Identity first; everything else is plumbing.

The mental model to hold for the whole lesson is the diagram below. Argo CD sits in the cluster. To its left is the identity edge that gives it (and its helpers) AWS permissions. To its right are the AWS services those permissions unlock. And wrapping the front is the ingress edge that lets you and the CLI reach it. Keep that shape in your head and each section slots into place.

---

The four AWS edges, at a glance

Before the deep dives, here is the whole wiring in one picture. Read it left to right: the EKS cluster runs Argo CD; a pod assumes an IAM role via IRSA or EKS Pod Identity; that identity lets the External Secrets Operator read Secrets Manager and lets Argo CD and the kubelet pull from ECR; and the AWS Load Balancer Controller turns an Ingress into an ALB that fronts the UI. The numbered badges mark the decisions and failure modes we unpack below.

Each edge is a distinct AWS integration with its own IAM surface, its own failure signature, and its own equivalent on the other two clouds:

Notice what is not cloud-specific: the Application, AppProject, ApplicationSet, sync waves, RBAC, and the whole GitOps loop are identical on every cloud. Only these four edges change. That is exactly why the course structures cloud lessons this way — you learn the neutral core once, then wire the edges per cloud.

---

Installing Argo CD on EKS

You reach an EKS cluster the same way you reach any: point kubectl at it. The one AWS-specific command writes a kubeconfig context that uses the AWS CLI to mint a short-lived token on every call.

# Write/refresh a kubeconfig context for the cluster (uses IAM to authenticate)
aws eks update-kubeconfig --name kloudvin-prod --region ap-south-1 --alias kloudvin-prod
# Added new context arn:aws:eks:ap-south-1:111122223333:cluster/kloudvin-prod to ~/.kube/config

kubectl config current-context     # kloudvin-prod
kubectl get nodes                  # confirm you can reach the API server
# NAME                          STATUS   ROLES    AGE   VERSION
# ip-10-0-12-31.ap-south-1...    Ready    <none>   9d    v1.30.2-eks-...

Under the hood that context runs aws eks get-token, so your kubectl access is governed by IAM — the same mechanism a remote Argo CD hub uses to register this cluster as a spoke. Two facts about EKS authorization matter here:

For a production install, run Argo CD in high availability. The Helm chart (argo/argo-cd, i.e. argo-helm) exposes a canonical HA shape — redundant replicas of every component and a Redis HA topology so a single node failure does not stall reconciliation:

# argocd-values.yaml — HA install tuned for running behind an ALB
redis-ha:
  enabled: true                 # 3-node Redis with sentinels (replaces the single redis Pod)

controller:
  replicas: 1                   # application-controller shards; scale with clusters/apps

server:
  replicas: 2                   # the API/UI server — HA behind the ALB
  autoscaling:
    enabled: true
    minReplicas: 2

repoServer:
  replicas: 2                   # manifest rendering (Helm/Kustomize/OCI pulls) — CPU-bound

applicationSet:
  replicas: 2

configs:
  params:
    server.insecure: true       # ALB terminates TLS; server speaks plain HTTP to the ALB target

helm repo add argo https://argoproj.github.io/argo-helm
helm upgrade --install argocd argo/argo-cd \
  --namespace argocd --create-namespace \
  --version 7.6.12 \
  -f argocd-values.yaml
# ... Argo CD installed. Components: application-controller, repo-server, server (x2), applicationset-controller, redis-ha

The server.insecure: true line is not a security hole — it is the deliberate choice to let the ALB terminate TLS with a real ACM certificate and speak HTTP to the Argo CD Service inside the VPC. We come back to the alternative (end-to-end TLS with backend-protocol: HTTPS) in the networking section; both are valid, and which you pick decides two Ingress annotations.

⚠️ Billing. An HA install is more Pods and a 3-node Redis; the bigger recurring cost is the ALB you are about to create (billed per hour + per LCU), any NAT gateway your private nodes use for egress, and — if you go private — the cross-VPC connectivity. None of these are free-tier. Tear the lab down when you are done (there is a teardown section).

---

IRSA vs EKS Pod Identity: how a pod assumes an IAM role

This is the foundation. Argo CD’s repo-server, the External Secrets Operator, the load balancer controller, and any ECR-refresh job all need AWS permissions, and the correct way to grant them is to map a Kubernetes ServiceAccount to an IAM role so the pod receives temporary credentials. On EKS there are two mechanisms to do this. Understanding both — and when to reach for each — is the single most valuable thing in this lesson.

IRSA: IAM Roles for Service Accounts

IRSA is the original mechanism (GA since 2019) and it works by OIDC federation. Each EKS cluster publishes an OIDC issuer URL. You register that issuer as an IAM OIDC identity provider, then write an IAM role whose trust policy says “allow a Kubernetes ServiceAccount from this cluster, with this exact name, to assume me via a web identity token.” Finally you annotate the ServiceAccount with the role ARN. A mutating webhook injects the token and environment variables, and the AWS SDK inside the pod does the rest.

There are four moving parts, and every IRSA failure is one of them being wrong:

# 1) Find the cluster's OIDC issuer
aws eks describe-cluster --name kloudvin-prod --region ap-south-1 \
  --query "cluster.identity.oidc.issuer" --output text
# https://oidc.eks.ap-south-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E

# 2) Create the IAM OIDC provider for the cluster (idempotent; one per cluster)
eksctl utils associate-iam-oidc-provider --cluster kloudvin-prod --region ap-south-1 --approve
# (or: aws iam create-open-id-connect-provider ... for a pure-CLI/Terraform path)

The trust policy is where the security lives. It pins the exact namespace:serviceaccount and the STS audience, so only that SA in that cluster can assume the role:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::111122223333:oidc-provider/oidc.eks.ap-south-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "oidc.eks.ap-south-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E:sub": "system:serviceaccount:external-secrets:external-secrets",
          "oidc.eks.ap-south-1.amazonaws.com/id/EXAMPLED539D4633E53DE1B716D3041E:aud": "sts.amazonaws.com"
        }
      }
    }
  ]
}

Then the ServiceAccount carries the role ARN as an annotation — this is the line that connects Kubernetes to IAM:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: external-secrets
  namespace: external-secrets
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::111122223333:role/eso-secrets-reader

When a pod uses that SA, the EKS Pod Identity Webhook (a cluster component, not to be confused with Pod Identity the feature) mutates the pod to inject a projected token and environment variables. You can prove it landed:

kubectl -n external-secrets exec deploy/external-secrets -- env | grep AWS_
# AWS_ROLE_ARN=arn:aws:iam::111122223333:role/eso-secrets-reader
# AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token
# AWS_STS_REGIONAL_ENDPOINTS=regional

Those three environment variables (plus the projected token file) are the entire contract between Kubernetes and the AWS SDK — the SDK reads them, exchanges the token, and gets temporary credentials:

EKS Pod Identity: the newer, simpler mechanism

EKS Pod Identity (GA late 2023) removes the two most painful parts of IRSA: the per-cluster OIDC provider and the trust-policy string-matching. Instead you install a small agent add-on once, and create a PodIdentityAssociation that maps (cluster, namespace, serviceaccount) → role ARN on the AWS side. No SA annotation, no OIDC provider, and the same IAM role can be reused across many clusters because its trust policy no longer names a specific cluster’s issuer.

# 1) Install the agent once per cluster (a DaemonSet delivered as a managed add-on)
aws eks create-addon --cluster-name kloudvin-prod --region ap-south-1 \
  --addon-name eks-pod-identity-agent

# 2) Associate a ServiceAccount with a role (no annotation, no OIDC)
aws eks create-pod-identity-association --cluster-name kloudvin-prod --region ap-south-1 \
  --namespace external-secrets \
  --service-account external-secrets \
  --role-arn arn:aws:iam::111122223333:role/eso-secrets-reader
# { "association": { "associationId": "a-abcd1234...", "associationArn": "..." } }

The trust policy for a Pod Identity role is dramatically simpler — a fixed service principal, and it must allow sts:TagSession alongside sts:AssumeRole (the agent passes cluster/namespace/SA as session tags):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "Service": "pods.eks.amazonaws.com" },
      "Action": ["sts:AssumeRole", "sts:TagSession"]
    }
  ]
}

That is the entire trust relationship — no account ID, no issuer, no sub. The association carries the binding, so the same role works on every cluster you associate it in.

Choosing between them

Both deliver the same outcome (a pod gets temporary, scoped credentials with no stored secret). They differ in setup cost, portability, and a few edge cases:

The practical guidance: for a new cluster, prefer Pod Identity — it is less JSON, less per-cluster ceremony, and role reuse across a fleet is genuinely simpler. Keep IRSA where it already works, where a tool’s docs only cover IRSA, or where you need the same federation model on non-EKS compute. Do not mix them for the same ServiceAccount: if a pod has both an IRSA annotation and a Pod Identity association, Pod Identity takes precedence, which is a classic “I changed the IAM policy and nothing happened” debugging session.

The same edge on AKS and GKE

Every cloud has a federated-identity story; the shapes rhyme. On AKS you use Azure Workload Identity: a user-assigned managed identity gets a federated credential trusting the cluster’s OIDC issuer, and the SA is annotated azure.workload.identity/client-id with the pod labeled azure.workload.identity/use: "true". On GKE, Workload Identity binds a Kubernetes SA to a Google service account via an IAM policy on serviceAccount:PROJECT.svc.id.goog[NAMESPACE/KSA], with the SA annotated iam.gke.io/gcp-service-account. This is the AKS lesson Argo CD on AKS and the GKE lesson Argo CD on GKE in depth.

---

Reading secrets from AWS Secrets Manager

You now have an identity. The first thing to point it at is AWS Secrets Manager, so that a database password lives in AWS — rotated, audited, KMS-encrypted — and only a reference to it lives in Git. The tool that does this is the External Secrets Operator (ESO), which we introduced generically in Secrets in GitOps; here we wire its AWS provider precisely.

ESO has three objects. A SecretStore (namespaced) or ClusterSecretStore (cluster-wide) says where secrets come from and how to authenticate. An ExternalSecret says which keys to pull and what Kubernetes Secret to create. ESO reads Secrets Manager on a schedule and owns the resulting Secret.

The IAM role from the previous section needs a tightly scoped read policy — GetSecretValue on exactly the secret ARNs it should see, nothing wider:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret"],
      "Resource": "arn:aws:secretsmanager:ap-south-1:111122223333:secret:prod/team-a/*"
    },
    {
      "Effect": "Allow",
      "Action": ["kms:Decrypt"],
      "Resource": "arn:aws:kms:ap-south-1:111122223333:key/EXAMPLE-KMS-KEY-ID",
      "Condition": { "StringEquals": { "kms:ViaService": "secretsmanager.ap-south-1.amazonaws.com" } }
    }
  ]
}

The kms:Decrypt statement is the single most-forgotten line: if the secret is encrypted with a customer-managed KMS key (not the AWS-managed default), reading it fails with AccessDenied on kms:Decrypt even though the Secrets Manager permission is correct.

The SecretStore differs slightly depending on which identity mechanism you chose. With IRSA, point ESO at the annotated ServiceAccount so it performs the web-identity exchange itself. With Pod Identity, associate the role with the ESO controller’s own SA and omit the auth block — ESO then uses the SDK default credential chain, which the agent satisfies:

# IRSA path — ESO assumes the role named by the referenced SA's annotation
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: aws-secretsmanager
  namespace: team-a
spec:
  provider:
    aws:
      service: SecretsManager
      region: ap-south-1
      auth:
        jwt:
          serviceAccountRef:
            name: eso-team-a          # this SA is annotated with the role ARN

# Pod Identity path — no auth block; ESO controller pod uses its ambient identity
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: aws-secretsmanager
  namespace: team-a
spec:
  provider:
    aws:
      service: SecretsManager
      region: ap-south-1
      # no auth: default credential chain, satisfied by the Pod Identity agent

The ExternalSecret then declares the materialization. Pull individual keys with data, or extract an entire JSON secret with dataFrom:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: app-db
  namespace: team-a
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secretsmanager
    kind: SecretStore
  target:
    name: app-db-credentials         # the Kubernetes Secret ESO will create and own
    creationPolicy: Owner
  data:
    - secretKey: password             # key inside the k8s Secret
      remoteRef:
        key: prod/team-a/db           # the Secrets Manager secret name
        property: password            # a field within the secret's JSON
  # dataFrom:                          # alternative: pull the whole JSON as keys
  #   - extract:
  #       key: prod/team-a/db

kubectl -n team-a get externalsecret app-db
# NAME     STORE                STATUS         READY
# app-db   aws-secretsmanager   SecretSynced   True
kubectl -n team-a get secret app-db-credentials -o jsonpath='{.data.password}' | base64 -d
# (the real value, fetched from Secrets Manager — never committed to Git)

Commit the SecretStore and ExternalSecret to Git and let Argo CD sync them; they contain no secret value, only references. There is one gotcha specific to Argo CD, carried over from the secrets lesson: ESO owns the materialized Secret and keeps mutating it, so if Argo CD also tracks that Secret it will report OutOfSync forever. Manage only the ExternalSecret in Git, and if Argo CD does see the Secret, add it to ignoreDifferences on data.

Parameter Store — the cheaper alternative. For non-rotating configuration (feature flags, endpoints, non-critical tokens), AWS Systems Manager Parameter Store is far cheaper: standard parameters are free versus roughly $0.40 per secret per month for Secrets Manager, though Parameter Store lacks native rotation. ESO speaks to it by changing one field:

spec:
  provider:
    aws:
      service: ParameterStore        # instead of SecretsManager
      region: ap-south-1

The same edge on AKS and GKE

---

ECR: pulling images and OCI Helm charts

The registry edge has a signature problem: an ECR authorization token is valid for exactly 12 hours. Any approach that stashes a token in a Kubernetes Secret works beautifully in the demo and breaks the next morning with a 401. The fix depends on who is pulling, and there are two distinct consumers with two different answers.

Image pulls: the node role already solves it

For ordinary workload images in the same account, you do nothing. The EKS node AMI ships an ECR credential provider that the kubelet uses to fetch a fresh token from the node’s instance role on every pull. Give the node role the read policy and image pulls just work — no imagePullSecret, no refresh job:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ecr:GetAuthorizationToken",
        "ecr:BatchCheckLayerAvailability",
        "ecr:GetDownloadUrlForLayer",
        "ecr:BatchGetImage"
      ],
      "Resource": "*"
    }
  ]
}

In practice you attach the AWS-managed AmazonEC2ContainerRegistryReadOnly policy (or the newer, tighter AmazonEC2ContainerRegistryPullOnly, 2024) to the node role and move on. Note that ecr:GetAuthorizationToken is an account-level action so its Resource is *; the per-image actions can be scoped to repository ARNs. Cross-account pulls need one extra thing: a repository policy on the ECR repo granting the puller’s account/role access — that is the common cause of a 403 when the image lives in a shared registry account.

OCI Helm charts: the repo-server needs a refreshed token

Argo CD pulling a Helm chart stored as an OCI artifact in ECR is a pod-level pull by the repo-server, and here you do need credentials in an Argo CD repository Secret. Argo CD does not natively refresh ECR tokens, so a static password rots after 12 hours:

apiVersion: v1
kind: Secret
metadata:
  name: ecr-oci-charts
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: repository
stringData:
  type: helm
  name: ecr-charts
  url: 111122223333.dkr.ecr.ap-south-1.amazonaws.com
  enableOCI: "true"
  username: AWS
  password: PLACEHOLDER_12H_ECR_TOKEN     # from `aws ecr get-login-password` — EXPIRES in 12h

The clean fix is a small CronJob that runs every few hours, mints a fresh token using IRSA or Pod Identity (no static AWS key), and patches the repository Secret. The identity you built earlier is exactly what makes this safe:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: ecr-token-refresh
  namespace: argocd
spec:
  schedule: "0 */6 * * *"          # every 6h; comfortably inside the 12h TTL
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: ecr-refresher   # IRSA/Pod Identity → ecr:GetAuthorizationToken
          restartPolicy: OnFailure
          containers:
            - name: refresh
              image: amazon/aws-cli:2.17.0
              command: ["/bin/sh", "-c"]
              args:
                - |
                  TOKEN=$(aws ecr get-login-password --region ap-south-1)
                  kubectl -n argocd patch secret ecr-oci-charts \
                    --type merge -p "{\"stringData\":{\"password\":\"$TOKEN\"}}"

⚠️ Billing. ECR charges for stored data and for cross-Region/cross-AZ data transfer on pulls. A busy cluster pulling large images across AZs adds up; keep the registry in the cluster’s Region and use VPC endpoints for ECR to avoid NAT egress charges on every pull.

The same edge on AKS and GKE

---

Exposing the UI with the AWS Load Balancer Controller

Argo CD’s server serves the web UI and the gRPC API on one port. To expose it on EKS the right way, you install the AWS Load Balancer Controller, which watches Ingress objects with ingressClassName: alb and provisions a real Application Load Balancer (or an NLB for Service type: LoadBalancer with its own annotations). This gives you an ACM certificate, WAF integration, and health checks — far better than a raw LoadBalancer Service.

The controller itself needs IRSA/Pod Identity (it creates ALBs, target groups, and listeners on your behalf), so it is another consumer of the identity edge:

helm repo add eks https://aws.github.io/eks-charts
helm upgrade --install aws-load-balancer-controller eks/aws-load-balancer-controller \
  --namespace kube-system \
  --set clusterName=kloudvin-prod \
  --set serviceAccount.create=false \
  --set serviceAccount.name=aws-load-balancer-controller   # pre-annotated with its IAM role

The Ingress for Argo CD carries the ALB configuration as annotations. This is the end-to-end-TLS variant that pairs with server.insecure: false (the server keeps its own TLS and the ALB re-encrypts to it):

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: argocd-server
  namespace: argocd
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/backend-protocol: HTTPS          # ALB re-encrypts to the server
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
    alb.ingress.kubernetes.io/ssl-redirect: '443'
    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:ap-south-1:111122223333:certificate/EXAMPLE-CERT-ID
    alb.ingress.kubernetes.io/healthcheck-path: /healthz?full=true
spec:
  ingressClassName: alb
  rules:
    - host: argocd.kloudvin.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: argocd-server
                port:
                  number: 443          # 443 for HTTPS backend; use 80 if server runs --insecure

If instead you set server.insecure: true (the simpler ALB-terminates-TLS model from the install section), change backend-protocol to HTTP and the service port to 80. Both are correct; pick one and keep the settings consistent — a mismatch between them is the usual cause of a redirect loop or a 502:

The subnet-tag trap

The single most common “my ALB never appeared” cause is subnet tagging. The controller auto-discovers which subnets to place the ALB in by reading tags. Miss them and the Ingress sits forever with no ADDRESS:

Why the CLI breaks behind an ALB — and --grpc-web

Here is the gotcha that costs everyone an afternoon. The web UI works fine through the ALB, but argocd login hangs or fails with rpc error: code = Unavailable. The reason: the Argo CD CLI speaks gRPC, and the server multiplexes gRPC and REST on one port. A single-listener ALB does not cleanly proxy that multiplexed gRPC. The fix is gRPC-Web, which tunnels gRPC over ordinary HTTP/1.1 so the ALB handles it like any web request:

# Log in through the ALB using gRPC-Web (works where plain gRPC stalls)
argocd login argocd.kloudvin.com --grpc-web
# 'admin:login' logged in successfully

argocd app list --grpc-web
# NAME              CLUSTER                         STATUS   HEALTH
# argocd/root       https://kubernetes.default.svc  Synced   Healthy

You can bake --grpc-web into the CLI context so you never forget it. On the server side, running with --insecure (ALB terminates TLS) is the most reliable pairing behind an ALB.

TargetGroupBinding: when Terraform owns the ALB

Sometimes you provision the ALB and target group in Terraform (for a shared, long-lived LB) and only want Kubernetes to keep Pod IPs registered. That is what TargetGroupBinding does — it binds an existing target group to a Service so the controller manages membership without owning the ALB:

apiVersion: elbv2.k8s.aws/v1beta1
kind: TargetGroupBinding
metadata:
  name: argocd-server
  namespace: argocd
spec:
  serviceRef:
    name: argocd-server
    port: 443
  targetGroupARN: arn:aws:elasticloadbalancing:ap-south-1:111122223333:targetgroup/argocd/EXAMPLE
  targetType: ip

The same edge on AKS and GKE

---

Private EKS and hub connectivity

Production clusters often make the API server endpoint private (endpointPublicAccess: false, endpointPrivateAccess: true) so the control plane is unreachable from the internet. That is good security and a real operational constraint: anything that talks to the API server — your kubectl, CI, and crucially a remote Argo CD hub registering this cluster as a spoke — must now sit inside a network path to the private endpoint.

If your Argo CD hub lives in a different VPC or account from a private spoke, the hub’s application-controller and repo-server must be able to dial the spoke’s private API endpoint. The options, roughly in order of how most teams do it:

The failure signature when the path is missing is unambiguous: the cluster registers fine (the Secret is written), but syncs fail with dial tcp <api-ip>:443: i/o timeout. Registration writing credentials and the network actually being reachable are two separate things — cluster registration handles the credentials half; this is the network half.

⚠️ Billing. Private endpoints, Transit Gateway attachments, and NAT gateways all bill hourly plus per-GB. A private-EKS design is the right call for production but is materially more expensive than a public-endpoint dev cluster — size it deliberately.

---

Terraform: wiring it as code

Everything above can be provisioned declaratively. Here is a representative snippet using the terraform-aws-eks module (v20+, which uses access entries by default) with the AWS provider v5, an IRSA role for ESO, the Pod Identity variant, and Argo CD installed via helm_release. This is the “forward-reference bootstrap” pattern: Terraform stands up the cluster and a seed Argo CD, and from then on Argo CD manages everything else (including, via an app-of-apps, itself).

terraform {
  required_providers {
    aws  = { source = "hashicorp/aws", version = "~> 5.60" }
    helm = { source = "hashicorp/helm", version = "~> 2.14" }
  }
}

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "~> 20.24"

  cluster_name    = "kloudvin-prod"
  cluster_version = "1.30"

  cluster_endpoint_public_access = true            # set false + private access for a private cluster
  enable_irsa                    = true            # publishes the OIDC provider for IRSA

  # v20 defaults to access entries; grant the Terraform caller admin
  authentication_mode                      = "API_AND_CONFIG_MAP"
  enable_cluster_creator_admin_permissions = true

  vpc_id     = var.vpc_id
  subnet_ids = var.private_subnet_ids

  eks_managed_node_groups = {
    default = { instance_types = ["m6i.large"], min_size = 2, max_size = 6, desired_size = 3 }
  }
}

# --- Identity option A: IRSA role for the External Secrets Operator ---
module "eso_irsa" {
  source  = "terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks"
  version = "~> 5.44"

  role_name                      = "eso-secrets-reader"
  attach_external_secrets_policy = true            # built-in least-privilege ESO policy

  oidc_providers = {
    main = {
      provider_arn               = module.eks.oidc_provider_arn
      namespace_service_accounts = ["external-secrets:external-secrets"]
    }
  }
}

# --- Identity option B: EKS Pod Identity (no OIDC, no SA annotation) ---
resource "aws_eks_pod_identity_association" "eso" {
  cluster_name    = module.eks.cluster_name
  namespace       = "external-secrets"
  service_account = "external-secrets"
  role_arn        = aws_iam_role.eso_podidentity.arn   # trust: pods.eks.amazonaws.com + TagSession
}

# --- Seed Argo CD; after this, GitOps takes over ---
resource "helm_release" "argocd" {
  name             = "argocd"
  namespace        = "argocd"
  create_namespace = true
  repository       = "https://argoproj.github.io/argo-helm"
  chart            = "argo-cd"
  version          = "7.6.12"
  values           = [file("${path.module}/argocd-values.yaml")]

  depends_on = [module.eks]
}

The attach_external_secrets_policy = true input is a convenience: the IAM module ships a curated least-privilege policy for ESO so you don’t hand-write the GetSecretValue JSON. The two identity options are shown side by side deliberately — pick one per ServiceAccount, not both.

---

Hands-on lab

This lab wires Argo CD on EKS end to end at the config level — every manifest is real and schema-correct, but because it provisions billable AWS resources, treat the outputs as representative and run it only against a cluster you own. Each step has the command, what you should see, and a one-line “what just happened.”

⚠️ Billing warning up front. Steps 5–6 create an ALB (hourly + LCU) and rely on a cluster whose nodes likely egress through a NAT gateway. Do the teardown at the end. Nothing here is free-tier.

Step 1 — Reach the cluster.

aws eks update-kubeconfig --name kloudvin-prod --region ap-south-1
kubectl get nodes
# NAME                        STATUS   ROLES    AGE   VERSION
# ip-10-0-11-12.ap-south-1... Ready    <none>   3h    v1.30.2-eks-...

What just happened: your kubectl now authenticates to EKS through IAM via aws eks get-token.

Step 2 — Install Argo CD (HA) and confirm the Pods.

helm repo add argo https://argoproj.github.io/argo-helm && helm repo update
helm upgrade --install argocd argo/argo-cd -n argocd --create-namespace \
  --version 7.6.12 -f argocd-values.yaml
kubectl -n argocd get deploy
# NAME                        READY   UP-TO-DATE   AVAILABLE
# argocd-server               2/2     2            2
# argocd-repo-server          2/2     2            2
# argocd-applicationset-...   2/2     2            2

What just happened: Argo CD is running in HA; the server has 2 replicas ready to sit behind the ALB.

Step 3 — Create the IRSA role and wire ESO’s ServiceAccount. (Pod Identity variant shown as an alternative.)

# OIDC provider (idempotent)
eksctl utils associate-iam-oidc-provider --cluster kloudvin-prod --region ap-south-1 --approve
# Create role eso-secrets-reader with the trust policy + GetSecretValue policy shown earlier,
# then annotate the SA:
kubectl -n external-secrets annotate sa external-secrets \
  eks.amazonaws.com/role-arn=arn:aws:iam::111122223333:role/eso-secrets-reader --overwrite

# --- Pod Identity alternative (no annotation) ---
aws eks create-addon --cluster-name kloudvin-prod --addon-name eks-pod-identity-agent --region ap-south-1
aws eks create-pod-identity-association --cluster-name kloudvin-prod --region ap-south-1 \
  --namespace external-secrets --service-account external-secrets \
  --role-arn arn:aws:iam::111122223333:role/eso-secrets-reader

What just happened: ESO’s pod can now assume an IAM role and read Secrets Manager with temporary credentials — no stored AWS key.

Step 4 — Deploy a SecretStore and ExternalSecret.

kubectl apply -f secretstore.yaml -f externalsecret.yaml
kubectl -n team-a get externalsecret app-db
# NAME     STORE                STATUS         READY
# app-db   aws-secretsmanager   SecretSynced   True

What just happened: ESO authenticated to Secrets Manager, read prod/team-a/db, and materialized the app-db-credentials Secret. The value came from AWS; only the reference is in Git.

Step 5 — Expose the UI via an ALB Ingress.

kubectl apply -f argocd-ingress.yaml
kubectl -n argocd get ingress argocd-server
# NAME            CLASS   HOSTS                 ADDRESS                                   PORTS
# argocd-server   alb     argocd.kloudvin.com   k8s-argocd-...ap-south-1.elb.amazonaws.com   80,443

What just happened: the AWS Load Balancer Controller saw the alb Ingress, discovered your tagged subnets, provisioned an ALB with your ACM cert, and populated ADDRESS. If ADDRESS stays empty, re-check subnet tags (the number-one cause).

Step 6 — Log in through the ALB with --grpc-web.

argocd login argocd.kloudvin.com --grpc-web
# 'admin:login' logged in successfully
argocd app list --grpc-web
# NAME          CLUSTER                          STATUS   HEALTH

What just happened: the CLI reached the gRPC API through the ALB by tunneling over gRPC-Web. Plain argocd login (without --grpc-web) would have hung — that is the expected failure, not a broken cluster.

Step 7 — Teardown.

kubectl delete -f argocd-ingress.yaml           # deletes the ALB (stops LB billing)
kubectl delete -f externalsecret.yaml -f secretstore.yaml
helm uninstall argocd -n argocd && kubectl delete ns argocd
aws eks delete-pod-identity-association --cluster-name kloudvin-prod --region ap-south-1 \
  --association-id a-abcd1234    # if you used the Pod Identity path
# (delete the IAM role/policies you created for the lab)

What just happened: deleting the Ingress deprovisions the ALB — the most important teardown step for billing. Then Argo CD and the ESO objects go, and you clean up the IAM you created.

---

Going deeper

The main lesson wired all four edges correctly. This section is for the reader who now has to operate them: which pods actually hold AWS credentials (fewer than you think), the ESO alternative you will eventually be asked to compare, why a “private cluster” breaks AWS calls and not just the API server, and how a deploy reaches a second AWS account.

Not the whole install needs AWS — grant per component

The most common over-grant is minting one fat argocd IAM role and wiring it to everything. Don’t. AWS identity is granted per ServiceAccount, and most of an Argo CD install needs no AWS permissions at all — it talks to Git and the Kubernetes API, both cloud-neutral. Give a scoped role only to the specific pods that make AWS API calls, so a mistake in one policy can’t blast the others.

Argo CD Image Updater is the one people forget is a separate AWS consumer — it polls the registry directly, on its own schedule, and needs its own scoped role, covered in Image Updater across registries. Give each of these its own role and its own association; never share one role “to save time,” because that single role becomes the blast radius for every one of them.

External Secrets Operator vs the Secrets Store CSI driver

ESO is not the only way to pull a Secrets Manager value into a pod. The Secrets Store CSI driver with the AWS provider (ASCP) does the same job with a different delivery model: it mounts secrets as files into a pod through a SecretProviderClass, and can optionally sync them into a real Kubernetes Secret.

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: app-db
  namespace: team-a
spec:
  provider: aws
  parameters:
    objects: |
      - objectName: "prod/team-a/db"
        objectType: "secretsmanager"
        jmesPath:
          - path: password
            objectAlias: password
  secretObjects:                     # optional: also mirror into a k8s Secret
    - secretName: app-db-credentials
      type: Opaque
      data:
        - objectName: password
          key: password

The consuming pod must mount a CSI volume that references this SecretProviderClass, and that mount is what triggers the fetch — the fundamental difference from ESO.

For Argo CD-managed workloads ESO usually wins, because Argo CD deploys the ExternalSecret declaratively and the resulting Secret is available to anything — other controllers, image-pull flows, envFrom — none of which can mount a CSI volume. Reach for the CSI driver when exactly one application wants its secrets as files and you would rather not have a standalone Secret object sitting in the namespace at all.

A private cluster breaks AWS calls, not just the API server

The Private EKS section above made the Kubernetes API server private. There is a second, independent privacy problem that bites harder: if your nodes have no internet egress (no NAT gateway, fully private subnets), then the AWS SDK calls that IRSA, ESO, ECR and the load balancer controller all depend on have nowhere to go. IRSA fails first and most confusingly — the pod cannot reach STS to exchange its token, so every AWS call fails with a timeout that reads like a permissions bug.

The fix is VPC interface endpoints (PrivateLink) for each service the cluster calls, plus the one gateway endpoint everyone forgets:

The S3 gateway endpoint for ECR layers is the single most common omission — authentication succeeds, so everything looks fine, but pulls stall because the layers themselves come from S3. Because the webhook already sets AWS_STS_REGIONAL_ENDPOINTS=regional, the STS interface endpoint in the cluster’s own Region is the one that matters. The broader private-connectivity design is covered in Private clusters & connectivity.

Cross-account: reaching a secret or a registry in another account

Real estates split resources across accounts — a shared secrets account, a central ECR account, a per-environment workload account. The pattern is role chaining: the pod assumes its own role via IRSA/Pod Identity, and that role assumes a role in the target account with sts:AssumeRole. ESO expresses it with a single field — the base identity authenticates, then role names the account-B role to assume:

apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: aws-secretsmanager-shared
  namespace: team-a
spec:
  provider:
    aws:
      service: SecretsManager
      region: ap-south-1
      role: arn:aws:iam::444455556666:role/eso-cross-account-reader   # assumed in account B
      auth:
        jwt:
          serviceAccountRef:
            name: eso-team-a                                          # base IRSA identity in account A

Two things must line up: the target role’s trust policy (in account 444455556666) must allow the source role to assume it, and the resource itself may need a resource policy — a secret resource policy for a cross-account Secrets Manager secret, or a repository policy for cross-account ECR (the usual cause of a 403 on a shared registry). The operational gotcha is the one-hour cap on role chaining: a chained session cannot exceed 3600 seconds, so long-running work must re-assume rather than hold a single credential. EKS Pod Identity offers a more direct route — an association can carry a --target-role-arn for cross-account assumption without hand-writing the chain. For cross-cluster deploys (a hub in one account driving a spoke in another), the mechanism is instead an EKS access entry for the hub’s role on the spoke, as in hub-and-spoke fleets.

---

Common mistakes and troubleshooting

Every failure below is a real symptom you will actually see. The table maps symptom → cause → fix; the prose after it covers the three nastiest.

The IRSA sub mismatch. Ninety percent of IRSA AccessDenied errors are a one-character difference in the trust policy’s sub condition. It must be exactly system:serviceaccount:<namespace>:<serviceaccount-name> — the literal prefix system:serviceaccount:, the namespace (not the cluster), then the ServiceAccount name. Deploy ESO into external-secrets but write the sub for default and you get a denial that looks like a permissions problem but is really a typo. When in doubt, kubectl -n <ns> exec <pod> -- env | grep AWS_ROLE_ARN and confirm the pod is even using the SA you think it is.

The 12-hour ECR cliff. This one is insidious because it works in the demo. You paste a token into the repository Secret, Argo CD syncs the OCI chart, everyone moves on — and the next morning syncs fail with a 403. The token expired. If you only remember one thing about ECR: never store a raw ECR token as if it were permanent. Node role for images, refresh CronJob for OCI chart repos, and the underlying credential is always minted fresh from ecr:GetAuthorizationToken using IRSA/Pod Identity.

The subnet-tag silence. The load balancer controller does not error loudly when subnets are untagged — the Ingress simply never gets an ADDRESS, which reads like “nothing is happening.” Check the controller logs (kubectl -n kube-system logs deploy/aws-load-balancer-controller) and you’ll see it complaining it can’t find subnets. Tag public subnets kubernetes.io/role/elb=1 for an internet-facing ALB, private subnets kubernetes.io/role/internal-elb=1 for an internal one, and the ALB appears within a minute.

---

Common beginner mistakes

These are conceptual mix-ups, not symptoms — the symptom→cause→fix pairs live in the troubleshooting table above. Each item below is a wrong mental model and the right one to replace it with.

• “IRSA gives Argo CD AWS permissions.” It grants nothing to “Argo CD” as a whole — it grants to a specific ServiceAccount’s pods. Argo CD is a dozen pods; most need zero AWS access. Think per-ServiceAccount, per-role, least privilege — not “the Argo CD role.”
• “server.insecure: true means the UI is insecure.” It only means the ALB terminates TLS and speaks plain HTTP to the server inside the VPC. The internet-facing listener is still HTTPS with a real ACM certificate. It is a topology choice; the alternative (backend-protocol: HTTPS) merely re-encrypts the in-VPC hop.
• “ESO puts my secret in Git.” Only the ExternalSecret reference is committed — the secret’s name and which key to pull. The value never touches Git; ESO fetches it at runtime and writes it into a Kubernetes Secret the workload reads.
• “Setting both an IRSA annotation and a Pod Identity association is belt-and-suspenders.” Pod Identity silently wins, so you can spend an afternoon editing the IRSA policy while nothing changes. Pick exactly one mechanism per ServiceAccount.
• “My OCI sync worked, so the ECR token is fine.” The token is valid for 12 hours. The demo works; the first sync after it expires fails with a 403. A stored ECR token is always a time bomb — mint it fresh from ecr:GetAuthorizationToken, never treat it as permanent.
• “Private EKS just means the API server is private.” It usually also means your pods cannot reach AWS APIs. Without a NAT gateway or VPC endpoints, IRSA cannot reach STS and every AWS call times out — a networking problem wearing a permissions costume.
• “The ARNs and account IDs in this lesson are copy-paste ready.” 111122223333, 444455556666, the EXAMPLE... identifiers and argocd.kloudvin.com are placeholders. Substitute your own account, Region, cluster, certificate and host everywhere before applying anything.

---

Cheat-sheet

The AWS-wiring quick reference:

The cross-cloud contrast, consolidated — the whole point of a multi-cloud course in one table:

---

Practice challenges

Six exercises, from “prove the wiring” to “operate it under real constraints.” Every command and manifest is schema-correct; the AWS resources are billable, so run them only against an account you own. Try each before opening the solution.

1. (Beginner) Prove a pod actually received an AWS identity. You annotated the ESO ServiceAccount but aren’t sure the webhook did its job. How do you confirm from inside the pod?

<details> <summary>Solution</summary>

kubectl -n external-secrets exec deploy/external-secrets -- env | grep AWS_
# AWS_ROLE_ARN=arn:aws:iam::111122223333:role/eso-secrets-reader
# AWS_WEB_IDENTITY_TOKEN_FILE=/var/run/secrets/eks.amazonaws.com/serviceaccount/token

Why: the AWS_* env vars are the entire contract between Kubernetes and the SDK — if they’re absent, the SA annotation (IRSA) or the association (Pod Identity) never took effect. </details>

2. (Beginner) Write the exact IRSA sub for the repo-server. It runs as ServiceAccount argocd-repo-server in namespace argocd. What sub condition goes in the role’s trust policy?

<details> <summary>Solution</summary>

"...:sub": "system:serviceaccount:argocd:argocd-repo-server"

Why: the literal prefix system:serviceaccount:, then namespace, then ServiceAccount — a one-character slip here is the cause of most IRSA AccessDenied errors. </details>

3. (Intermediate) Convert ESO from IRSA to Pod Identity. ESO currently works via an annotated SA. Migrate it. List the steps and the trust-policy change.

<details> <summary>Solution</summary>

aws eks create-addon --cluster-name kloudvin-prod --region ap-south-1 \
  --addon-name eks-pod-identity-agent
aws eks create-pod-identity-association --cluster-name kloudvin-prod --region ap-south-1 \
  --namespace external-secrets --service-account external-secrets \
  --role-arn arn:aws:iam::111122223333:role/eso-secrets-reader
kubectl -n external-secrets annotate sa external-secrets eks.amazonaws.com/role-arn-   # remove annotation

Change the role’s trust policy from the OIDC Federated principal to {"Service":"pods.eks.amazonaws.com"} with ["sts:AssumeRole","sts:TagSession"].

Why: it exercises the real difference — Pod Identity drops the OIDC provider and the SA annotation and swaps the trust principal; the association is now what takes effect. </details>

4. (Intermediate) Fix a CMK AccessDenied. ESO returns AccessDenied on a secret even though the role has secretsmanager:GetSecretValue. The secret is encrypted with a customer-managed KMS key. Fix the policy.

<details> <summary>Solution</summary>

{ "Effect": "Allow", "Action": ["kms:Decrypt"],
  "Resource": "arn:aws:kms:ap-south-1:111122223333:key/EXAMPLE-KMS-KEY-ID",
  "Condition": { "StringEquals": { "kms:ViaService": "secretsmanager.ap-south-1.amazonaws.com" } } }

Why: a customer-managed key needs an explicit kms:Decrypt; the Secrets Manager permission alone can’t decrypt the ciphertext. The kms:ViaService condition keeps the grant tight. </details>

5. (Advanced) Schedule the ECR refresh with a safety margin. The OCI-chart repository Secret holds a 12-hour ECR token. Pick a CronJob schedule and justify it — why not 0 */11 * * *?

<details> <summary>Solution</summary>

schedule: "0 */6 * * *"   # every 6h — two refreshes fit inside the 12h TTL

Why: refreshing every 6 hours replaces the token with ~6 hours to spare, so one failed run still leaves a valid token. */11 leaves a one-hour margin — a single missed run and the token expires before the next attempt. </details>

6. (Advanced) Diagnose IRSA failing on a fully private cluster. You moved nodes to private subnets with no NAT gateway. Suddenly every AWS call from ESO times out, though nothing in IAM changed. What’s missing, and which endpoints restore it?

<details> <summary>Solution</summary>

Add VPC interface endpoints for sts, secretsmanager, ecr.api, ecr.dkr (and elasticloadbalancing / ec2 for the LB controller), plus the S3 gateway endpoint for ECR layers:

com.amazonaws.ap-south-1.sts             # IRSA token exchange
com.amazonaws.ap-south-1.secretsmanager  # ESO
com.amazonaws.ap-south-1.ecr.api / .ecr.dkr
com.amazonaws.ap-south-1.s3  (gateway)   # ECR layer downloads

Why: with no internet egress the SDK can’t reach STS to exchange the IRSA token, so every call times out and looks like a permissions bug. The S3 gateway endpoint is the classic omission — auth succeeds but image layers never download. </details>

---

Interview and exam questions

Q: What problem does IRSA solve, and what are its four moving parts? A: IRSA lets a pod assume an IAM role using short-lived, federated credentials instead of a stored access key. The four parts are: the cluster’s IAM OIDC provider; the IAM role’s trust policy pinning the ServiceAccount sub and sts.amazonaws.com audience; the ServiceAccount annotation eks.amazonaws.com/role-arn; and the Pod Identity Webhook that injects the token and AWS_* env vars. Any IRSA failure is one of those four being wrong.

Q: How is EKS Pod Identity different from IRSA, and when would you choose it? A: Pod Identity replaces per-cluster OIDC federation with a cluster agent add-on plus a PodIdentityAssociation that binds a ServiceAccount to a role on the AWS side. There’s no OIDC provider and no SA annotation, and the role’s trust policy is a fixed pods.eks.amazonaws.com principal (with sts:TagSession), so one role is trivially reusable across clusters. Choose it for new clusters and fleets; keep IRSA where it already works or where you need federation on non-EKS compute. If both are configured for one SA, Pod Identity wins.

Q: How does the External Secrets Operator authenticate to Secrets Manager on EKS? A: It uses the pod’s IAM identity. With IRSA, the SecretStore references the annotated ServiceAccount via auth.jwt.serviceAccountRef and ESO performs the web-identity exchange. With Pod Identity, you associate the role with ESO’s controller SA and omit the auth block so ESO uses the SDK default chain. The IAM role needs secretsmanager:GetSecretValue on the secret ARN, plus kms:Decrypt if a customer-managed key encrypts it.

Q: Why does an ECR-based imagePullSecret break the next day, and what’s the right fix? A: An ECR authorization token is valid for only 12 hours, so a token stashed in a Secret expires. For workload images, don’t use a Secret at all — the kubelet’s ECR credential provider fetches fresh tokens from the node role. For Argo CD pulling OCI Helm charts, use a repository Secret refreshed by a CronJob that mints a new token via ecr:GetAuthorizationToken using IRSA/Pod Identity every few hours.

Q: Your Argo CD Ingress never gets an ADDRESS. Walk through the diagnosis. A: First confirm the AWS Load Balancer Controller is installed and running (kubectl -n kube-system get deploy aws-load-balancer-controller). Then check subnet tags — the number-one cause — public subnets need kubernetes.io/role/elb=1 (or internal-elb for internal). Read the controller logs for “unable to discover subnets.” Confirm ingressClassName: alb and a valid certificate-arn in the ALB’s Region.

Q: The web UI works but argocd login hangs behind the ALB. Why, and how do you fix it? A: The CLI speaks gRPC, and a single-port ALB can’t cleanly proxy the server’s multiplexed gRPC/REST. Use argocd login <host> --grpc-web to tunnel gRPC over HTTP/1.1, and run the server --insecure behind the TLS-terminating ALB. Bake --grpc-web into the CLI context so it’s automatic.

Q: What breaks when you make the EKS API endpoint private, for a hub-and-spoke Argo CD? A: Registration still succeeds (the cluster Secret is written), but the hub’s controller and repo-server can no longer reach the private API server, so syncs fail with dial tcp ... i/o timeout. You need a network path — VPC peering, Transit Gateway, or PrivateLink — between the hub’s network and the spoke’s private endpoint. Credentials and connectivity are separate problems.

Q: What’s the cheaper alternative to Secrets Manager, and what do you give up? A: SSM Parameter Store — standard parameters are free versus ~$0.40/secret/month. ESO switches to it by setting service: ParameterStore and the IAM action becomes ssm:GetParameter. You give up native rotation, so use it for non-rotating config and keep Secrets Manager for credentials that must rotate.

Q: What is a TargetGroupBinding and when do you use it? A: It’s a CRD from the AWS Load Balancer Controller that binds an existing target group to a Kubernetes Service, so the controller keeps Pod IPs registered without creating or owning the ALB. Use it when Terraform (or another team) provisions a shared, long-lived ALB and you only want Kubernetes to manage target membership.

Q: Why prefer EKS access entries over the aws-auth ConfigMap for registering a cluster with Argo CD? A: Access entries are an IAM-side API (GA late 2023) that maps IAM roles to cluster access without editing the aws-auth ConfigMap — a ConfigMap whose malformed edit can lock everyone out of the cluster. They’re managed by API/Terraform, are auditable, and are the recommended way to grant a hub’s IAM role RBAC on a spoke. New clusters should run authentication_mode: API (or the hybrid).

Q: Map each AWS edge to its AKS and GKE equivalent. A: Identity: IRSA/Pod Identity ≈ Azure Workload Identity ≈ GKE Workload Identity. Secrets: Secrets Manager ≈ Key Vault ≈ Secret Manager (ESO providers aws/azurekv/gcpsm). Registry: ECR ≈ ACR ≈ Artifact Registry. Load balancer: ALB (AWS LB Controller) ≈ Application Gateway (AGIC) ≈ GCLB (GKE Ingress). The Argo CD objects are identical across all three; only these edges change.

---

Glossary

• IRSA (IAM Roles for Service Accounts) — the 2019 mechanism that lets a pod assume an IAM role by federating the cluster’s OIDC identity; needs an OIDC provider, a trust policy, and a ServiceAccount annotation.
• EKS Pod Identity — the newer (2023) mechanism that binds a ServiceAccount to an IAM role through an agent add-on and an association, with no OIDC provider and no annotation.
• OIDC provider — an IAM object registering a cluster’s OpenID Connect issuer so AWS will trust tokens it signs; one per cluster, required for IRSA.
• Trust policy — the part of an IAM role that says who may assume it; for IRSA it pins the ServiceAccount sub and STS aud, for Pod Identity it names pods.eks.amazonaws.com.
• sub / aud — the subject (system:serviceaccount:<ns>:<sa>) and audience (sts.amazonaws.com) claims an IRSA trust policy checks on the projected token.
• STS / AssumeRoleWithWebIdentity — AWS Security Token Service and the call that exchanges a projected OIDC token for temporary credentials.
• Session tags / sts:TagSession — key-value tags attached to an assumed-role session; the Pod Identity agent passes cluster/namespace/SA as tags, so its trust policy must allow sts:TagSession.
• ServiceAccount — the Kubernetes identity a pod runs as; the anchor that both IRSA and Pod Identity map to an IAM role.
• External Secrets Operator (ESO) — a controller that reads an external store (here Secrets Manager) and writes the value into a Kubernetes Secret it owns.
• SecretStore / ClusterSecretStore — ESO objects declaring where secrets come from and how to authenticate, namespaced or cluster-wide.
• ExternalSecret — the ESO object declaring which keys to pull and which Kubernetes Secret to create.
• Secrets Store CSI driver / SecretProviderClass — the alternative that mounts secrets into a pod as files, with an optional sync to a Kubernetes Secret.
• AWS Secrets Manager — the managed store for rotating credentials, KMS-encrypted and audited; roughly $0.40 per secret per month.
• Parameter Store — SSM’s cheaper store (standard tier free) for non-rotating configuration; ESO reaches it with service: ParameterStore.
• KMS / CMK — the Key Management Service and a customer-managed key; reading a CMK-encrypted secret needs an extra kms:Decrypt grant.
• ECR — Elastic Container Registry, AWS’s private registry for images and OCI Helm charts.
• ECR authorization token — the 12-hour credential minted by ecr:GetAuthorizationToken; must be refreshed, never stored as permanent.
• ECR credential provider — the helper baked into EKS node AMIs that fetches fresh ECR tokens for the kubelet automatically, so same-account image pulls need no Secret.
• AWS Load Balancer Controller — the controller that turns an ingressClassName: alb Ingress into a real Application Load Balancer.
• ALB / ACM — the Application Load Balancer (L7) and AWS Certificate Manager, whose certificate the ALB presents for HTTPS.
• gRPC-Web — the transport (argocd login --grpc-web) that tunnels the CLI’s gRPC over HTTP/1.1 so a single-port ALB can proxy it.
• TargetGroupBinding — a controller CRD that registers Pod IPs into an existing target group without owning the ALB.
• EKS access entries — the IAM-side API (2023) that grants an IAM role cluster RBAC without editing the aws-auth ConfigMap.
• VPC interface endpoint (PrivateLink) — a private in-VPC route to an AWS API (STS, Secrets Manager, ECR…) so a cluster with no internet egress can still call it.
• Role chaining — one assumed role assuming another (sts:AssumeRole) to reach a second account; capped at a one-hour session.

---

Key takeaways

• Argo CD is cloud-neutral; four edges are not. On EKS you must wire identity, secrets, registry, and networking — and each has one right answer (short-lived federated credentials) and several traps (stored access keys, expired tokens).
• Identity is the foundation. A pod assumes an IAM role via IRSA (OIDC provider + trust policy + SA annotation) or the newer EKS Pod Identity (agent add-on + association, no OIDC, no annotation). Prefer Pod Identity for new clusters; if both are set, Pod Identity wins.
• Secrets Manager via ESO keeps only a reference in Git. Scope IAM to GetSecretValue on the secret ARN and remember kms:Decrypt for customer-managed keys. Parameter Store is the cheaper store for non-rotating config.
• ECR’s 12-hour token is the registry gotcha: node role + kubelet credential provider for images (no Secret), and a refresh CronJob (minting tokens via IRSA/Pod Identity) for OCI chart repositories. Never ship a static ECR token.
• The AWS Load Balancer Controller turns an ingressClassName: alb Ingress into an ALB with ACM TLS — but only if subnets are tagged (kubernetes.io/role/elb=1), and the CLI needs --grpc-web to work through it.
• Private EKS secures the API server but requires a real network path (peering/Transit Gateway/PrivateLink) for a remote hub, or syncs time out. Registration and connectivity are separate concerns.
• Terraform provisions all of it — the terraform-aws-eks v20 module (access entries), an IRSA/Pod Identity role, and a seed Argo CD via helm_release that then self-manages.
• The pattern transfers: every edge maps cleanly to AKS (Workload Identity, Key Vault, ACR, Application Gateway) and GKE (Workload Identity, Secret Manager, Artifact Registry, GCLB). Learn it once on AWS and you have learned it three times.