Argo CD Beyond the Big Three: OpenShift (GitOps Operator), Rancher & On-Prem/Edge Clusters

In a nutshell

You already know how to run Argo CD on a hyperscaler. This lesson is about everywhere else: the OpenShift cluster inside a bank, the Rancher-managed fleet in a data centre, the bare-metal rack under a raised floor, and the tiny K3s box on a factory wall with no internet at all. The good news for a beginner: the Argo CD inside is the same on every one of them. The surprises all live at the platform’s edges — how you install it, how you expose the UI, how people log in, where storage comes from, and how images get pulled.

The analogy: think of Argo CD as one engine, and each platform as a different chassis. Drop the same engine into a race car, a tractor, or a boat and it runs identically — but the engine mounts, the fuel line, the ignition switch and the cooling are wired up completely differently each time. You are not learning a new engine here; you are learning the wiring around it.

Level: Advanced · Time: ~58 min

Prerequisites: you can already install Argo CD and sync an Application on a cloud cluster, and you’re comfortable with kubectl, Helm/Kustomize, and what a pod’s securityContext is. After this you can: install Argo CD through the OpenShift GitOps operator, read an event log and fix a Security Context Constraint rejection, expose the UI with a Route or MetalLB instead of a cloud load balancer, register an RKE2/K3s spoke without the rotating-token trap, and mirror the images for a fully air-gapped install.

---

Every other lesson in this course quietly assumed a hyperscaler. Install Argo CD, expose it behind an Application Gateway or an ALB or a Google Cloud Load Balancer, wire SSO to Entra ID or Cognito or Google Workspace, pull secrets from Key Vault or Secrets Manager or Secret Manager, register a spoke and let the cloud’s IAM handle the token. That is the world of AKS, EKS and GKE — the “big three” — and it is a comfortable world, because the platform hands you a load balancer, a certificate, a DNS name, a CSI driver and an identity provider for free.

Most Kubernetes on Earth does not run there. It runs on OpenShift in a regulated enterprise, on Rancher-managed RKE2 in a data centre, on kubeadm bare-metal under a raised floor, on K3s at the edge in a factory or a retail store, and — more often than cloud engineers expect — in an air-gapped network with no route to the internet at all. Argo CD is genuinely cloud-agnostic; the same argocd-application-controller reconciles the same Application CRD on every one of these. But the edges — install, ingress, identity, storage, registry — are radically different, and each platform has one or two sharp gotchas that will cost you an afternoon the first time you meet them.

This lesson is the map for everything the big-three lessons left out. Read it as the multi-platform sibling of the multi-cloud lessons: instead of “here’s the AKS way, the EKS way, the GKE way,” it’s “here’s the OpenShift way, the Rancher way, the bare-metal way, the air-gapped-edge way” — with the same insistence on real manifests, real commands and real failure modes.

---

Why this matters

Here is the mental model to hold for the whole lesson: Argo CD’s core is identical everywhere; only the platform’s edges change. The server, the repo-server, the application-controller, the Redis cache, the Application/ApplicationSet/AppProject CRDs — byte for byte the same container images on OpenShift as on a Raspberry Pi. What changes is the ring of platform-specific concerns around Argo CD:

• How you install it. A kubectl apply and a Helm chart on most platforms; a declarative operator on OpenShift that you do not helm install at all.
• How you expose the UI. A cloud LoadBalancer on the big three; an OpenShift Route; MetalLB or a NodePort on bare-metal because a LoadBalancer Service otherwise hangs <pending> forever.
• How users log in. A cloud IdP’s OIDC on the big three; Dex wired to OpenShift OAuth on OpenShift; Dex + LDAP or an on-prem Keycloak where no cloud IdP is reachable.
• Whether pods are even allowed to start. On OpenShift, Security Context Constraints reject the exact securityContext a chart that “works on EKS” happily runs — the single most common porting failure in this whole space.
• Where storage and images come from. No cloud CSI (so local-path or Longhorn) and no public registry reachable (so a mirrored private registry).

Get this model right and every platform below is just “the core plus a known set of edges.” Get it wrong and you will keep asking “why does Argo CD behave differently here?” when the honest answer is that Argo CD behaves identically — it’s the platform underneath that differs.

---

The platform landscape: what actually changes

Before the deep dives, here is the whole territory in one view. This is the table to come back to; every later section expands one column of it.

Two honest framings before we dive in. First, these categories blur: OpenShift often is on-prem, Rancher often manages edge K3s, and air-gapped can describe any of them. The columns are axes of difference, not mutually exclusive products — a single deployment can be “OpenShift, on-prem, air-gapped” all at once, in which case you inherit every gotcha in those three columns. Second, the core promise still holds: a working Application manifest from the EKS lesson syncs unchanged on all of these. You are not relearning Argo CD; you are learning four rings of platform edges.

The whole territory reads left to right: one Argo CD control plane, then the four non-cloud platforms, each adding its own edge — OpenShift’s operator/SCC/Route, Rancher’s spoke registration, bare-metal’s MetalLB, and the air-gapped/edge private-registry-plus-pull. The numbered points are the decisions and failure lines each section unpacks.

The rest of the lesson walks the columns: OpenShift (and its SCC and Route rows), Rancher (and the Fleet question), bare-metal (the LoadBalancer and storage rows), air-gapped (the registry row), edge (the K3s row), and identity (the SSO row).

---

OpenShift GitOps: Argo CD as a Red Hat operator

On the big three you install Argo CD yourself — you own the Helm release, you pick the version, you upgrade on your schedule (that whole flow is the Installing Argo CD lesson). On OpenShift you do something structurally different: you install an operator, and the operator installs and continuously reconciles Argo CD for you. The product is called Red Hat OpenShift GitOps, and it is genuinely just upstream Argo CD wrapped in Red Hat packaging, support, and a Kubernetes-native install experience.

The pieces fit together like this:

Install the operator declaratively — which is itself GitOps-friendly, and the recommended way — with an OLM Subscription:

# openshift-gitops-operator subscription (apply once, cluster-wide)
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: openshift-gitops-operator
  namespace: openshift-operators
spec:
  channel: latest                 # or a pinned channel like gitops-1.14
  installPlanApproval: Automatic
  name: openshift-gitops-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace

oc apply -f subscription.yaml
# subscription.operators.coreos.com/openshift-gitops-operator created

# Watch the operator install, then confirm the default instance appears
oc get csv -n openshift-operators | grep gitops
# openshift-gitops-operator.v1.14.0   Red Hat OpenShift GitOps   1.14.0   Succeeded   (representative)

oc get pods -n openshift-gitops
# NAME                                                    READY   STATUS    RESTARTS   AGE
# openshift-gitops-application-controller-0               1/1     Running   0          3m
# openshift-gitops-applicationset-controller-...          1/1     Running   0          3m
# openshift-gitops-dex-server-...                         1/1     Running   0          3m
# openshift-gitops-redis-...                              1/1     Running   0          3m
# openshift-gitops-repo-server-...                        1/1     Running   0          3m
# openshift-gitops-server-...                             1/1     Running   0          3m   (representative)

Notice what you did not do: you never ran helm install, never applied install.yaml, never created the namespace. The operator did all of that when it reconciled a default ArgoCD custom resource named openshift-gitops. That is the fundamental shift — the install is declarative and self-healing. Delete the argocd-server Deployment and the operator recreates it; the desired state of Argo CD itself is now a CR under version control, not a Helm release you must remember to re-apply.

The ArgoCD custom resource

Everything you would set in Helm values.yaml on the big three, you set as fields on the ArgoCD CR here. This is the single most important object in OpenShift GitOps. A representative instance that turns on the OpenShift-native goodies:

apiVersion: argoproj.io/v1beta1
kind: ArgoCD
metadata:
  name: argocd
  namespace: gitops-tenant          # a self-managed instance in its own namespace
spec:
  # --- expose the UI via an OpenShift Route, not an Ingress ---
  server:
    route:
      enabled: true
      tls:
        termination: reencrypt       # passthrough | reencrypt | edge
        insecureEdgeTerminationPolicy: Redirect
  # --- SSO: Dex pre-wired to OpenShift's own OAuth ---
  sso:
    provider: dex
    dex:
      openShiftOAuth: true           # log in with your OpenShift identity
  # --- map an OpenShift group to Argo CD admin ---
  rbac:
    defaultPolicy: 'role:readonly'
    policy: |
      g, cluster-admins, role:admin
    scopes: '[groups]'
  # --- turn on the ApplicationSet controller and HA ---
  applicationSet: {}
  ha:
    enabled: false

The fields that matter, and how they differ from the vanilla/Helm world:

The openShiftOAuth: true line is the headline convenience. On AKS/EKS/GKE you register an app in Entra/Cognito/Google, copy a client ID and secret into argocd-cm, and hand-write a Dex OIDC connector. On OpenShift, that one boolean tells the operator to configure Dex against the cluster’s built-in OAuth server, so anyone who can log into OpenShift can log into Argo CD, and their OpenShift groups flow through as Argo CD RBAC subjects. No external IdP, no app registration, no secret to rotate.

Default instance vs your own instance

The operator creates one instance for you automatically. Know its coordinates:

# The default instance's admin password (the initial one)
oc get secret openshift-gitops-cluster -n openshift-gitops \
  -o jsonpath='{.data.admin\.password}' | base64 -d ; echo
# 8fJ2c...redacted...   (representative)

# The UI URL (an OpenShift Route, HTTPS by default)
oc get route openshift-gitops-server -n openshift-gitops \
  -o jsonpath='{.spec.host}' ; echo
# openshift-gitops-server-openshift-gitops.apps.mycluster.example.com  (representative)

The default instance is powerful — by design it is granted broad, cluster-wide permissions so it can deploy anywhere on the cluster. That is convenient for a platform team and dangerous for multi-tenancy: it is effectively a cluster-admin sitting behind your SSO. For anything with tenants, the pattern is to leave the default instance to the platform team and stand up namespace-scoped instances (a second ArgoCD CR in a tenant namespace, as in the manifest above) governed by AppProject boundaries — exactly the tenancy model from the AppProjects lesson, now enforced per-instance by the operator.

The other OpenShift-only nicety is console integration: the operator installs a dynamic console plugin, so the OpenShift web console gains a GitOps view and an app-launcher link straight to the Argo CD UI. It is cosmetic but it is the reason OpenShift admins often “find” Argo CD already running — the operator ships enabled in many install profiles.

---

The SCC problem: why your EKS chart won’t run on OpenShift

This is the deep one, and it is worth every paragraph, because it is the number-one reason a Helm chart that runs perfectly on EKS fails on OpenShift. The symptom is a Deployment that creates a ReplicaSet that creates… nothing, and an event that reads:

Error creating: pods "myapp-7c9d..." is forbidden: unable to validate against
any security context constraint:
[provider "restricted-v2": Forbidden: not usable by user or serviceaccount,
 ... runAsUser: Invalid value: 1000: must be in the ranges: [1000700000, 1000709999],
 ... capabilities.add: Invalid value: "NET_ADMIN": capability may not be added]  (representative)

No pod is ever created. kubectl get pods shows nothing; only oc get events or the ReplicaSet’s status reveals why. To fix it you must understand Security Context Constraints (SCCs).

What an SCC is

An SCC is OpenShift’s admission-time policy for what a pod’s securityContext is allowed to request. It predates and is stricter than upstream Pod Security Standards. Every pod is validated against the SCCs available to its ServiceAccount; if none of them permit what the pod asks for, the pod is refused at admission — it never schedules. Since OpenShift 4.11 the default SCC bound to ordinary workloads is restricted-v2, and it is deliberately paranoid:

The crucial behaviour of restricted-v2: it does not let you choose your UID. OpenShift allocates each namespace a UID range (an annotation like openshift.io/sa.scc.uid-range: 1000700000/10000), and restricted-v2 forces the pod to run as some UID inside that range, with GID 0 as the group. A chart that hardcodes runAsUser: 1000 is asking for a UID outside the allowed range, so restricted-v2 rejects it — and because 1000 is not root either, anyuid is overkill; the honest fix is usually to stop hardcoding at all.

Here is exactly what restricted-v2 forbids that a permissive cloud cluster allows:

The three ways to fix it

When a chart is rejected, you have three levers, in order of preference:

Fix 1 — fix the chart. Most well-maintained charts expose the pod and container securityContext as values. The OpenShift-friendly move is to null out the hardcoded UID/GID and let the platform decide:

# Helm values that stop fighting restricted-v2
podSecurityContext:
  runAsNonRoot: true
  # NO runAsUser / fsGroup — let OpenShift inject a UID from the namespace range
  seccompProfile:
    type: RuntimeDefault
securityContext:
  allowPrivilegeEscalation: false
  runAsNonRoot: true
  capabilities:
    drop: ["ALL"]

For images that write files, the durable image-level fix is to make the writable directories owned by group 0 and group-writable (chgrp -R 0 /data && chmod -R g=u /data in the Dockerfile), because OpenShift runs your container with a random UID but always GID 0. Get that right and the same image runs on EKS and OpenShift with no per-platform values.

Fix 2 — bind a looser SCC declaratively. The imperative way is oc adm policy add-scc-to-user anyuid -z myapp -n my-app, but that is a click you cannot commit to Git. The GitOps-native way is a plain RoleBinding to the auto-generated ClusterRole system:openshift:scc:<name> — which Argo CD can sync like any other manifest:

# Grant the 'myapp' ServiceAccount permission to USE the anyuid SCC
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: myapp-anyuid
  namespace: my-app
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:openshift:scc:anyuid   # OpenShift auto-creates one per SCC
subjects:
  - kind: ServiceAccount
    name: myapp
    namespace: my-app

This is the “SCC binding” you will write most often. It says: the myapp ServiceAccount may run pods that request any UID. Keep the subject as narrow as possible — one ServiceAccount in one namespace, never a broad group — because you are handing that workload a real privilege escalation relative to the default.

Fix 3 — a custom SCC. When a workload needs one specific thing (say, MustRunAsNonRoot but with RunAsAny fsGroup for an NFS mount) and anyuid is too broad, author a minimal SCC and bind it the same way:

apiVersion: security.openshift.io/v1
kind: SecurityContextConstraints
metadata:
  name: argo-workload-scc
allowPrivilegeEscalation: false
allowPrivilegedContainer: false
requiredDropCapabilities: ["ALL"]
runAsUser:
  type: MustRunAsNonRoot     # any non-root UID, not tied to the namespace range
seLinuxContext:
  type: MustRunAs
fsGroup:
  type: RunAsAny             # allow an arbitrary fsGroup for shared storage
volumes: ["configMap", "secret", "persistentVolumeClaim", "projected", "emptyDir"]
users: []                    # bind via a RoleBinding to system:openshift:scc:argo-workload-scc

Note the SCC’s fields sit at the top level, not under spec — SCCs are an unusual CRD in that respect. Bind it with the same RoleBinding-to-system:openshift:scc:argo-workload-scc pattern.

The takeaway for a platform engineer: treat SCC as a first-class part of any chart you port to OpenShift. When someone says “the Bitnami/community chart works on EKS but the pods never come up on OpenShift,” the diagnosis is almost always oc get events -n <ns> | grep -i "security context constraint", and the fix is one of these three — reaching for Fix 1 first and Fix 2 only when you truly cannot rebuild the image.

---

Routes vs Ingress: exposing the Argo CD UI on OpenShift

On the big three you expose argocd-server with an Ingress and let a cloud load-balancer controller (App Gateway, ALB, GCLB) give it a public address. OpenShift has Ingress, but its native, first-class object is the Route — and the operator creates one for you when you set spec.server.route.enabled: true. If you are running vanilla Argo CD on OpenShift (not the operator), you write the Route yourself:

apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: argocd-server
  namespace: argocd
spec:
  to:
    kind: Service
    name: argocd-server
  port:
    targetPort: https           # argocd-server's 8080 TLS port
  tls:
    termination: passthrough    # let argocd-server terminate its own TLS
    insecureEdgeTerminationPolicy: Redirect

The single decision that trips people up is TLS termination mode, and it interacts badly with Argo CD’s use of gRPC (the argocd CLI and the UI’s streaming both speak gRPC/HTTP2):

The classic failure — worth memorising — is: the UI loads fine in a browser but argocd login <route-host> hangs or fails with an rpc error or a protocol error. Ninety percent of the time the Route is edge-terminated, which downgrades the gRPC stream. The fix is to switch the Route to passthrough (or reencrypt), so the HTTP2/gRPC path survives to the pod. Put another way: browsers are forgiving, gRPC is not.

Route vs Ingress, side by side, for the times you must choose:

Because OpenShift ships the HAProxy router and a wildcard *.apps domain, a Route gives you a working, TLS-terminated URL with no cloud load balancer and no DNS work — one of the genuine ergonomic wins of the platform. If you want portability across OpenShift and other clusters, you can still use Ingress (OpenShift will honour it), but for the Argo CD UI specifically, the operator-managed Route with reencrypt or passthrough is the path of least resistance.

---

Rancher: managing Argo CD across RKE2/K3s — and the Fleet question

Rancher (now SUSE Rancher) is a multi-cluster management platform: it provisions and imports Kubernetes clusters — typically RKE2 (its hardened, CIS/FIPS-friendly distro) and K3s (its lightweight one) — and gives you one console over the fleet. Argo CD slots into a Rancher estate in the usual hub-and-spoke way: one Argo CD reaches many Rancher-managed clusters. But Rancher raises a question no cloud does, and you should answer it deliberately: why Argo CD and not Fleet?

Fleet vs Argo CD — the honest comparison

Fleet (fleet.cattle.io) is Rancher’s own built-in GitOps engine. It ships with Rancher, is managed from the Rancher UI, and is engineered for one thing extremely well: pushing bundles of manifests to thousands of clusters selected by labels. It is not a worse Argo CD; it is a different tool with a different centre of gravity.

Choose Fleet when you are all-in on Rancher, need fleet-scale bundle distribution to many edge clusters, and want zero extra tooling. Choose Argo CD when you want the app-centric UI and health model, sync waves and hooks, ApplicationSets, a team that already knows Argo, or portability off Rancher later. Plenty of shops run both: Fleet for Rancher’s own platform components, Argo CD for application teams. The wrong move is to adopt Argo CD and Fleet for the same workloads and let them fight over the same namespaces — pick one owner per resource, exactly as you would never point two Argo CD instances at the same app.

Registering RKE2/K3s clusters — the kubeconfig gotcha

Registering a Rancher-managed cluster in Argo CD is the standard cluster-registration flow — argocd cluster add <context> writes a labelled cluster Secret. But two Rancher-specific traps bite here.

Trap 1 — the RKE2/K3s kubeconfig points at localhost. The kubeconfig a node writes locally uses 127.0.0.1, which is meaningless to a remote Argo CD:

You must rewrite that server: to the cluster’s reachable API address (a node IP, a VIP, or a load-balanced control-plane endpoint) before argocd cluster add can use it:

# Copy the K3s kubeconfig and point it at the real API address
sudo cat /etc/rancher/k3s/k3s.yaml > k3s.yaml
sed -i 's#https://127.0.0.1:6443#https://10.10.0.11:6443#' k3s.yaml
export KUBECONFIG=$PWD/k3s.yaml
argocd cluster add default --name edge-k3s-01
# INFO ... ServiceAccount "argocd-manager" created ...
# Cluster 'https://10.10.0.11:6443' added   (representative)

Trap 2 — the Rancher-proxied kubeconfig uses a rotating token. If you download the kubeconfig from the Rancher UI, its server: may be the Rancher proxy (https://rancher.example.com/k8s/clusters/c-m-xxxxx) authenticated with a Rancher session token that expires. Register with that and Argo CD works for a day, then every sync flips to Failed with Unauthorized once the token rotates. Two durable fixes: enable Rancher’s Authorized Cluster Endpoint (ACE) to get a direct API path that bypasses the proxy, or (better for automation) let argocd cluster add create its own long-lived argocd-manager ServiceAccount on the downstream cluster — which is exactly what the command does when you point it at a direct endpoint. The rule: never register a spoke with a credential that rotates out from under Argo CD.

---

On-prem and bare-metal: no cloud to lean on

Strip away the cloud and two things you never thought about suddenly become your problem: how does a LoadBalancer Service get an IP, and where do PersistentVolumes come from. Argo CD installs identically — the manifests and the Helm chart from the installation lesson apply unchanged — but exposing its UI and giving your workloads storage are now on you.

The no-LoadBalancer problem

On a managed cloud, kubectl expose ... --type=LoadBalancer triggers a cloud controller that provisions a real LB in a minute. On bare-metal there is no cloud controller, so the Service sits <pending> forever:

kubectl get svc argocd-server -n argocd
# NAME            TYPE           CLUSTER-IP     EXTERNAL-IP   PORT(S)
# argocd-server   LoadBalancer   10.43.51.12    <pending>     80:31080/TCP,443:31443/TCP
#                                               ^^^^^^^^^ never resolves on bare-metal

Four ways to actually reach the UI without a cloud:

MetalLB is the standard answer when you want cloud-like LoadBalancer behaviour. You give it a pool of spare LAN addresses and an advertisement mode (Layer2 = one node answers ARP for the VIP; BGP = peer with your routers). Two small CRs:

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: argocd-pool
  namespace: metallb-system
spec:
  addresses:
    - 192.168.1.240-192.168.1.250     # spare, un-DHCP'd LAN IPs
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: argocd-l2
  namespace: metallb-system
spec:
  ipAddressPools:
    - argocd-pool

With MetalLB running, the same argocd-server LoadBalancer Service is assigned, say, 192.168.1.240, and the UI is reachable at that stable LAN IP — cloud-like behaviour, no cloud. On K3s specifically, you often need none of this: K3s ships ServiceLB (Klipper), so a LoadBalancer Service is handed the node IPs automatically. That is one reason K3s is so pleasant at the edge.

One gRPC caveat mirrors the OpenShift Route story: if you front Argo CD with ingress-nginx, the CLI’s gRPC needs either SSL passthrough (nginx.ingress.kubernetes.io/ssl-passthrough: "true") or the backend-protocol: HTTPS annotation with argocd-server serving TLS. A plain HTTP ingress will serve the browser UI but break argocd login.

Storage without a cloud CSI

Argo CD itself is effectively stateless — server, repo-server and application-controller keep no durable data, and Redis is an in-memory cache whose loss just forces a re-sync — so installing Argo CD needs no storage class at all. The storage problem is about the workloads Argo CD deploys: the moment a synced app has a PersistentVolumeClaim, it needs a StorageClass, and on bare-metal there is no managed-csi/gp3/standard-rwo waiting for you.

The failure mode to recognise: a synced app’s pod stuck Pending with pod has unbound immediate PersistentVolumeClaims, and the app stuck Progressing in Argo CD, because no default StorageClass exists. Install one (local-path for dev, Longhorn for prod) and mark it default. This is the on-prem equivalent of “the cloud just had a default StorageClass” — here you provide it.

What you lose without a cloud — and the replacement

None of this changes Argo CD — it changes the platform Argo CD sits on. Budget for it: standing up MetalLB, a StorageClass, an internal CA and a registry is the real “day zero” of on-prem GitOps.

---

Air-gapped installs: Argo CD with no internet

An air-gapped cluster has no route to the public internet — common in defence, finance, OT/industrial, and sovereign environments. Argo CD can run fully disconnected, but three assumptions in a normal install break, and you fix each one deliberately.

Break 1 — image pulls fail

The default install pulls three images from public registries. In an air-gap every one of them fails ImagePullBackOff:

The fix is to mirror them into your internal registry (Harbor/Nexus/Artifactory) from a connected jump host, using skopeo (no Docker daemon needed):

# From a connected host, copy each image into the private registry
skopeo copy docker://quay.io/argoproj/argocd:v2.13.3 \
  docker://registry.internal:5000/argoproj/argocd:v2.13.3
skopeo copy docker://ghcr.io/dexidp/dex:v2.41.1 \
  docker://registry.internal:5000/dexidp/dex:v2.41.1
skopeo copy docker://public.ecr.aws/docker/library/redis:7.4.1-alpine \
  docker://registry.internal:5000/library/redis:7.4.1-alpine
# (tags are illustrative — pin to exactly what your Argo CD version's install.yaml references)

Then override the image references so the install points at the mirror. Two clean approaches:

# Approach A — Kustomize image transformer over the upstream install.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: argocd
resources:
  - install.yaml            # a local copy of the pinned upstream manifest
images:
  - name: quay.io/argoproj/argocd
    newName: registry.internal:5000/argoproj/argocd
    newTag: v2.13.3
  - name: ghcr.io/dexidp/dex
    newName: registry.internal:5000/dexidp/dex
    newTag: v2.41.1
  - name: public.ecr.aws/docker/library/redis
    newName: registry.internal:5000/library/redis
    newTag: 7.4.1-alpine

# Approach B — Helm values (argo/argo-cd chart), the cleaner path for air-gap
global:
  image:
    repository: registry.internal:5000/argoproj/argocd
    tag: v2.13.3
dex:
  image:
    repository: registry.internal:5000/dexidp/dex
    tag: v2.41.1
redis:
  image:
    repository: registry.internal:5000/library/redis
    tag: 7.4.1-alpine

Break 2 — Helm/OCI repos are unreachable

Argo CD’s repo-server pulls Helm charts and OCI artifacts at sync time. Air-gapped, https://charts.bitnami.com and ghcr.io/... do not resolve, so syncs fail with failed to get repo or a TLS/DNS error. You must repoint every chart and OCI reference at your internal mirror (Harbor and Nexus both proxy Helm/OCI), register those as Argo CD repositories with credentials, and trust the mirror’s CA in the repo-server. The desired-state repos themselves must be an internal Git (a self-hosted GitLab/Gitea), not github.com.

Break 3 — external SSO and secret stores are unreachable

No Entra ID, no Okta, no Google — and no Key Vault / Secrets Manager / Secret Manager either. That reshapes two edges:

• Identity falls back to an on-prem IdP (Keycloak) or Dex + LDAP against internal Active Directory, or local users (next section).
• Secrets can no longer use the cloud-store path from the secrets lesson. The air-gap-friendly options are Sealed Secrets (encrypted ciphertext committed to your internal Git, decrypted by an in-cluster controller — no external calls) or a self-hosted Vault with ESO pointed at it. Both keep the entire secret lifecycle inside the perimeter.

The mental shift: in air-gap, everything Argo CD reaches out to must have an internal replica — registry, chart repo, Git, IdP, secret store, CA. Argo CD’s pull model actually helps here (it dials out to Git and registries you control, rather than something dialling in), but only once every external dependency has a local stand-in.

---

Edge and K3s: the pull model earns its keep

The edge — a store, a factory floor, a cell tower, a wind turbine — is defined by two constraints: small nodes and flaky connectivity. K3s is the usual distro because it is a single ~60 MB binary with a sane memory footprint, and it is where the GitOps pull model stops being a philosophical nicety and becomes a genuine operational advantage.

The architectural nuance worth internalising: there are two ways to do edge GitOps, and they fail very differently.

• Hub-into-edge (push over the network). One central Argo CD registers each edge cluster as a spoke and dials into its API server to reconcile. Clean to operate — until the edge link drops, at which point that cluster is simply unreachable and drifts unmanaged until connectivity returns. The hub cannot reconcile what it cannot dial.
• Argo-on-edge (pull). Argo CD (or a lightweight per-cluster agent) runs on the edge cluster and pulls desired state from Git whenever it has a link. When the network is down it holds the last-known-good state; when the link returns it reconciles to the latest commit on its own. Intermittent connectivity is survivable, because reconciliation is initiated from the edge outward, not the centre inward.

For anything genuinely disconnected or bandwidth-starved, prefer Argo-on-edge: the pull model means a site rides out an outage and self-heals to Git the moment it reconnects, with no operator and no reachable control plane. This is the same “pull beats push” argument from the GitOps-principles lesson, but at the edge it is not a preference — it is the difference between a site that recovers itself and a site that needs a truck roll. (The community argocd-agent project is emerging to make the lightweight on-edge footprint smaller still; today, a trimmed non-HA Argo CD per site is the pragmatic pattern.)

---

Identity without a cloud IdP

The big-three lessons wired SSO to a cloud IdP. Off-cloud you have three honest options, in rough order of how much you’d want them in production:

Keycloak is the on-prem default when you want cloud-grade SSO without a cloud. Argo CD talks to it directly — no Dex needed — via oidc.config in argocd-cm:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  url: https://argocd.internal.example.com
  oidc.config: |
    name: Keycloak
    issuer: https://keycloak.internal.example.com/realms/argocd
    clientID: argocd
    clientSecret: $oidc.keycloak.clientSecret   # resolves from argocd-secret
    requestedScopes: ["openid", "profile", "email", "groups"]
    requestedIDTokenClaims: {"groups": {"essential": true}}

Dex + LDAP is the choice when your source of truth is Active Directory and you want to reuse it directly. Dex is already in the Argo CD install; you just give it an LDAP connector in argocd-cm:

data:
  dex.config: |
    connectors:
      - type: ldap
        id: ldap
        name: Corporate LDAP
        config:
          host: ldap.internal.example.com:636
          bindDN: cn=argocd,ou=svc,dc=example,dc=com
          bindPW: $dex.ldap.bindPW           # from argocd-secret
          userSearch:
            baseDN: ou=people,dc=example,dc=com
            filter: "(objectClass=person)"
            username: uid
            idAttr: uid
            emailAttr: mail
            nameAttr: cn
          groupSearch:
            baseDN: ou=groups,dc=example,dc=com
            filter: "(objectClass=groupOfNames)"
            userMatchers:
              - userAttr: DN
                groupAttr: member
            nameAttr: cn

In both cases the group names Dex/Keycloak return (cn values, or Keycloak group claims) become the subjects in your argocd-rbac-cm policy — g, cn=platform,ou=groups,dc=example,dc=com, role:admin. Local users remain useful as a break-glass account even when SSO works: if the IdP is down (very possible in a small air-gapped site), a local admin login is your way back in. The OpenShift case is the happy exception to this whole section — openShiftOAuth: true gives you SSO with zero of this wiring, because the cluster is the IdP.

---

Going deeper

The sections above cover the day-one edges. Here are five sharper realities that bite once OpenShift, Rancher and air-gap move from a lab into a production platform — each a genuine gap the walkthrough above does not close.

The operator reconciles you back — don’t oc edit what it owns

On the big three you own the Helm release, so helm upgrade --set or a quick kubectl edit deploy/argocd-server sticks. On OpenShift GitOps it does not. The operator continuously reconciles every resource it generates from the ArgoCD CR, so a manual oc edit deployment openshift-gitops-server is reverted within seconds — the same self-healing you love for your apps, now applied to Argo CD itself. The rule is: the ArgoCD CR is the only supported surface. Everything you’d have set in Helm values.yaml goes there.

For a setting the CR does not model as a first-class field, use the escape hatches rather than fighting the reconcile loop:

# ArgoCD CR: the supported way to inject config the CR doesn't model
spec:
  extraConfig:
    "application.instanceLabelKey": "argocd.argoproj.io/instance"
  server:
    extraCommandArgs:
      - --request-timeout=600s

The upgrade discipline matters more than it looks. With installPlanApproval: Automatic, a new operator version rolls out on its own and can move your Argo CD to a new minor — a surprise you did not schedule. Set it to Manual, review the pending InstallPlan (oc get installplan -n openshift-operators), and approve upgrades on your maintenance window, exactly as you’d pin a Helm chart version elsewhere.

Getting the operator itself into an air-gapped OpenShift

The air-gap section above mirrored the three runtime images for a vanilla install. Disconnected OpenShift has an extra, earlier problem the operator model creates: OperatorHub can’t reach registry.redhat.io, so the operator won’t even appear to install. Two facts change here. First, the operand images are Red Hat builds pulled from registry.redhat.io/openshift-gitops-1/..., not quay.io/argoproj/... — a different set of names to mirror. Second, you mirror an operator catalog, not just images, and OpenShift needs to be told where the mirror lives.

The tool is oc-mirror driving an ImageSetConfiguration; the outputs are a CatalogSource (so OperatorHub sees the operator offline) plus an ImageDigestMirrorSet (IDMS) — the current object that redirects registry.redhat.io pulls to your internal registry (IDMS replaced the older, now-deprecated ImageContentSourcePolicy/ICSP):

# imageset-config.yaml — mirror just the GitOps operator and its bundle
apiVersion: mirror.openshift.io/v2alpha1
kind: ImageSetConfiguration
mirror:
  operators:
    - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.16   # match your OCP minor
      packages:
        - name: openshift-gitops-operator
          channels:
            - name: latest

# From a connected host, mirror to the internal registry (oc-mirror v2; flag to your version)
oc mirror --v2 --config imageset-config.yaml docker://registry.example.com/mirror
# ... then apply the generated CatalogSource + ImageDigestMirrorSet on the disconnected cluster:
oc apply -f cluster-resources/          # CatalogSource, IDMS (representative)

The mental model: in a disconnected OpenShift you mirror two products — the platform’s way of installing Argo CD (the catalog) and Argo CD’s runtime (the operand images) — where a vanilla air-gap only had the second.

Two Argo CD instances on one cluster: resource-tracking collisions

It is common to run the cluster-scoped default openshift-gitops instance (platform team) and a namespace-scoped tenant instance on the same cluster. The trap: Argo CD marks every resource it manages so it can tell “mine” from “not mine.” By default that mark is the shared app.kubernetes.io/instance label — which other tools (Helm, Kustomize) also write. Two instances using the same tracking scheme can each believe they own the same object and fight over it: endless OutOfSync flapping, or one instance pruning the other’s resources.

The fix is to make tracking unambiguous per instance. Prefer annotation tracking, which stamps argocd.argoproj.io/tracking-id (app name + the app’s own identity) instead of the collision-prone shared label:

# ArgoCD CR — annotation tracking avoids the shared instance label
spec:
  resourceTrackingMethod: annotation      # label | annotation | annotation+label

If you must keep label tracking, give each instance a distinct application.instanceLabelKey (via spec.extraConfig) so their marks never overlap. This is the same ownership hygiene the scaling and sharding lesson applies to multi-instance estates — on OpenShift it is simply more likely, because the operator hands you a powerful default instance whether you asked for one or not.

Apps in any namespace on the operator

By default an operator-managed instance only reconciles Application CRs that live in its own namespace — so tenants file tickets, or you centralise every app manifest. “Apps in any namespace” lets a tenant keep its Application next to its workloads. On the operator you don’t set controller flags by hand; you list the namespaces on the ArgoCD CR and the operator wires the --application-namespaces flag and labels the namespaces for you:

# ArgoCD CR — permit Applications to live in tenant namespaces
spec:
  sourceNamespaces:
    - team-a
    - team-b

The guard rail is that the governing AppProject must also allow those namespaces, or the app is refused — belt and braces so a tenant can’t smuggle an app past its project boundary:

# AppProject — the project must whitelist the same namespaces
spec:
  sourceNamespaces:
    - team-a

This pairs exactly with the multi-tenancy model from the AppProjects lesson: sourceNamespaces on the instance says where an app may live, the AppProject says what it may deploy.

Fleet-scale registration without argocd cluster add: RHACM’s GitOpsCluster

Running argocd cluster add by hand does not scale to a fleet, and on OpenShift it is often unnecessary. If the estate is managed by Red Hat Advanced Cluster Management (RHACM), a GitOpsCluster resource binds a Placement (a label-based cluster selector) to an Argo CD instance, and RHACM automatically creates and removes the Argo CD cluster Secrets as managed clusters join or leave the placement — no manual token wrangling, no rotating-credential trap:

apiVersion: apps.open-cluster-management.io/v1beta1
kind: GitOpsCluster
metadata:
  name: gitops-fleet
  namespace: openshift-gitops
spec:
  argoServer:
    argoNamespace: openshift-gitops
  placementRef:
    apiGroup: cluster.open-cluster-management.io
    kind: Placement
    name: all-prod-clusters        # selects managed clusters by label

An ApplicationSet then targets exactly those clusters with the cluster-decision-resource generator, which reads the Placement’s decisions (via a small acm-placement ConfigMap RHACM ships) rather than a static cluster list. This is the OpenShift-native answer to the hub-and-spoke registration problem the hub-spoke multi-cluster lesson solves manually — the placement is the spoke list, and it stays current on its own.

One more bare-metal nuance: MetalLB L2 funnels, BGP spreads

The MetalLB table above treats L2 and BGP as interchangeable advertisement modes. At scale they are not. In Layer 2 mode, exactly one node is elected to answer ARP for each VIP, so all traffic to that Service funnels through that single node, and failover is an ARP re-announcement (a few seconds of blip). That is completely fine for the Argo CD UI — low traffic, brief failover tolerated — but a real bottleneck for a high-throughput data-plane Service. BGP mode peers MetalLB with your routers and ECMP-spreads the VIP across many nodes, which is the answer for throughput — at the cost of needing cooperative routers and a little networking-team goodwill. Rule of thumb: L2 for control-plane and low-traffic UIs like Argo CD; BGP when a Service must actually scale past one node’s NIC.

---

Hands-on lab

Two config-level mini-flows, one per world: (A) OpenShift — install via the operator, expose with a Route, and fix an SCC rejection; (B) bare-metal — install vanilla Argo CD, expose via MetalLB/NodePort, and note the air-gap image-mirroring steps. These are written to be read and reasoned about even if you don’t have each platform handy; every manifest is real and every output is representative and labelled. Where a step needs a specific platform, it’s flagged.

No cluster is spun up for you here — the outputs below are representative of a correct run, not a live capture. Apply them on a real OpenShift cluster (or the free OpenShift Local / CRC) and a bare-metal/K3s cluster to see the actual results.

Flow A — OpenShift: operator, Route, and an SCC fix

A1 — Install the operator.

oc apply -f subscription.yaml       # the Subscription from earlier
# subscription.operators.coreos.com/openshift-gitops-operator created
oc get csv -n openshift-operators | grep -i gitops
# openshift-gitops-operator.v1.14.0 ... Succeeded   (representative)

What just happened: OLM pulled the operator and it reconciled a default ArgoCD instance into openshift-gitops. You installed Argo CD without ever touching a Helm chart.

A2 — Get the URL and password.

oc get route openshift-gitops-server -n openshift-gitops -o jsonpath='{.spec.host}{"\n"}'
# openshift-gitops-server-openshift-gitops.apps.crc.testing   (representative)
oc get secret openshift-gitops-cluster -n openshift-gitops \
  -o jsonpath='{.data.admin\.password}' | base64 -d ; echo
# r7Kd2...redacted   (representative)

What just happened: The operator created an OpenShift Route (no cloud LB) and stored the initial admin password in openshift-gitops-cluster. You can also just click “Cluster Argo CD” in the console app-launcher and log in with your OpenShift identity via openShiftOAuth.

A3 — Deploy a sample app, and watch SCC reject it. Point an Application at a chart that hardcodes a non-root UID (many community charts do). The Application syncs, but the workload’s pods never appear:

oc get application sample-app -n openshift-gitops
# NAME         SYNC STATUS   HEALTH STATUS
# sample-app   Synced        Progressing        <-- stuck; the Deployment exists but no pods
oc get events -n sample | grep -i "security context constraint"
# ... Error creating: pods "sample-..." is forbidden: unable to validate against
#     any security context constraint: ... runAsUser: Invalid value: 1000:
#     must be in the ranges: [1000700000, 1000709999]   (representative)

What just happened: restricted-v2 rejected the pod because the chart requested runAsUser: 1000, outside the namespace’s allocated UID range. Argo CD reports the app Synced (the manifests applied) but Progressing/Degraded (the pods never became healthy) — a textbook SCC failure.

A4 — Fix it the GitOps way. Commit a RoleBinding granting the app’s ServiceAccount the anyuid SCC (Fix 2), or better, null out the hardcoded UID in the chart values (Fix 1). The binding, synced by Argo CD:

# after committing the RoleBinding from the SCC section and syncing:
argocd app sync sample-app
oc get pods -n sample
# NAME               READY   STATUS    RESTARTS   AGE
# sample-6b9f...     1/1     Running   0          20s   (representative)
argocd app get sample-app | grep Health
# Health Status:     Healthy   (representative)

What just happened: With permission to run its requested UID (or with the UID no longer hardcoded), the pod passed SCC admission, scheduled, and went Healthy. You resolved a platform-specific admission failure declaratively — the fix itself is a manifest in Git.

A5 — Teardown.

argocd app delete sample-app --yes
oc delete rolebinding myapp-anyuid -n sample
oc delete subscription openshift-gitops-operator -n openshift-operators
oc delete csv -n openshift-operators -l operators.coreos.com/openshift-gitops-operator.openshift-operators
oc delete namespace openshift-gitops sample --ignore-not-found

Flow B — Bare-metal: vanilla install, MetalLB, and air-gap notes

B1 — Install Argo CD (identical to any cluster).

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
# ... many resources created ...
kubectl -n argocd rollout status deploy/argocd-server
# deployment "argocd-server" successfully rolled out   (representative)

What just happened: The core install is platform-agnostic — the same command you’d run on EKS. Nothing about bare-metal changed Argo CD itself.

B2 — Try a LoadBalancer and watch it hang.

kubectl patch svc argocd-server -n argocd -p '{"spec":{"type":"LoadBalancer"}}'
kubectl get svc argocd-server -n argocd
# argocd-server   LoadBalancer   10.43.51.12   <pending>   80:31080/TCP,443:31443/TCP   (representative)

What just happened: No cloud controller exists, so the external IP is <pending> indefinitely. This is the defining bare-metal moment.

B3 — Give it an IP with MetalLB (or fall back to NodePort).

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.14.8/config/manifests/metallb-native.yaml
kubectl -n metallb-system rollout status deploy/controller
kubectl apply -f metallb-pool.yaml          # the IPAddressPool + L2Advertisement from earlier
kubectl get svc argocd-server -n argocd
# argocd-server   LoadBalancer   10.43.51.12   192.168.1.240   80:31080/TCP,443:31443/TCP   (representative)

What just happened: MetalLB assigned 192.168.1.240 from your pool; the UI is now reachable at that stable LAN IP — cloud-like behaviour with no cloud. If you’d rather not run MetalLB, --type=NodePort and browsing to https://<node-ip>:31443 works immediately (and on K3s, ServiceLB would have assigned node IPs without any of this).

B4 — First login.

kubectl -n argocd get secret argocd-initial-admin-secret \
  -o jsonpath='{.data.password}' | base64 -d ; echo
# 4xTq...redacted   (representative)
argocd login 192.168.1.240 --username admin --insecure
# 'admin:login' logged in successfully   (representative)

What just happened: Same initial-admin-secret flow as any cluster. --insecure here is just to skip the self-signed cert prompt on a lab LAN — in production you’d front this with cert-manager and an internal CA.

B5 — Air-gap note (read, don’t run). In a disconnected build you’d have prefaced B1 by mirroring the three images (skopeo copy to registry.internal:5000/...) and installing with the Kustomize/Helm image overrides from the air-gap section, plus pointing Helm/OCI repos and Git at internal mirrors. The install steps are identical; only the sources move inside the perimeter.

B6 — Teardown.

argocd logout 192.168.1.240
kubectl delete -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
kubectl delete namespace argocd
kubectl delete -f https://raw.githubusercontent.com/metallb/metallb/v0.14.8/config/manifests/metallb-native.yaml

---

Practice challenges

Six exercises that escalate from “click around the operator” to “install into an air-gapped OpenShift.” Each is written to be reasoned through on paper; run them on OpenShift Local (CRC) and a K3s/bare-metal node to see the real output. Try each before opening the solution.

1 (beginner) — Find the default instance. After installing the OpenShift GitOps operator, retrieve the Argo CD UI URL and the initial admin password without touching a Helm chart.

<details> <summary>Solution</summary>

oc get route openshift-gitops-server -n openshift-gitops -o jsonpath='{.spec.host}{"\n"}'
oc get secret openshift-gitops-cluster -n openshift-gitops \
  -o jsonpath='{.data.admin\.password}' | base64 -d ; echo

Why: the operator creates the default instance in openshift-gitops, exposes it via a Route (no cloud LB), and stores the initial password in the openshift-gitops-cluster secret — three coordinates worth memorising. </details>

2 (beginner→intermediate) — Read an SCC rejection. A synced app shows Synced / Progressing but has zero pods. Produce the one command that reveals why, and name the exact field the chart got wrong.

<details> <summary>Solution</summary>

oc get events -n <ns> | grep -i "security context constraint"
# ... runAsUser: Invalid value: 1000: must be in the ranges: [1000700000, 1000709999]

Why: restricted-v2 allocates each namespace a UID range and forbids a hardcoded runAsUser: 1000 outside it. The app is Synced (manifests applied) but Progressing because admission refused the pods — they never scheduled, so kubectl get pods shows nothing. </details>

3 (intermediate) — Fix the SCC rejection in Git. Grant the myapp ServiceAccount permission to run its pods, declaratively (no oc adm policy), and say why this is still the second-best fix.

<details> <summary>Solution</summary>

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: myapp-anyuid
  namespace: my-app
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:openshift:scc:anyuid   # OpenShift auto-creates one ClusterRole per SCC
subjects:
  - kind: ServiceAccount
    name: myapp
    namespace: my-app

Why: a RoleBinding to system:openshift:scc:anyuid is a manifest Argo CD can sync, unlike the imperative oc adm policy add-scc-to-user. It’s second-best because anyuid lets the pod run as root — the best fix (Fix 1) is to null the hardcoded UID in the chart so restricted-v2 accepts it unprivileged. </details>

4 (intermediate) — Give a <pending> Service a real IP. On bare-metal, argocd-server as a LoadBalancer sits <pending> forever. Assign it a stable LAN IP with MetalLB in Layer 2 mode.

<details> <summary>Solution</summary>

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: argocd-pool
  namespace: metallb-system
spec:
  addresses:
    - 192.168.1.240-192.168.1.250     # spare, un-DHCP'd LAN IPs
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: argocd-l2
  namespace: metallb-system
spec:
  ipAddressPools:
    - argocd-pool

Why: there is no cloud controller to fulfil a LoadBalancer on bare-metal, so MetalLB assigns the IP itself. L2 mode has one node answer ARP for the VIP — fine for the Argo CD UI. On K3s you’d often need none of this, because ServiceLB (Klipper) assigns node IPs out of the box. </details>

5 (advanced) — The browser works but the CLI hangs. The Argo CD UI loads over an OpenShift Route, but argocd login <route-host> fails with an rpc error. Diagnose and fix, and explain the root cause in one sentence.

<details> <summary>Solution</summary>

# Switch the Route off edge-termination so the gRPC/HTTP2 stream survives to the pod
spec:
  tls:
    termination: passthrough        # or reencrypt; NOT edge
    insecureEdgeTerminationPolicy: Redirect

Why: the argocd CLI and the UI’s streaming use gRPC (HTTP/2); an edge-terminated Route downgrades the connection, breaking gRPC while a plain-HTTPS browser session still works. passthrough (or reencrypt) keeps HTTP/2 intact end-to-end. </details>

6 (advanced) — Install into a disconnected OpenShift. You must stand up OpenShift GitOps in an air-gapped cluster. List what you mirror and how each piece is made visible to the cluster — and note the one thing that differs from a vanilla air-gapped Argo CD.

<details> <summary>Solution</summary>

# imageset-config.yaml — oc-mirror pulls the operator catalog + operand images
apiVersion: mirror.openshift.io/v2alpha1
kind: ImageSetConfiguration
mirror:
  operators:
    - catalog: registry.redhat.io/redhat/redhat-operator-index:v4.16
      packages:
        - name: openshift-gitops-operator
          channels: [{ name: latest }]

Then oc mirror --v2 --config imageset-config.yaml docker://registry.example.com/mirror, and apply the generated CatalogSource (so OperatorHub sees the operator) plus ImageDigestMirrorSet (so registry.redhat.io pulls redirect to your mirror).

Why: the operator model adds a step a vanilla air-gap doesn’t have — you mirror an operator catalog, not just images, and the operand images come from registry.redhat.io/openshift-gitops-1/*, not quay.io/argoproj/*. You’re getting two products offline: OpenShift’s way of installing Argo CD, and Argo CD’s runtime. </details>

---

Common mistakes and troubleshooting

Every row here is a real failure mode from the platforms above. The states are genuine Argo CD/OpenShift/Kubernetes messages.

Three gotchas earn extra words:

1. SCC is the tax on every OpenShift port. The instinct after an SCC rejection is to reach for anyuid and move on. Resist it. anyuid lets the pod run as root, which is a real regression from restricted-v2 and will show up in your next security review. Spend the ten minutes to null out the hardcoded UID and make written directories GID-0-writable; then the same image runs unprivileged on OpenShift and the big three, and you never touch an SCC binding again. Reserve anyuid/custom SCCs for images you genuinely cannot rebuild.

2. gRPC is the silent Route/Ingress killer. The reason “the UI works but the CLI doesn’t” recurs on both OpenShift Routes and bare-metal ingress is the same: Argo CD’s CLI and streaming use gRPC (HTTP/2), and edge-terminating proxies love to downgrade it to HTTP/1.1. Whenever argocd login misbehaves against a URL whose browser UI is fine, suspect TLS termination before anything else — passthrough on a Route, ssl-passthrough on nginx.

3. Rotating credentials are a time bomb in cluster Secrets. The Rancher token trap is a specific case of a general rule: a spoke’s cluster Secret must hold a credential that outlives the token you happened to have in your shell. Rancher session tokens, cloud CLI tokens baked into a kubeconfig, and short-lived SA tokens all “work” at registration and fail silently later. Register with something durable — the argocd-manager ServiceAccount Argo CD creates, or an exec-plugin that mints tokens on demand — and your fleet doesn’t rot.

---

Common beginner mistakes

These are misconceptions, not symptoms — the wrong mental model that produces the failures in the troubleshooting table above. Fix the model and the symptoms stop recurring.

“Argo CD behaves differently on OpenShift / bare-metal.” It does not. The controller, repo-server, server, Redis and the Application/ApplicationSet/AppProject CRDs are byte-for-byte identical everywhere. When something “behaves differently,” it is the platform underneath — SCC admission, a missing StorageClass, an edge-terminated Route — not Argo CD. Right model: same engine, different chassis; diagnose the chassis.

“OpenShift GitOps is a Red Hat fork of Argo CD.” It’s upstream Argo CD, packaged and supported by Red Hat and installed by an operator. There is no divergent codebase to relearn — an Application from the EKS lesson syncs unchanged. Right model: the operator is a delivery and lifecycle wrapper, not a different product.

“anyuid is the fix for SCC errors.” It’s a fix, and usually the wrong first reach — it lets the pod run as root, a real regression from restricted-v2 that your next security review will flag. Right model: fix the chart first (null the hardcoded UID, make written dirs GID-0-writable); reserve anyuid for images you genuinely cannot rebuild.

“The UI loads, so the install is healthy.” Browsers speak plain HTTPS and are forgiving; the argocd CLI speaks gRPC (HTTP/2) and is not. An edge-terminated Route or a plain-HTTP nginx ingress serves the UI perfectly while argocd login hangs. Right model: a working browser proves nothing about the CLI — test both.

“I’ll just oc edit the Argo CD deployment to change a flag.” On the operator, your edit is reverted within seconds — it reconciles Argo CD the way Argo CD reconciles your apps. Right model: the ArgoCD CR is the only durable surface; unmodelled settings go through spec.extraConfig / spec.server.extraCommandArgs.

“Air-gapped just means mirroring the container images.” Images are one of six things Argo CD reaches for. Helm/OCI chart repos, the Git source, the SSO IdP, the secret store, and the trusted CA all need internal replicas too — and on OpenShift, so does the operator catalog. Right model: every external dependency needs an internal stand-in, not just the registry.

“I can register the Rancher cluster with the kubeconfig I downloaded.” That kubeconfig may point its server: at 127.0.0.1 (meaningless remotely) or carry a rotating Rancher session token that expires — the spoke works for a day, then every sync goes Unauthorized. Right model: register against a reachable API address with a durable credential (the argocd-manager ServiceAccount, or ACE), never a token that rotates out from under you.

“Bare-metal install needs a StorageClass for Argo CD.” Argo CD’s server, repo-server and controller are stateless, and Redis is an in-memory cache — the install needs no PersistentVolumes. Storage becomes a concern only for the workloads Argo CD deploys: the moment a synced app has a PVC, it needs a default StorageClass (local-path for dev, Longhorn for prod). See the RBAC and local-users lesson for keeping a break-glass admin when the IdP is the thing that’s down.

---

Cheat-sheet

OpenShift GitOps

On-prem / bare-metal exposure

Air-gap image mirror

---

Interview and exam questions

Q: On OpenShift, how is installing Argo CD different from a helm install on EKS? A: You don’t install Argo CD directly — you install the OpenShift GitOps operator via OLM (a Subscription), and the operator reconciles an ArgoCD custom resource that it uses to create and continuously self-heal the Argo CD deployment. The install becomes declarative and operator-managed: delete a component and the operator recreates it, and the version is chosen by the operator channel rather than a Helm chart version you pick.

Q: A Helm chart runs on EKS but its pods never appear on OpenShift. What’s your first diagnostic and the likely cause? A: oc get events -n <ns> | grep -i "security context constraint". The likely cause is that OpenShift’s default restricted-v2 SCC rejected the pod — usually because the chart hardcodes a runAsUser outside the namespace’s allocated UID range, requests an added capability, or wants privilege escalation. The app shows Synced (manifests applied) but Progressing/Degraded (no healthy pods).

Q: Give the three ways to fix an SCC rejection, best first. A: (1) Fix the chart/image — remove the hardcoded UID/fsGroup so OpenShift injects a valid UID, and make written dirs GID-0-writable. (2) Bind a looser SCC — a RoleBinding from the workload’s ServiceAccount to system:openshift:scc:anyuid (or nonroot-v2). (3) Author a custom SCC granting exactly what’s needed and bind it. Prefer (1); reach for (2)/(3) only when you can’t rebuild the image.

Q: Why does the argocd CLI sometimes fail against an OpenShift Route even though the browser UI works? A: The CLI uses gRPC (HTTP/2). If the Route is edge-terminated, the router can downgrade the connection and break gRPC while the browser’s plain HTTPS still works. Fix by using passthrough or reencrypt termination so the HTTP/2 stream survives to argocd-server.

Q: When would you use Rancher Fleet instead of Argo CD, and vice versa? A: Fleet when you’re all-in on Rancher and need to push bundles to thousands of clusters by label with no extra tooling, managed from the Rancher console. Argo CD when you want the app-centric UI and health/diff model, sync waves and hooks, ApplicationSets, native OIDC/RBAC, the wider Argo ecosystem, or portability off Rancher. Many shops run both — Fleet for Rancher platform components, Argo CD for app teams — but never both owning the same resources.

Q: A LoadBalancer Service is stuck <pending> on a bare-metal cluster. Why, and what are your options? A: There’s no cloud load-balancer controller to fulfil the Service, so the external IP never gets assigned. Options: install MetalLB (an IP pool + L2 or BGP advertisement) for real LoadBalancer behaviour, use a NodePort, front it with an ingress controller, or — on K3s — rely on the built-in ServiceLB (Klipper), which assigns node IPs automatically.

Q: What must you do to install Argo CD in an air-gapped environment? A: Mirror the three images (quay.io/argoproj/argocd, ghcr.io/dexidp/dex, public.ecr.aws/.../redis) into an internal registry with skopeo, override the image references (Kustomize images: or Helm global.image/dex.image/redis.image), point Helm/OCI chart repos and Git at internal mirrors with credentials and CA trust, and replace external SSO/secret stores with on-prem equivalents (Keycloak/Dex+LDAP, Sealed Secrets or self-hosted Vault).

Q: Why does the GitOps pull model matter more at the edge than in the cloud? A: Edge links are intermittent. A hub-into-edge push (a central Argo CD dialling into each edge API server) fails whenever the link is down — the hub can’t reconcile what it can’t reach. Argo-on-edge (pull) runs Argo CD or an agent on the edge cluster; it holds last-known-good state when offline and reconciles to the latest Git commit when the link returns, with no operator and no reachable control plane. At the edge that’s the difference between self-recovery and a truck roll.

Q: You registered a Rancher-managed cluster in Argo CD and it worked for a day, then every sync went Unauthorized. What happened? A: You almost certainly registered using the Rancher-proxied kubeconfig whose credential is a rotating session token. When it expired, Argo CD lost authorization to the spoke. Fix by registering against a durable path — Rancher’s Authorized Cluster Endpoint (ACE) or a direct API address — so argocd cluster add provisions its own long-lived argocd-manager ServiceAccount instead of relying on a token that rotates.

Q: argocd cluster add on an RKE2/K3s cluster fails with dial tcp 127.0.0.1:6443. Why? A: The RKE2/K3s kubeconfig (/etc/rancher/{rke2,k3s}/...yaml) has server: https://127.0.0.1:6443, which is meaningless from a remote Argo CD. Rewrite the server: field to the node’s reachable API IP (or a control-plane VIP) before adding the cluster.

Q: Does installing Argo CD itself require a StorageClass on bare-metal? A: No — Argo CD’s server, repo-server and application-controller are stateless, and Redis is an in-memory cache, so the install needs no PersistentVolumes. Storage becomes a concern for the workloads Argo CD deploys: any app with a PVC needs a default StorageClass, which on bare-metal you provide with local-path (dev) or Longhorn/Ceph/NFS (production).

Q: What does spec.sso.dex.openShiftOAuth: true buy you, and what’s the off-cloud equivalent? A: It tells the operator to configure Dex against OpenShift’s built-in OAuth server, so anyone who can log into OpenShift can log into Argo CD and their OpenShift groups become RBAC subjects — SSO with zero external IdP wiring. Off-cloud there’s no such shortcut: you wire Dex to LDAP/AD, point oidc.config at an on-prem Keycloak, or fall back to local users.

---

Glossary

• OpenShift GitOps — Red Hat’s supported distribution of upstream Argo CD, installed and lifecycle-managed by an operator rather than a Helm chart.
• OLM (Operator Lifecycle Manager) — OpenShift’s operator package manager; you install an operator by creating a Subscription.
• Subscription — the OLM object that says “install this operator from this channel”; installPlanApproval controls whether upgrades roll out automatically or wait for approval.
• CSV (ClusterServiceVersion) — the installed-operator record OLM creates; oc get csv shows an operator’s version and Succeeded/Failed status.
• ArgoCD CR — the custom resource (argoproj.io/v1beta1) that is your Argo CD config on OpenShift; every Helm value becomes a field here, and the operator reconciles it.
• extraConfig — the ArgoCD CR escape hatch for argocd-cm keys the CR doesn’t model as first-class fields.
• SCC (Security Context Constraints) — OpenShift’s admission-time policy for what a pod’s securityContext may request; stricter than upstream Pod Security Standards.
• restricted-v2 — the default SCC since OCP 4.11: random non-root UID from the namespace range, drop-ALL capabilities, no privilege escalation. The usual cause of “runs on EKS, rejected on OpenShift.”
• anyuid — a looser SCC that lets a pod run as any UID including root; grant it narrowly, via a RoleBinding to system:openshift:scc:anyuid.
• UID range — the block of user IDs OpenShift allocates per namespace (e.g. 1000700000/10000); restricted-v2 forces pods to run inside it, so hardcoded UIDs fail.
• Route — OpenShift’s native ingress object (route.openshift.io/v1), served by the built-in HAProxy router; TLS modes are edge, passthrough, reencrypt.
• passthrough / reencrypt / edge — Route TLS termination modes; use passthrough or reencrypt for Argo CD so the CLI’s gRPC survives (edge breaks it).
• openShiftOAuth — an ArgoCD CR flag that wires Dex to OpenShift’s built-in OAuth, so OpenShift logins and groups flow into Argo CD with no external IdP.
• Fleet — Rancher’s own GitOps engine (fleet.cattle.io), built to push Bundles to thousands of label-selected clusters; an alternative to Argo CD, not a worse one.
• RKE2 / K3s — Rancher’s Kubernetes distros: RKE2 is the hardened, CIS/FIPS-friendly server distro; K3s is the lightweight single-binary distro popular at the edge.
• ServiceLB (Klipper) — K3s’s built-in load balancer that hands LoadBalancer Services the node IPs, so bare-metal K3s needs no MetalLB.
• MetalLB — a load-balancer implementation for bare-metal that assigns real LAN IPs to LoadBalancer Services; L2 mode funnels a VIP through one node, BGP mode ECMP-spreads it across nodes.
• ACE (Authorized Cluster Endpoint) — Rancher feature giving a direct API path to a downstream cluster, bypassing the proxy whose token rotates.
• air-gapped — a cluster with no route to the public internet; every external dependency (images, charts, Git, IdP, secrets, CA) must have an internal replica.
• skopeo — a daemonless tool to copy container images between registries (skopeo copy docker://… docker://…), used to mirror Argo CD’s images into a private registry.
• oc-mirror — the OpenShift tool that mirrors operator catalogs and images for disconnected installs, driven by an ImageSetConfiguration.
• IDMS (ImageDigestMirrorSet) — the cluster object that redirects registry pulls to an internal mirror; replaces the deprecated ImageContentSourcePolicy (ICSP).
• CatalogSource — the object that makes a mirrored operator catalog visible to OperatorHub on a disconnected cluster.
• Dex — the identity broker bundled with Argo CD; bridges OIDC to connectors like LDAP/Active Directory when no cloud IdP exists.
• Keycloak — a self-hosted OIDC provider; the on-prem default when you want cloud-grade SSO with groups but no cloud, wired via oidc.config.
• Sealed Secrets — encrypts a Secret into ciphertext safe to commit to internal Git, decrypted by an in-cluster controller — no external calls, ideal for air-gap.
• RHACM — Red Hat Advanced Cluster Management; its GitOpsCluster + Placement auto-register managed clusters as Argo CD spokes, replacing manual argocd cluster add.
• resource tracking method — how Argo CD marks the resources it owns (label / annotation / annotation+label); use annotation to avoid two instances colliding on the shared app.kubernetes.io/instance label.

---

Key takeaways

• Argo CD’s core is identical on every platform; only the edges change. Install, ingress, identity, storage and registry differ across OpenShift, Rancher, bare-metal and edge — the controller and CRDs do not. Learn the edges, not a new Argo CD.
• OpenShift ships Argo CD as an operator. You install the OpenShift GitOps operator (OLM), and it reconciles an ArgoCD CR — a declarative, self-healing install with a Route and openShiftOAuth SSO built in, not a Helm release you own.
• SCC is the number-one OpenShift porting failure. restricted-v2 rejects hardcoded UIDs, added capabilities and privilege escalation that run fine on EKS. Fix the chart/image first (null the UID, GID-0-writable dirs); bind anyuid/a custom SCC only when you can’t rebuild.
• Routes and gRPC: expose the UI with a Route, and use passthrough/reencrypt termination — an edge-terminated Route breaks the argocd CLI’s gRPC even when the browser UI works.
• Rancher raises the Fleet question: Fleet for Rancher-native, fleet-scale bundle distribution; Argo CD for the app-centric UI, waves/hooks, ApplicationSets and ecosystem. Register RKE2/K3s spokes with a durable credential and a reachable API address, never a rotating Rancher token or 127.0.0.1.
• Bare-metal takes away the cloud freebies. No LoadBalancer (MetalLB/NodePort/K3s ServiceLB), no default StorageClass (local-path/Longhorn), no managed certs or registry — Argo CD is unchanged, but you must supply the platform underneath.
• Air-gapped means everything external needs an internal replica: mirror the three Argo images to a private registry and override them, point Helm/OCI/Git at internal mirrors, and swap cloud SSO/secret stores for Keycloak/LDAP and Sealed Secrets/Vault.
• At the edge, the pull model earns its keep: run Argo CD (or an agent) on each K3s site so it survives flaky links and self-heals to Git when connectivity returns — a hub dialling into the edge fails the moment the link drops.