Compliance & Audit in GitOps: Who Deployed What, Drift Evidence & Policy with OPA/Kyverno

In a nutshell

Picture your production cluster as a bank vault. In the old world anyone with a key could walk in, move things around, and leave — and afterwards you’d try to reconstruct who did what from a sign-in sheet nobody filled in. GitOps rebuilds the vault so the only door is a camera-covered airlock: every change is written down first (a Git commit), countersigned by a second person (a pull-request review), and carried in by a single trusted courier (Argo CD) who never improvises. Nobody else can reach the shelves. Compliance stops being a folder of screenshots you scramble to assemble at audit time and becomes a by-product the system emits whether you want it to or not.

This lesson takes that idea and makes it defensible to an auditor. You’ll learn to read Git history and Argo CD’s sync history as an audit log, to treat drift (the cluster quietly diverging from Git) as an unauthorized-change alarm, to prove who authored a change with signed commits and what artifact runs with signed images, and to enforce the rules automatically with policy-as-code — checked once in the pull request for fast feedback and again at admission as a gate nobody can skip. You finish by exporting the exact evidence bundle an auditor accepts.

Level: Expert · Time: ~57 min

Prerequisites: you should be comfortable with Argo CD Application/AppProject objects, automated sync with selfHeal/prune, and basic Kubernetes RBAC. After this you can: map GitOps artifacts to SOC 2 / ISO 27001 / PCI DSS / NIST controls, reconstruct “who deployed what,” enforce signed commits and images, write Kyverno and Gatekeeper policies (and reach for the newer in-tree option), and hand an auditor a complete, reproducible evidence bundle.

Ask an auditor what they spend their week doing and the answer is nearly always the same: chasing evidence. Who approved this change? When did it actually go to production? Who could have changed it, and how do you know they didn’t? In a click-ops world those answers live in someone’s memory, a ticketing system nobody fills in consistently, and a cluster whose current state is a mystery until you kubectl get it. Reconstructing the trail after the fact is slow, incomplete, and — worst of all — not tamper-evident.

GitOps quietly solves the hardest part of that problem, almost as a side effect. When the only way a change reaches production is a commit merged through a pull request and reconciled by Argo CD, your delivery pipeline is your audit trail. Every change is a signed, timestamped, attributable, reviewed record in Git; every deployment is recorded by Argo CD; and every out-of-band change lights up as drift. You do not build a compliance system alongside your deployment system — the deployment system already emits the evidence, because it cannot operate without producing it.

This lesson is about turning that latent evidence into a real, defensible compliance posture. We will map GitOps artifacts to the actual controls auditors test (SOC 2 CC8.1, ISO 27001, PCI DSS, NIST 800-53, SOX ITGCs), reconstruct “who deployed what” from Git history plus Argo CD’s own history and events, treat drift as the unauthorized-change control it is, lock provenance down with GPG-signed commits, and enforce compliance with policy-as-code using both OPA Gatekeeper and Kyverno — at the pull request and at admission. Everything targets Argo CD 2.13+/3.x on Kubernetes 1.29+.

There is no cluster attached to this lesson, so every command output below is labelled representative — it shows the shape to expect, not a live run. Every manifest, ClusterPolicy, ConstraintTemplate and AppProject is schema-correct with real fields; secrets and key IDs are placeholders you replace. Policy patterns are the upstream Kyverno/Gatekeeper canonical forms, not invented ones.

Why this matters

Change management is the control auditors care about most, because uncontrolled change is where breaches and outages come from. Every framework has a version of it: changes to production must be authorized, tested, approved, documented, and traceable to a person. The traditional way to satisfy that is a change-advisory board, tickets, and screenshots — a paper trail bolted onto a process that can be bypassed by anyone with cluster credentials. The paper says one thing; the cluster may say another; and proving they match is manual.

GitOps inverts this. The desired state lives in Git, and a controller — Argo CD — continuously makes the cluster match Git and nothing else. That single architectural choice hands you four properties auditors love, for free: attribution (every change is a commit by a known author), authorization (it merged through a reviewed PR), traceability (the history is immutable and diffable), and detection (anything the cluster does that Git didn’t say is drift). The job of this lesson is to make each of those properties legible to an auditor and to enforce the parts that are otherwise just convention.

The mental model to carry through: the pipeline is the evidence, and policy-as-code is the enforcement. Git gives you the tamper-evident record of intent; Argo CD gives you the record of execution and the drift signal; and OPA/Kyverno give you the machine-checked guarantees that what got deployed actually met the rules — checked twice, once at the PR (fast feedback) and once at admission (an un-bypassable backstop). Miss any one leg and you have a story with a hole an auditor will find.

The contrast with the old way is stark, and it is worth having this table ready because it is the “before/after” a compliance lead will immediately grasp:

Here is the whole chain, end to end. Read it left to right: a proposed change enters only as a signed, reviewed pull request; policy runs at PR time and again as an admission gate; Argo CD verifies the commit signature, syncs, and records it; drift detection watches continuously; and every stage drops an artifact into the evidence bundle.

The six badges mark the load-bearing controls: separation of duties is structural, not a policy document (1); policy runs twice on purpose (2); signed commits make provenance forgery-resistant (3); admission is the backstop that catches what shift-left misses (4); drift is the continuous unauthorized-change control (5); and the evidence assembles itself as a by-product (6). The rest of the lesson is those six ideas, in depth, with real manifests.

Git as a tamper-evident audit log

Start with the artifact every other control leans on: the Git commit. A commit is not just a diff. It is a record that binds what changed (the patch), when (the author and committer timestamps), who (author identity, and with signing, cryptographic proof of it), and — through the commit message and the linked pull request — why. Stack commits into a history and you have an append-only, content-addressed ledger: every commit’s hash includes its parent’s hash, so you cannot alter a historical commit without changing every hash after it. That is what “tamper-evident” means precisely — you can rewrite history, but you cannot do it invisibly, and branch protection plus mirrored remotes make even that detectable.

Map the anatomy of a commit-plus-PR onto the questions an auditor asks, and the fit is exact:

The strategic payoff is that this one artifact satisfies the change management control across every framework at once. You do not implement SOC 2 change management and separately implement ISO change management — you implement “all production change flows through a reviewed, signed PR reconciled by Argo CD,” and it maps onto all of them. Here is that mapping explicitly; this table is the one to show an auditor first, because it turns your pipeline into a controls narrative:

One honest caveat so you do not oversell it: Git is tamper-evident, not tamper-proof. Someone with force-push rights on a protected branch could rewrite history, and the fix is procedural — protected branches, required signatures, disallowed force-push, and a second mirrored remote (or the SIEM copy of the events) so any rewrite is detectable. Auditors accept “tamper-evident with detection” as a strong control; they do not accept “trust me.” The rest of this lesson is about making the evidence both complete and hard to forge.

Who deployed what: reconstructing the deployment history

“Who deployed what, when?” is the single most common audit question, and in GitOps it has a precise, multi-source answer. Git tells you what the intent was and who authored it; Argo CD tells you when that intent was actually applied to a cluster and by whom it was triggered. You need both, because a commit merging is not the same event as the change reaching a cluster — auto-sync may lag, a sync window may hold it, or a human may have triggered it manually.

Start with Git. The manifest path is the history of that workload:

# What is running in prod for checkout, and who last touched each file?
git log --oneline --decorate -- apps/checkout/prod/
# a1b9f3c (HEAD -> main, tag: prod-2026.07.14) chore: bump checkout to v3.4.1 (#812)
# 7d2e004 fix: raise checkout memory limit to 512Mi (#805)
# c40188a feat: add owner label to checkout workloads (#799)

# Who changed THIS line, in which commit, when?
git blame -L 18,24 apps/checkout/prod/deployment.yaml
# a1b9f3c2 (Priya Nair 2026-07-14 09:12:41 +0000 19)         image: acme.azurecr.io/checkout:v3.4.1

What this proves: every line of desired state is attributable to a commit, an author, and a timestamp, and the commit links to PR #812 with its approval. That is attribution and authorization in two commands. Now cross to Argo CD for the execution record — when that revision actually synced to the cluster:

# Argo CD's own record of what synced, when
argocd app history checkout-prod
# ID  DATE                           REVISION
# 6   2026-07-14 09:14:02 +0000 UTC  main (a1b9f3c)
# 5   2026-07-12 15:30:11 +0000 UTC  main (7d2e004)
# 4   2026-07-09 08:02:55 +0000 UTC  main (c40188a)

The history is stored on the Application itself in .status.history (bounded by spec.revisionHistoryLimit, default 10), so it is declarative and backup-able like everything else. Rolling back is argocd app rollback checkout-prod 5 — and note that the rollback is itself a recorded event. These sources answer different questions, and an auditor will want the right one for each:

Argo CD’s audit events

Be honest about what Argo CD is here: it is not a dedicated audit-log product with an immutable store. Its audit trail is two streams — Kubernetes Events it emits into the argocd namespace for Application activity, and the structured request logs the argocd-server writes for API calls (logins, syncs, RBAC decisions, terminate-op, rollbacks). Every meaningful action produces one or both. The event reasons you care about:

# Representative: Argo CD events for one application
kubectl get events -n argocd --field-selector involvedObject.name=checkout-prod \
  --sort-by=.lastTimestamp
# LAST SEEN   TYPE     REASON               OBJECT                    MESSAGE
# 4m          Normal   OperationStarted     application/checkout-prod  Initiated sync to main (a1b9f3c) by priya@acme.io
# 3m          Normal   OperationCompleted   application/checkout-prod  Sync operation to a1b9f3c succeeded

There is one operational trap that is really a compliance trap: Kubernetes Events are ephemeral. The API server garbage-collects them after roughly an hour (--event-ttl, default 1h). If your only copy of “who synced what” is kubectl get events, your audit trail evaporates every hour. The control is to ship events and server logs off-cluster to a SIEM with retention — and because logging destinations are cloud-specific, this is where the multi-cloud edge appears:

The immutable-store column matters more than it looks: an auditor’s first challenge to any log-based control is “could someone delete or edit these logs to hide their tracks?” WORM storage — Azure immutable blobs, S3 Object Lock in compliance mode, GCS bucket retention lock — answers that with “no, not even an administrator, until the retention period expires.” That converts your shipped Argo CD events from a convenience into defensible evidence.

Drift is evidence: continuous unauthorized-change detection

Here is the idea most teams miss: drift is not a nuisance, it is a control. Argo CD compares Git (desired) to the live cluster (actual) on every reconcile, a few minutes apart by default. When they diverge, the Application goes OutOfSync. In a properly run GitOps estate the only legitimate cause of OutOfSync is a commit that has not been synced yet — because humans are not supposed to touch the cluster directly. So any OutOfSync that is not explained by a pending commit is, by definition, an out-of-band change: someone ran kubectl edit, a controller mutated something, or an attacker altered a workload. That is exactly the “detect unauthorized changes to production” control (SOC 2 CC7.2, NIST SI-7, PCI Req 10 / 11.5) — and Argo CD runs it continuously, for free, on everything it manages.

Read the states through a compliance lens:

The distinguishing move is argocd app diff, which shows exactly what diverged. If the diff is empty but the app is OutOfSync, the cause is a pending revision; if the diff shows a changed field that no commit introduced, that is your out-of-band change, and it needs to become an alert and a ticket:

# What exactly drifted? (empty output = live matches the target revision)
argocd app diff checkout-prod
# ===== apps/Deployment checkout ======
# 27c27
# <       replicas: 3        # Git says 3
# ---
# >       replicas: 8        # someone scaled it live — out-of-band change

Alerting on drift (tie it to notifications)

Detection is only a control if someone is told. Argo CD Notifications turns a state change into a message to Slack/Teams and — crucially for audit — to a webhook you can point at your SIEM. There is a built-in trigger for degraded health and sync failures; for pure OutOfSync drift you write a small custom trigger. The config lives in argocd-notifications-cm:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-notifications-cm
  namespace: argocd
data:
  service.slack: |
    token: $slack-token          # resolved from argocd-notifications-secret
  # Custom trigger: fire when an app drifts out of sync
  trigger.on-out-of-sync: |
    - when: app.status.sync.status == 'OutOfSync'
      oncePer: app.status.sync.revision
      send: [drift-audit]
  # Built-in-style triggers you also want for audit
  trigger.on-health-degraded: |
    - when: app.status.health.status == 'Degraded'
      send: [drift-audit]
  trigger.on-sync-status-unknown: |
    - when: app.status.sync.status == 'Unknown'
      send: [drift-audit]
  template.drift-audit: |
    message: |
      :rotating_light: {{.app.metadata.name}} is {{.app.status.sync.status}}/{{.app.status.health.status}}.
      Possible out-of-band change on {{.app.spec.destination.server}}.
      Target revision: {{.app.status.sync.revision}}.
    webhook:
      siem:
        method: POST
        body: |
          {"event":"argocd_drift","app":"{{.app.metadata.name}}",
           "sync":"{{.app.status.sync.status}}","health":"{{.app.status.health.status}}",
           "revision":"{{.app.status.sync.revision}}"}

You subscribe an application (or a whole project by default subscription) with annotations, routing the audit copy to a security channel and the SIEM webhook:

metadata:
  annotations:
    notifications.argoproj.io/subscribe.on-out-of-sync.slack: sec-gitops-audit
    notifications.argoproj.io/subscribe.on-health-degraded.slack: sec-gitops-audit
    notifications.argoproj.io/subscribe.on-out-of-sync.siem: ""   # the webhook service

oncePer: app.status.sync.revision is the detail that keeps this usable — it fires once per drifting revision instead of every reconcile, so the SIEM sees one clean event per incident rather than a storm. Argo CD’s notifications engine is covered in depth in its own lesson elsewhere in the course; the compliance-specific point here is that the drift trigger is what makes drift auditable — a timestamped, retained event — rather than merely visible in the UI until the next reconcile.

selfHeal as enforcement — and when it is the wrong control

syncPolicy.automated.selfHeal: true upgrades drift from detected to auto-remediated: when live diverges from Git, Argo CD reverts the cluster back to Git on the next reconcile. As a control this is powerful — it means an out-of-band change to a self-healing app is not just alerted, it is undone within minutes, so the window of unauthorized state is tiny. For a regulated prod workload, selfHeal is usually the right posture.

But selfHeal has a sharp edge that is itself a compliance concern: it will happily revert a change you meant to keep. During a break-glass emergency, if an on-call engineer patches a live Deployment to stop an incident and selfHeal is on, Argo CD reverts the fix and the incident resumes — and now your emergency action left no durable state and fought your tooling. The rule: selfHeal enforces the invariant “the cluster equals Git,” so during any authorized deviation you must either land the change in Git first (so there is nothing to revert) or disable selfHeal on that one app for the window. We will make that explicit in the break-glass runbook below.

spec:
  syncPolicy:
    automated:
      prune: true
      selfHeal: true       # revert out-of-band changes automatically = enforcement
    syncOptions:
      - RespectIgnoreDifferences=true   # don't fight controllers you deliberately ignore

Think of drift handling as a ladder of increasing control strength — pick the rung each app’s risk warrants, and be able to name which rung you are on when an auditor asks how unauthorized change is controlled:

Signed commits and verified provenance

Attribution is only as strong as the identity behind it, and plain Git identity is trivially forged — git config user.email ceo@acme.io and every commit you push claims to be from the CEO. For a control that says “changes are attributable to a person,” that is a real gap. The fix is cryptographically signed commits: the author signs each commit with a GPG (or SSH/Sigstore) key, and Argo CD refuses to deploy a revision unless its top commit is signed by a key you have explicitly trusted. Now write access to Git is no longer sufficient to deploy — an attacker would also need a trusted private signing key, which lives on a hardware token or in a developer’s keyring, not in the repo.

Argo CD enforces this at the project level with AppProject.spec.signatureKeys. List one or more trusted GPG key IDs; any Application in the project must resolve a targetRevision whose tip commit is signed by one of them, or the sync is refused:

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: prod
  namespace: argocd
spec:
  description: Regulated production — signed commits required
  sourceRepos:
    - https://github.com/acme/platform-gitops.git
  destinations:
    - server: https://prod-eu.example.com
      namespace: 'checkout-*'
  clusterResourceWhitelist: []        # deny cluster-scoped resources by default
  signatureKeys:
    - keyID: 4AEE18F83AFDEB23          # a GPG key ID you have registered & trusted

The trusted public keys are managed cluster-wide (they live in the argocd-gpg-keys-cm ConfigMap) via the CLI, and the project only references key IDs:

What actually happens on sync, depending on the commit’s signature:

# Representative: an unsigned commit blocked by a signed-commits project
argocd app get checkout-prod
# ...
# Health Status:   Healthy
# Sync Status:     OutOfSync
# Conditions:      ComparisonError: target revision a1b9f3c in Git is not signed, but a signature is required

Two honest caveats. First, signature verification checks the tip commit of the resolved revision — pin targetRevision to a signed tag or a branch whose merges are all signed, and require signed commits in branch protection so an unsigned commit can never become the tip. Second, this is one leg of provenance; the other is signed images (cosign) verified at admission, covered in the policy section. Commit signing proves who authored the desired state; image signing proves what artifact runs. Together they make provenance end to end — and both tie back to the hardening lesson Hardening Argo CD: Least Privilege, Network Policy & Admission Control, which treats the signing keys themselves as high-value secrets.

Policy-as-code gates: OPA Gatekeeper and Kyverno

Signed, reviewed commits prove who and what, but they do not prove the change was compliant — a signed, approved PR can still deploy a privileged pod from an untrusted registry with no owner label if nothing checks. That is the job of policy-as-code: machine-checked rules that pass or fail a manifest against your compliance requirements. Two engines dominate the Kubernetes world, and it is worth knowing both because auditors and platform teams encounter each.

Kyverno writes policies as Kubernetes-native YAML — no new language — and can validate, mutate, generate resources, and verify image signatures. OPA Gatekeeper writes policies in Rego (a purpose-built policy language) via ConstraintTemplate + Constraint, and is the Kubernetes front end for the broader OPA ecosystem. The comparison that actually drives a choice:

Whichever you pick, the same principle applies as with drift and signing: enforce at admission, but check first at the PR. Admission control is the un-bypassable backstop — the API server itself rejects a non-compliant resource no matter who applies it (Argo CD, a Helm release, or a human with kubectl). But admission gives feedback late, only when the change tries to land. Running the same rules at PR time with a CLI (conftest, kyverno apply, gator test) gives the developer feedback in seconds, in the PR, before merge. You want both, and they are genuinely complementary:

Kyverno: the compliance policy set

A Kyverno ClusterPolicy is a set of rules; each rule matches resources and then validates, mutates, generates, or verifies images. The rule types map cleanly onto compliance needs:

Here is a real, schema-correct policy bundling the compliance requirements the brief calls for — require an owner label, forbid the mutable :latest tag, disallow privileged containers, and restrict images to your three cloud registries. These are the upstream Kyverno canonical patterns, not invented ones:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: baseline-compliance
  annotations:
    policies.kyverno.io/title: Baseline compliance for regulated workloads
spec:
  background: true                 # also scan existing resources → PolicyReports
  rules:
    # 1) Attribution: every workload must carry an owner label
    - name: require-owner-label
      match:
        any:
          - resources:
              kinds: [Pod]        # Kyverno autogen extends this to Deployments etc.
      validate:
        failureAction: Enforce     # 1.10+ per-rule form; older: spec.validationFailureAction
        message: "The 'owner' label is required for audit attribution."
        pattern:
          metadata:
            labels:
              owner: "?*"          # ?* = must be present and non-empty

    # 2) No mutable :latest tag — require a tag, then forbid :latest
    - name: require-image-tag
      match:
        any:
          - resources:
              kinds: [Pod]
      validate:
        failureAction: Enforce
        message: "An explicit image tag is required."
        pattern:
          spec:
            containers:
              - image: "*:*"
    - name: forbid-latest-tag
      match:
        any:
          - resources:
              kinds: [Pod]
      validate:
        failureAction: Enforce
        message: "The mutable ':latest' tag is not allowed."
        pattern:
          spec:
            containers:
              - image: "!*:latest"

    # 3) No privileged containers
    - name: disallow-privileged
      match:
        any:
          - resources:
              kinds: [Pod]
      validate:
        failureAction: Enforce
        message: "Privileged mode is disallowed."
        pattern:
          spec:
            =(ephemeralContainers):
              - =(securityContext):
                  =(privileged): "false"
            =(initContainers):
              - =(securityContext):
                  =(privileged): "false"
            containers:
              - =(securityContext):
                  =(privileged): "false"

    # 4) Allowed registries only (ACR / ECR / Artifact Registry)
    - name: restrict-registries
      match:
        any:
          - resources:
              kinds: [Pod]
      validate:
        failureAction: Enforce
        message: "Images must come from an approved registry."
        pattern:
          spec:
            containers:
              - image: "acme.azurecr.io/* | *.dkr.ecr.*.amazonaws.com/* | *-docker.pkg.dev/*"

A few load-bearing details. The =() in the privileged rule is Kyverno’s conditional anchor — “if this field exists, it must match” — which is how you say “privileged, if set, must be false” without forcing every pod to declare it. The ?* anchor means “must exist and be non-empty,” which is exactly what an attribution label needs. And you do not repeat the rule for Deployment, StatefulSet, DaemonSet, Job, CronJob — Kyverno’s auto-gen feature (pod-policies.kyverno.io/autogen-controllers) synthesizes the equivalent rules against the pod template of those controllers automatically. That single behavior is why a Pod-level policy actually governs your whole estate.

Read the policy back as a compliance catalog — requirement, the rule that enforces it, and the anchor that does the work — because this is the map you keep next to the framework controls:

The action is controlled per rule by validate.failureAction (Kyverno 1.10+); the older cluster-wide equivalent is spec.validationFailureAction, and per-namespace exceptions use spec.validationFailureActionOverrides:

The rollout discipline that keeps you from an outage: ship every new policy as Audit first. Let it run for a sprint, read the PolicyReports to see what would have been blocked, fix the offenders, then flip to Enforce. Going straight to Enforce on a cluster full of pre-existing violations blocks the next redeploy of half your workloads.

The verifyImages rule closes the supply-chain leg — only run images a trusted key signed:

    - name: verify-signed-images
      match:
        any:
          - resources:
              kinds: [Pod]
      verifyImages:
        - imageReferences:
            - "acme.azurecr.io/*"
          failureAction: Enforce
          attestors:
            - entries:
                - keys:
                    publicKeys: |-
                      -----BEGIN PUBLIC KEY-----
                      MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE...   # your cosign public key
                      -----END PUBLIC KEY-----

OPA Gatekeeper: the same control in Rego

The equivalent in Gatekeeper is two objects: a ConstraintTemplate that carries the Rego and defines a new Constraint kind, and a Constraint that instantiates it with parameters and a match. The canonical “required labels” template (used verbatim from the upstream docs so you can trust it):

apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: k8srequiredlabels
spec:
  crd:
    spec:
      names:
        kind: K8sRequiredLabels
      validation:
        openAPIV3Schema:
          type: object
          properties:
            labels:
              type: array
              items:
                type: string
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8srequiredlabels
        violation[{"msg": msg, "details": {"missing_labels": missing}}] {
          provided := {label | input.review.object.metadata.labels[label]}
          required := {label | label := input.parameters.labels[_]}
          missing := required - provided
          count(missing) > 0
          msg := sprintf("you must provide labels: %v", [missing])
        }

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels          # the kind the template defined
metadata:
  name: require-owner-label
spec:
  enforcementAction: deny        # deny | dryrun | warn
  match:
    kinds:
      - apiGroups: ["apps"]
        kinds: ["Deployment", "StatefulSet", "DaemonSet"]
  parameters:
    labels: ["owner"]

The component model and where the audit evidence comes from:

enforcementAction: dryrun is Gatekeeper’s equivalent of Kyverno’s Audit — same “observe before you enforce” discipline. And Gatekeeper’s audit controller periodically re-evaluates existing resources against every constraint and writes the failures to status.violations, so you can hand an auditor “here is every current violation of every policy” without waiting for the next deploy.

Testing policy at the PR (shift-left)

Both engines have offline CLIs so the exact same rules run in CI, blocking the merge before a bad manifest can even reach admission:

# Kyverno: run the cluster policy against rendered manifests in CI
kyverno apply baseline-compliance.yaml --resource ./rendered/ --audit-warn
# Applying 5 policy rule(s) to 12 resource(s)...
# pass: 10, fail: 2, warn: 0, error: 0, skip: 0
# policy baseline-compliance -> rule forbid-latest-tag FAILED: checkout uses acme.azurecr.io/checkout:latest

# Gatekeeper: gator tests constraints against files offline
gator test --filename ./rendered/ --filename ./policy/
# FAIL: require-owner-label  Deployment/checkout  you must provide labels: {"owner"}

# conftest: run raw Rego against config in CI
conftest test ./rendered/deployment.yaml -p ./policy/
# FAIL - deployment.yaml - main - image uses a disallowed registry: docker.io/library/nginx

A non-zero exit fails the CI job, the PR check goes red, and the change never merges — fast feedback, and a CI artifact you attach to the PR as evidence that policy passed at the time of the change.

Separation of duties: no human writes to prod

Separation of duties (SoD) is the control auditors probe hardest, because it is where insider risk and fraud live: no single person should be able to author a change and push it to production unchecked. In click-ops this is hard — anyone with cluster-admin can do both, and you enforce SoD with policy documents and hope. GitOps makes it structural: the only path to prod is a PR that a different person approves, merged into a branch that only Argo CD reads, and applied by Argo CD’s service account — which no human logs in as.

Lay out who can do what, and the separation is visible:

The keystone is the last row: Argo CD is the only writer to production, and it only ever applies what Git says. No human holds prod write credentials, so there is no one to bypass the review. That single fact maps directly onto the SoD controls:

To make it real you must remove human prod-write access — SoD is meaningless if a developer still has cluster-admin on the side. That means scoping Argo’s own RBAC (covered in RBAC: Local Users & Policies) so app teams get project-scoped sync/get at most, and scoping the reconcile boundary with AppProject (see AppProjects & Multi-Tenancy Boundaries). Branch protection does the Git half: require a review, require the author to be different from the approver, require signed commits, and forbid force-push. The two halves together are the control — and note it is the pull-based model that makes it possible, which is exactly why GitOps Principles: Push vs Pull matters here: a push pipeline needs prod credentials sitting in CI, re-opening the very hole SoD is meant to close.

Regulated delivery: change windows, break-glass, and data residency

Regulated environments add three requirements on top of “every change is reviewed”: deploys must respect change windows, genuine emergencies need a break-glass path that is still auditable, and data must stay in its jurisdiction.

Change windows are handled by AppProject.spec.syncWindows. A window is allow or deny, on a cron schedule, for a set of apps/namespaces/clusters. A deny window during business hours is a classic change-freeze:

spec:
  syncWindows:
    - kind: deny
      schedule: '0 9 * * 1-5'    # 09:00 Mon–Fri...
      duration: 8h               # ...for 8h = a 09:00–17:00 business-hours freeze
      applications:
        - '*'
      manualSync: false          # even manual syncs are blocked during the freeze
      timeZone: Europe/London

# Representative: confirm the freeze is active
argocd proj windows list prod
# ID  STATUS  KIND  SCHEDULE     DURATION  APPLICATIONS  MANUALSYNC
# 0   Active  deny  0 9 * * 1-5  8h        *             Disabled

Break-glass is the escape hatch for a real incident, and its whole point is that it stays auditable — the emergency change must leave the same trail a normal one does. The runbook:

The failure mode to design against is step 3 done as bare kubectl edit with no ticket and no backfill — that is an unauthorized change that also fights selfHeal. A good break-glass keeps the PR path even under pressure (a pre-approved emergency reviewer, a hotfix branch), so the trail is intact and Git stays the source of truth.

Data residency is where the cloud edge returns: keep both the workloads and the evidence in the required jurisdiction. Argo CD routes workloads to region-pinned clusters via AppProject destinations and cluster labels; the evidence stays in-region through the logging store you chose earlier:

Assembling the audit evidence (and reporting)

Everything above produces artifacts; the final skill is assembling them into the bundle an auditor accepts, and standing up the dashboards that show the posture continuously. The bundle is short because the pipeline did the work:

For continuous reporting rather than point-in-time export, the surfaces are:

# Representative: standing compliance posture from Kyverno PolicyReports
kubectl get clusterpolicyreport -o wide
# NAME                          KIND      PASS   FAIL   WARN   ERROR   SKIP   AGE
# cpol-baseline-compliance      Cluster   214    3      0      0       0      21d

# Drill into the failures — this is what the auditor wants to see closing over time
kubectl get polr -A -o jsonpath='{range .items[*].results[?(@.result=="fail")]}{.policy}{" "}{.rule}{" "}{.resources[0].name}{"\n"}{end}'
# baseline-compliance forbid-latest-tag legacy-cron
# baseline-compliance require-owner-label sandbox-debug

The point of the report surfaces is that compliance stops being a quarterly fire drill and becomes a number you watch: fail count on the ClusterPolicyReport trending to zero, drift alerts closing within an SLA, and every prod sync in argocd app history traceable to a signed, approved PR.

Going deeper

The core lesson placed policy in two spots — the PR and admission — as complementary halves. What it did not yet unpack is what actually happens to an Argo CD sync when an admission policy says no, why a Kyverno mutation can pin an app into permanent OutOfSync, and where the Kubernetes-native admission primitives are heading. These are the edges you hit the moment policy-as-code meets a real reconciler in production.

When admission policy fails an Argo CD sync

Admission control is “un-bypassable” precisely because it lives in the API-server request path — which means it also sits directly in Argo CD’s write path. When the application-controller applies a resource that a Kyverno ClusterPolicy (or a Gatekeeper Constraint) rejects, the apply call returns the webhook’s denial, the sync operation fails, and Argo surfaces the raw admission error on the Application:

# Representative: a sync blocked by an enforcing Kyverno policy
argocd app get checkout-prod
# ...
# Sync Status:   OutOfSync
# Operation:     Sync failed
# Message:       one or more objects failed to apply, reason: admission webhook
#                "validate.kyverno.svc-fail" denied the request: baseline-compliance:
#                forbid-latest-tag: 'The mutable '':latest'' tag is not allowed.'

The failure is recorded in .status.operationState (phase Failed) and emitted as an OperationCompleted event — so, usefully, a blocked non-compliant deploy is itself an audit event. But there is a sharp edge when the change spans sync waves. Argo applies waves in order and waits for each to reconcile before the next; if a resource in wave 1 is rejected at admission, the sync stops there and later waves never apply. A single policy violation can therefore leave an app partially synced — earlier waves in, later waves absent — which is a worse state than “nothing changed.” That is the fundamental trade-off, and it is exactly why shift-left is not optional:

A second, scarier coupling hides in that “availability” row. Every admission webhook has a failurePolicy; enforcing policies use Fail (fail closed) so a violation can’t slip through while the engine is down. But failurePolicy: Fail means that if the Kyverno or Gatekeeper webhook is itself unavailable, every create/update of a matched resource is rejected — and Argo CD syncs across the estate start failing with webhook-timeout errors that have nothing to do with your manifests. A policy-engine outage becomes a delivery outage. The mitigations are ordinary production hygiene: run the engine highly available (multiple replicas + a PodDisruptionBudget), exclude control-plane and Argo’s own namespaces from the webhook so a self-inflicted deadlock can’t wedge the platform, and reserve failurePolicy: Ignore for policies whose bypass you can tolerate. Sync failures caused this way are diagnosed like any other — see Troubleshooting Sync Failures, Hooks & Pruning — but the fix lives in the webhook config, not the app.

Kyverno mutate / generate vs Argo CD’s diff

mutate is the Kyverno feature that quietly breaks GitOps if you don’t expect it. A mutate rule rewrites a resource at admission — injecting a securityContext, a sidecar, default labels, an imagePullPolicy. The result is a live object carrying fields that are not in Git. Argo CD’s whole job is to compare Git (desired) against live (actual) and flag any difference, so those injected fields show up as a diff and the app goes — and stays — OutOfSync. Worse, with selfHeal on, Argo strips the injected fields to match Git, Kyverno re-injects them on the next admission, and you get a slow flap between two controllers.

The clean fix is ignoreDifferences scoped to the mutating field manager. Under server-side apply every field is owned by a field manager; Kyverno’s mutate writes as manager kyverno, so you can tell Argo to ignore precisely the fields Kyverno owns without listing paths by hand:

spec:
  ignoreDifferences:
    - group: apps
      kind: Deployment
      managedFieldsManagers:
        - kyverno          # ignore fields last set by Kyverno's mutate webhook
  syncPolicy:
    syncOptions:
      - RespectIgnoreDifferences=true   # so selfHeal won't fight the ignored fields

RespectIgnoreDifferences=true is the load-bearing companion (it appears in the selfHeal example earlier): without it, ignoreDifferences hides the diff from the view but selfHeal still reverts the field. You can also target a single mutated field with jsonPointers / jqPathExpressions. The deeper design lesson: prefer generate (which creates separate objects like a NetworkPolicy or ResourceQuota, not edits to your Argo-managed resource) over mutate on resources Argo owns, and when you must mutate, own the drift deliberately. This is the same drift machinery covered in Diffing, Drift & ignoreDifferences, and it interacts with Sync Waves & Resource Hooks — a generated dependency that isn’t in Git can change what “healthy” means for a wave.

ValidatingAdmissionPolicy: native CEL, no webhook

Kyverno and Gatekeeper are both admission webhooks — external pods the API server calls out to, with the availability coupling described above. Kubernetes now ships an in-tree alternative: ValidatingAdmissionPolicy (VAP), which runs policy as CEL expressions inside the API server itself — no webhook to deploy, secure, or keep alive. It graduated to GA in Kubernetes 1.30, so on 1.30+ clusters it is a first-class option for your most critical guardrails, the ones you never want to fail because a webhook pod is down.

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: require-owner-label
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - apiGroups: ["apps"]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["deployments"]
  validations:
    - expression: "has(object.spec.template.metadata.labels) && 'owner' in object.spec.template.metadata.labels"
      message: "The 'owner' label is required for audit attribution."
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: require-owner-label-binding
spec:
  policyName: require-owner-label
  validationActions: [Deny, Audit]   # Deny blocks; Audit also records to the audit log
  matchResources:
    namespaceSelector: {}             # every namespace

VAP is validate-only for now (the MutatingAdmissionPolicy counterpart is newer and not yet GA as of early 2026 — treat it as preview in a regulated estate), it has no PolicyReport ecosystem, and complex logic is harder in raw CEL than in Kyverno YAML. So it does not replace Kyverno or Gatekeeper — but it is the right tool for a handful of un-killable invariants, and the ecosystems are converging: both Kyverno and Gatekeeper can now generate VAPs from their own policies, letting you author in the tool your team knows while enforcement runs in-tree with no webhook in the hot path.

Supply-chain provenance beyond a signature

The main lesson verifies an image signature (proof the artifact wasn’t swapped). The next rung is verifying attestations — signed in-toto predicates that travel with the image and describe how it was built: SLSA build provenance, an SBOM, a passing vulnerability scan. cosign attaches them (cosign attest --type slsaprovenance --predicate provenance.json <image>), and Kyverno’s verifyImages can require them, not just a bare signature:

    - name: require-slsa-provenance
      match:
        any:
          - resources:
              kinds: [Pod]
      verifyImages:
        - imageReferences: ["acme.azurecr.io/*"]
          failureAction: Enforce
          attestations:
            - type: https://slsa.dev/provenance/v1
              attestors:
                - entries:
                    - keys:
                        publicKeys: |-
                          -----BEGIN PUBLIC KEY-----
                          MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE...
                          -----END PUBLIC KEY-----
              conditions:
                - all:
                    # predicate field paths depend on your provenance format
                    - key: "{{ builder.id }}"
                      operator: Equals
                      value: "https://example.com/acme-ci@refs/heads/main"

Now “only signed images run” strengthens to “only images built by our pipeline, provably, run” — which is what SLSA-style controls and the integrity requirement in NIST SI-7 are really asking for. Commit signing (signatureKeys) covers who wrote the desired state; image signing covers what runs; attestations cover how it was made — the three together are end-to-end provenance an auditor can follow from a developer’s keyboard to a running container.

Hands-on lab

This lab builds the compliance stack at the config level — every manifest is real and schema-correct, and every output is labelled representative because there is no cluster attached. You can apply these against any Argo CD 2.13+/3.x install with Kyverno installed to make them live. We will: (1) require signed commits on a prod project, (2) add the Kyverno compliance policy, (3) wire drift alerting, and (4) assemble the evidence bundle.

Step 1 — Require signed commits on the prod AppProject.

# Trust the release signing key cluster-wide, then confirm
argocd gpg add --from ./release-signing.pub
argocd gpg list
# KEYID             TYPE  IDENTITY
# 4AEE18F83AFDEB23  rsa   Acme Release Signing <release@acme.io>

# prod-appproject.yaml — signatureKeys makes unsigned commits un-deployable
apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: prod
  namespace: argocd
spec:
  sourceRepos: ["https://github.com/acme/platform-gitops.git"]
  destinations:
    - server: https://prod-eu.example.com
      namespace: 'checkout-*'
  clusterResourceWhitelist: []
  signatureKeys:
    - keyID: 4AEE18F83AFDEB23
  syncWindows:
    - kind: deny
      schedule: '0 9 * * 1-5'
      duration: 8h
      applications: ['*']
      manualSync: false
      timeZone: Europe/London

What just happened: any Application in prod now refuses to sync a revision whose tip commit is not signed by the trusted key — provenance enforced by the platform, not by convention. The deny sync window adds a business-hours change freeze.

Step 2 — Apply the Kyverno compliance policy in Audit, then Enforce.

# Apply the policy (start in Audit — see the earlier baseline-compliance.yaml)
kubectl apply -f baseline-compliance.yaml
# clusterpolicy.kyverno.io/baseline-compliance created

# After a background scan, read what WOULD be blocked
kubectl get clusterpolicyreport cpol-baseline-compliance -o wide
# NAME                        PASS  FAIL  WARN  ERROR  SKIP  AGE
# cpol-baseline-compliance    118   6     0     0      0     3m

What just happened: background: true scanned existing workloads and produced a ClusterPolicyReport showing six pre-existing violations — the list you fix before flipping failureAction to Enforce, so enforcement never breaks a legitimate redeploy.

Step 3 — Prove enforcement blocks a non-compliant deploy.

# Representative: try to run a privileged, :latest, no-owner pod under Enforce
kubectl run bad --image=docker.io/nginx:latest --privileged
# Error from server: admission webhook "validate.kyverno.svc-fail" denied the request:
# baseline-compliance:
#   require-owner-label: 'The ''owner'' label is required for audit attribution.'
#   forbid-latest-tag: 'The mutable '':latest'' tag is not allowed.'
#   disallow-privileged: 'Privileged mode is disallowed.'
#   restrict-registries: 'Images must come from an approved registry.'

What just happened: four controls fired at admission in one shot — attribution, no mutable tag, no privilege, approved registry — and the request was rejected regardless of who sent it. That is the un-bypassable backstop.

Step 4 — Wire drift alerting to Slack and the SIEM.

# Apply the notifications config (see argocd-notifications-cm above) and subscribe the app
kubectl apply -f argocd-notifications-cm.yaml
kubectl annotate app checkout-prod -n argocd \
  notifications.argoproj.io/subscribe.on-out-of-sync.slack=sec-gitops-audit
# application.argoproj.io/checkout-prod annotated

# Simulate an out-of-band change and watch drift appear
kubectl scale deploy/checkout -n checkout-eu --replicas=8   # someone bypasses Git
argocd app get checkout-prod
# Sync Status:  OutOfSync   (drift detected → notification fired to sec-gitops-audit + SIEM)

What just happened: an out-of-band scale produced OutOfSync, which fired the on-out-of-sync trigger once (thanks to oncePer) to the security channel and the SIEM webhook — drift became a timestamped, retained audit event.

Step 5 — Assemble the evidence bundle.

# The four artifacts an auditor asks for, produced in four commands
git log --format=fuller -- apps/checkout/prod/ > evidence/01-git-history.txt
argocd app history checkout-prod            > evidence/02-sync-history.txt
kubectl get polr,cpolr -A -o yaml           > evidence/03-policy-reports.yaml
kubectl get appproject prod -n argocd -o yaml \
  cm argocd-rbac-cm -n argocd -o yaml       > evidence/04-access-controls.yaml
# Bundle: change history + authorship, what synced when, policy compliance, access controls

What just happened: the entire change-management, monitoring, and access-control story exported in four commands — because the pipeline emitted the evidence as it ran.

Teardown.

# Remove lab objects (config-only; no billable cloud resources were created)
kubectl delete clusterpolicy baseline-compliance
kubectl delete -f prod-appproject.yaml
kubectl delete cm argocd-notifications-cm -n argocd
argocd gpg rm 4AEE18F83AFDEB23
rm -rf evidence/

Common mistakes and troubleshooting

The failure modes here are subtle because a misconfigured compliance control looks the same as a working one until an auditor — or an attacker — finds the gap. Use real Argo CD/Kyverno states to diagnose.

Three gotchas cost the most audit credibility:

1. Audit mode mistaken for enforcement. A Kyverno policy in Audit (or a Gatekeeper Constraint in dryrun) records violations but allows them. Teams ship the policy, see it in the report, and believe production is protected — while non-compliant workloads keep deploying. Audit is a discovery stage, not a control. The control is Enforce/deny, and an auditor will ask which mode you are in. Roll out in Audit, then flip, and keep the flip in Git so the change to Enforce is itself an approved, dated commit.

2. Ephemeral evidence. The most common real hole is that the audit trail exists only in places that expire: Kubernetes Events (GC’d hourly), a local terminal’s scrollback, or a Slack channel with a 90-day retention on a free plan. If the evidence is not in a retained, ideally immutable store, you do not have a control — you have a demo. Ship events and logs to a SIEM with WORM retention before you claim the monitoring control, because the auditor’s question is always “show me last quarter,” not “show me now.”

3. SoD with a side door. GitOps SoD is only real if humans genuinely cannot write to prod. The classic finding is a beautifully reviewed PR pipeline sitting next to a cluster-admin kubeconfig that three engineers still use “for emergencies.” That side door voids the entire separation-of-duties narrative. Enforcing SoD means removing the human prod-write path (scope Argo RBAC and AppProject, delete standing admin credentials) and alerting on any manual change — not just building the nice path and leaving the old one open.

Practice challenges

Work these in order — they climb from “read the evidence” to “keep enforcement from wedging your platform.” Every answer is config-only; there is no cluster attached, so treat any output as representative.

1 · Beginner — reconstruct “who deployed what.” For the service checkout, give the two commands that together answer “who authored the change now running in prod, and when did it actually sync?”

<details> <summary>Show solution</summary>

git log --oneline -- apps/checkout/prod/     # authored intent + author + PR link
argocd app history checkout-prod             # when each revision actually synced

Add git blame -L 18,24 apps/checkout/prod/deployment.yaml to attribute an exact line.

Why: intent (Git) and execution (Argo CD) are two different events — a merge is not the same moment as the change reaching the cluster — so a complete answer always needs both sources. </details>

2 · Beginner — write the attribution rule. Write the Kyverno validate rule that makes a non-empty cost-center label mandatory on every Pod, enforced at admission.

<details> <summary>Show solution</summary>

- name: require-cost-center
  match:
    any:
      - resources:
          kinds: [Pod]
  validate:
    failureAction: Enforce
    message: "A non-empty 'cost-center' label is required."
    pattern:
      metadata:
        labels:
          cost-center: "?*"

Why: ?* means “must exist and be non-empty” — the correct anchor for any attribution label — and Kyverno auto-gen extends a Pod rule to Deployments, StatefulSets, Jobs, etc. automatically, so one rule governs the estate. </details>

3 · Intermediate — enforce signed commits. Make the prod AppProject trust GPG key 4AEE18F83AFDEB23 and refuse unsigned commits. What condition does an unsigned tip commit produce?

<details> <summary>Show solution</summary>

argocd gpg add --from ./release-signing.pub   # trust the public key cluster-wide

spec:
  signatureKeys:
    - keyID: 4AEE18F83AFDEB23

An unsigned tip revision is refused; the app shows ComparisonError with a condition containing ...is not signed, but a signature is required.

Why: verification is a project-level control that checks the resolved tip commit, so Git write access alone can no longer ship a change — the attacker would also need a trusted private signing key. </details>

4 · Intermediate — roll a new policy out safely, then make drift durable. You’re adding disallow-privileged to a cluster that already runs privileged pods. Describe the two-phase rollout and the command that shows what would break — then name the piece that turns a drift event into a retained record.

<details> <summary>Show solution</summary>

Ship with failureAction: Audit and background: true; read kubectl get cpolr,polr -A to list pre-existing fails; fix the offenders; then flip to failureAction: Enforce — as a dated commit in Git. For durability, a custom on-out-of-sync notification trigger with oncePer: app.status.sync.revision routes one event per incident to a SIEM webhook (see Notifications: Slack, Teams & Webhooks).

Why: going straight to Enforce on a dirty cluster blocks the next redeploy of every already-violating workload; oncePer stops a reconcile storm from drowning the SIEM. </details>

5 · Advanced — stop mutate-induced drift. A Kyverno mutate injects a logging sidecar into every Deployment. Your apps are now permanently OutOfSync and selfHeal keeps flapping. Fix it without disabling the policy.

<details> <summary>Show solution</summary>

spec:
  ignoreDifferences:
    - group: apps
      kind: Deployment
      managedFieldsManagers: ["kyverno"]
  syncPolicy:
    syncOptions:
      - RespectIgnoreDifferences=true

Why: the injected fields aren’t in Git, so Argo correctly sees a diff; telling Argo to ignore fields owned by Kyverno’s field manager (and making selfHeal respect that with RespectIgnoreDifferences) resolves the fight without loosening the control. </details>

6 · Advanced — an enforcement outage that wedges delivery. Your Kyverno admission webhook (failurePolicy: Fail) crashed, and now every Argo CD sync fails with a webhook-timeout error — even for compliant manifests. Explain the coupling and give two mitigations.

<details> <summary>Show solution</summary>

An enforcing webhook fails closed, so while the engine is unavailable the API server rejects all matched create/update calls — including Argo’s applies — and syncs fail estate-wide. Mitigations: (a) run Kyverno HA (multiple replicas + a PodDisruptionBudget) and exclude kube-system/argocd so a self-inflicted deadlock can’t wedge the platform; (b) move the few un-killable invariants to an in-tree ValidatingAdmissionPolicy (no webhook in the path); or scope failurePolicy: Ignore for policies whose momentary bypass you can tolerate.

Why: with failurePolicy: Fail, enforcement availability becomes a delivery dependency — design the webhook so a policy-engine outage can’t take the whole estate offline. </details>

Common beginner mistakes

These are misconceptions about the model, distinct from the symptom→fix table above. Fixing the mental model is what stops the mistake from recurring.

“We use Argo CD, so we’re automatically compliant.” GitOps emits the raw material of compliance, but the tool is the substrate, not the certificate. The posture only holds when the whole chain is present: a signed, reviewed PR; policy checked at the PR and at admission; drift alerted and retained; and human prod-write access actually removed. Miss any one leg and there’s a hole an auditor will find. Right model: Argo CD is necessary, not sufficient — you still assemble the controls on top of it.

“Signed commits mean my images are trusted too.” These are two independent provenance legs. signatureKeys on the AppProject proves who authored the desired state; cosign + Kyverno verifyImages proves what artifact actually runs. A perfectly signed commit can still reference an unsigned, tampered image. Right model: enforce commit signing and image signing separately — and, for the strongest posture, image attestations (how it was built) on top.

“OutOfSync means something is broken.” OutOfSync is a signal, not a verdict. Run argocd app diff: an empty diff means a revision is merged but not yet applied (benign, transient); a changed field that no commit introduced is the out-of-band change you must alert on. Right model: triage every OutOfSync into “pending revision” vs “unauthorized change” — only the second is an incident.

“Policy-as-code is just a CI linter.” CI checks are fast and developer-friendly but bypassable — a force-merge or a direct kubectl apply sails straight past them. Admission control is the un-bypassable half, but it gives feedback late and can fail a sync mid-wave. Right model: run the same rules in both places; they answer different guarantees (fast feedback vs. a gate nobody can skip), and you need both.

“Kyverno vs Gatekeeper is a winner-take-all choice, and one is objectively better.” They’re engines with different ergonomics — Kyverno is Kubernetes-native YAML and can generate resources; Gatekeeper is Rego and shares a lineage with non-K8s OPA. Teams even run both, and both can now emit native ValidatingAdmissionPolicy objects. Right model: choose by your team’s skills and needs, not a leaderboard, and treat the in-tree VAP as a third tool for un-killable invariants.

“To undo a bad deploy, I’ll delete the commit and force-push the history clean.” Rewriting history is the one move that destroys the audit trail — and it’s exactly what auditors flag as a broken change-management control. Right model: roll forward. git revert creates a new, attributable commit that undoes the change, and argocd app rollback re-points the app at an earlier revision; both are themselves recorded events, so the trail stays intact.

Glossary

Plain-language definitions of the terms this lesson leans on. Skim it first if the compliance vocabulary is new.

Cheat-sheet

Control → GitOps evidence (the auditor’s first table):

Policy-as-code quick reference:

Audit-evidence checklist (run before an audit):

Interview and exam questions

Q: Why do auditors consider Git a good audit log, and where is that claim weak? A: Git binds what/when/who/why into an append-only, content-addressed history where each commit hash includes its parent’s, so historical changes can’t be altered invisibly — it is tamper-evident. The weakness is that it is not tamper-proof: someone with force-push on a protected branch could rewrite history. You close that with protected branches, required signatures, no force-push, and an off-cluster (SIEM/mirror) copy so any rewrite is detectable.

Q: Map SOC 2 CC8.1 onto specific GitOps artifacts. A: CC8.1 (change management) requires changes to be authorized, tested, approved, documented, and implemented. The PR is the change (documented in the diff and message), the required review is the authorization/approval, CI checks are the testing, the merge is the record of approval, and the Argo CD sync (in argocd app history) is the implementation record. One artifact chain satisfies the whole control.

Q: In GitOps, what does OutOfSync mean from a compliance standpoint, and how is it a control? A: It means live cluster state diverges from Git. Since humans shouldn’t write to prod directly, an OutOfSync not explained by a pending commit is an out-of-band (unauthorized) change — which is exactly the “detect unauthorized change” control (SOC 2 CC7.2, NIST SI-7). It becomes a real control when you alert on it (notifications → SIEM) and optionally auto-revert with selfHeal. Use argocd app diff to tell a pending-commit OutOfSync from a genuine live edit.

Q: How do you enforce that only signed commits can be deployed, and what does it protect against? A: Set signatureKeys (a list of trusted GPG key IDs) on the AppProject; Argo CD then refuses to sync any revision whose tip commit isn’t signed by a trusted key. Trusted public keys are managed with argocd gpg add. It protects against forged authorship and against an attacker with mere Git write access — deploying now also requires a trusted private signing key.

Q: Kyverno vs OPA Gatekeeper — when would you choose each? A: Kyverno writes policies in Kubernetes-native YAML and can validate, mutate, generate, and verify images, with native PolicyReport CRDs — lower learning curve, and it can generate resources (NetworkPolicies, quotas). Gatekeeper uses Rego via ConstraintTemplate/Constraint — steeper, but a fit if you have Rego expertise or share policy with non-Kubernetes OPA. Choose Kyverno for fast K8s-native policy and generation; Gatekeeper when Rego/OPA is already your standard.

Q: Why run policy at both the PR and admission if admission is un-bypassable? A: Admission is the backstop but gives feedback late — only when a change tries to land, and after review effort is spent. PR-time checks (conftest, kyverno apply, gator test) give the developer feedback in seconds and block the merge, and produce a CI artifact proving policy passed at change time. Admission catches anything that skipped CI (force-merge, direct apply). Together = fast feedback plus an un-bypassable guarantee.

Q: What’s the difference between Kyverno Audit and Enforce, and how do you roll a new policy out safely? A: Audit records violations in PolicyReports but allows the request; Enforce rejects it at admission. Roll out as Audit with background: true first, read the reports to find pre-existing violations, fix them, then flip to Enforce. Going straight to Enforce blocks the next redeploy of every already-violating workload.

Q: How does GitOps enforce separation of duties structurally rather than by policy document? A: The only path to prod is a PR approved by someone other than the author (branch protection), merged into a branch only Argo CD reads, and applied by Argo CD’s service account — which no human logs in as. Author ≠ approver ≠ applier (NIST AC-5). It’s only real if you also remove human prod-write access; otherwise a side-door kubeconfig voids it.

Q: A developer patches a live prod Deployment during an incident and it keeps reverting. What’s happening and what’s the correct process? A: selfHeal: true is reverting the out-of-band change back to Git every reconcile. The correct break-glass is: open an incident ticket, either land the fix as a fast-tracked signed PR (so Git already matches) or disable selfHeal on that one app for the window, apply, backfill to Git, then re-enable selfHeal — keeping the audit trail intact throughout.

Q: Kubernetes Events show who synced, but the auditor wants last quarter’s. What went wrong and how do you fix it? A: Events are garbage-collected after ~1 hour (--event-ttl), so kubectl get events only shows the recent window. The fix is to ship Argo CD events and api-server logs to a SIEM with retention — ideally immutable (WORM) storage (Azure immutable blob, S3 Object Lock, GCS retention lock) — before relying on it as a monitoring control.

Q: What exactly would you hand an auditor to prove change management for one production service? A: git log/git blame on the manifest path (change history + authorship + timestamps), the PR records with approvals (authorization + SoD), argocd app history (what synced when), the SIEM records of who triggered syncs, the CI policy-run logs and PolicyReports (policy compliance), and the drift-alert history (continuous monitoring). The pipeline produced all of it as a by-product.

Q: How do you keep both workloads and audit evidence within a data-residency boundary? A: Pin workloads to region-labeled clusters via AppProject destinations and cluster labels so Argo only deploys them in-region, and keep the evidence in an in-region logging store (in-region Log Analytics/CloudWatch/Cloud Logging plus in-region immutable object storage). For strict sovereignty use the cloud’s boundary — Azure sovereign cloud, AWS GovCloud, Google Assured Workloads.

Key takeaways

• The pipeline is the evidence. In GitOps every production change is a signed, reviewed, timestamped commit reconciled by Argo CD, so change management, attribution, and authorization are emitted as by-products — map them to SOC 2 CC8.1, ISO 27001 A.8.32, PCI 6.5, and NIST CM-3/AC-5 directly.
• “Who deployed what” is a two-source answer: Git (git log/git blame) for authored intent, Argo CD (argocd app history, events) for actual execution and who triggered it. Ship the events to a SIEM because Kubernetes Events expire in ~1 hour.
• Drift is a control, not a nuisance. OutOfSync unexplained by a pending commit is an unauthorized change; alert on it (notifications → SIEM) and optionally enforce with selfHeal. That is continuous unauthorized-change detection for free.
• Sign your provenance. AppProject.signatureKeys makes unsigned commits un-deployable, so Git write access alone can’t ship a change — pair it with cosign image verification for end-to-end provenance.
• Policy-as-code, checked twice. Run the same compliance rules (owner label, no :latest, no privileged, approved registries) at the PR with conftest/kyverno apply/gator for fast feedback and at admission with Kyverno/Gatekeeper as an un-bypassable backstop. Roll out in Audit/dryrun, then Enforce/deny.
• Separation of duties is structural: author ≠ approver ≠ the Argo CD service account that applies — but only if you remove human prod-write access, or a side-door kubeconfig voids the whole control.
• Assemble evidence continuously, not quarterly. PolicyReport/ClusterPolicyReport CRDs, Constraint status.violations, argocd app history, and SIEM dashboards turn compliance into a number you watch trend to zero — and the audit bundle exports in a handful of commands.