Argo CD Notifications: Triggers, Templates, Slack, Teams & Webhooks

In a nutshell

Argo CD watches your clusters and quietly drives them back to match Git. “Quietly” is the whole problem: when a deploy fails or an app turns unhealthy at 2 a.m., the controller notices, records it faithfully, and moves on — but nobody is looking at that screen. Notifications are the part that taps a human on the shoulder. They turn Argo CD’s own state changes into a Slack message, a Microsoft Teams card, or an HTTP call to any system you run.

Think of Argo CD as a diligent night-shift operator who keeps a perfect logbook but never phones anyone. Notifications hand that operator a phone and a contact list: “if a sync fails, message #platform-alerts; if the payments app goes degraded, message the on-call.” You wire it once, and from then on every failure announces itself.

The entire feature is four small pieces you’ll meet in order — a trigger (WHEN to speak), a template (WHAT to say), a service (WHERE to send it: Slack, Teams, webhook), and a subscription (WHO receives it). Get those four question-words straight and everything else is filling in YAML.

Level: Intermediate · Time: ~43 min

Prerequisites: you can apply a ConfigMap and a Secret with kubectl, you know what Synced/Healthy/Degraded mean on an app (Sync Status & Health Assessment), and you’ve seen an Application manifest (Your First Application). After this you can enable the built-in triggers, write a custom one, format a Slack / Teams / webhook message, scope a subscription to one app or a whole project, and test the entire chain with argocd admin notifications — no broken deploy required.

---

A GitOps controller that reconciles silently is a controller nobody trusts. The whole promise of Argo CD is that it drives your clusters to match Git on its own — which is exactly why the dangerous failures are the quiet ones. A sync fails at 02:00 and sits OutOfSync until the morning standup. An app slides to Degraded after a bad image tag and stays there because no human happened to be looking at the right project in the UI. The reconcile loop did its job and reported the result faithfully — to a dashboard nobody was watching.

Notifications close that gap. They turn Argo CD’s internal state transitions — sync succeeded, sync failed, health degraded, deploy finished — into messages that arrive where your team already lives: a Slack channel, a Microsoft Teams channel, PagerDuty, an internal change-management system. This lesson teaches the notification engine from first principles: the four building blocks (triggers, templates, services, subscriptions), the exact ConfigMap and Secret schema, and three delivery paths — Slack, Teams, and a generic webhook you can point at anything. Then you’ll build the one alert every platform needs first — on-sync-failed to Slack — end to end, and test it with argocd admin notifications without touching a live cluster.

Everything here targets Argo CD 2.13+ and 3.x, where notifications ship inside Argo CD. If you followed an old blog that told you to kubectl apply a separate argocd-notifications project, that era is over — more on why in a moment.

---

Why this matters

There are three questions a delivery platform must answer without a human staring at a screen, and notifications answer all three:

• Did the thing I just shipped actually land? A merge to main should end in a message that says “checkout v1.8.2 is Synced and Healthy on prod-eu,” not in you refreshing the UI for five minutes.
• Did something break that I didn’t touch? Self-heal reverting drift, a node going away, an image that fails its readiness probe — these degrade an app with no commit from you. You want to hear about it in seconds, not discover it in a customer ticket. And when you run progressive delivery, the automation may already be reacting: a failed canary analysis can trigger an automatic rollback (see Rollouts: Analysis, Metrics & Auto-Rollback) — a notification is how the humans learn what the automation already did.
• Did an automated action fail? Automated sync with prune and selfHeal is powerful precisely because it acts without you. When it can’t act — a SyncFailed, a hook that errored — that silence is the most important signal in the system.

The naive alternative is to point a generic Prometheus alert at Argo CD’s metrics and call it done. Metrics are essential for continuous signals — sync latency creeping up, reconcile queue depth, controller saturation — and the companion lesson Observability: Metrics with Prometheus & Grafana covers that surface in depth. But metrics are the wrong tool for discrete events. “Application payments transitioned to Degraded at 02:14 on cluster prod-eu, here is the deep link” is an event with rich context attached to one specific object. Notifications are event-shaped; metrics are rate-shaped. A mature platform runs both.

The mental model to hold for the rest of the lesson is a four-part pipeline, and it maps to four English question words:

Get those four words straight and every manifest in this lesson slots into place. When a notification misbehaves, you debug it by asking which of the four is wrong: is the trigger not matching, the template rendering blank, the service unauthenticated, or the subscription pointed at the wrong object?

---

The notifications controller: built in, not a separate install

Argo CD Notifications began life as argocd-notifications, a standalone project under argoproj-labs. It was folded into Argo CD proper in the 2.3 release, and on any 2.13+/3.x install it is simply there — a Deployment named argocd-notifications-controller running in the argocd namespace alongside the application-controller, repo-server, and API server. You do not install it separately. You do not manage a second Helm release. You configure it by editing two objects that already exist (empty) in a standard install.

The controller is a single-purpose reconciler. It runs an informer on Application resources in the namespaces Argo CD manages; when an app’s status changes, it evaluates every configured trigger against that app, and for each trigger that fires and has a matching subscription, it renders the trigger’s template and hands the result to the service. It writes a small bookkeeping annotation back onto the app (notified.notifications.argoproj.io) so it knows what it has already sent — that annotation is the machinery behind oncePer, which we’ll get to.

Two things follow from “it watches Application status.” First, notifications are event-driven, not scheduled — nothing fires on a timer; something fires when the observed state changes. Second, the quality of your notifications is bounded by the quality of Argo CD’s state assessment underneath them. A trigger that keys off Degraded is only as good as the health check that produces Degraded in the first place, which is the subject of Sync Status & Health Assessment. Notifications don’t compute health; they react to it.

Everything the controller reads out of argocd-notifications-cm is one of a small set of key prefixes. Learn these six and you can read any team’s config at a glance:

---

The four building blocks in one pipeline

Before the details, see the whole thing as one left-to-right flow. An Application changes state; a trigger’s when decides whether that change is worth a notification; oncePer stops it re-firing on every reconcile; the matched template renders a message from the app’s data; a service delivers it using a secret; and a subscription decides which channel or person actually receives it.

The badges mark the six places this pipeline goes wrong in practice: the event has to be a real transition (1); the when expression has to actually match, nil-safe (2); oncePer has to be present or you spam yourself (3); template paths have to be exact or fields render blank (4); the service has to be authenticated from the secret (5); and the subscription has to be on the right object (6). The rest of this lesson walks the pipeline one block at a time, then assembles the canonical on-sync-failed → Slack alert that runs the length of it.

---

Triggers — deciding WHEN

A trigger is a named list of conditions. Each condition has a when (a boolean expression), a send (the templates to render), and optionally a oncePer (a de-duplication key). Argo CD ships a catalog of triggers you can enable by name; here are the ones that matter, reproduced with their exact built-in definitions so you can see the real expressions:

Read three things out of that table. First, several triggers guard with app.status.operationState != nil before touching operationState.phase — because a brand-new app has no operation yet, and dereferencing a nil field would error. Second, the ?. in app.status.operationState?.syncResult?.revision is the optional-chaining operator of the expression language: if any link is nil, the whole thing evaluates to nil instead of crashing. Third, note the semantic difference between on-sync-succeeded (fires when the sync operation succeeded) and on-deployed (fires when the sync succeeded and the app is Healthy, keyed oncePer the sync revision). on-deployed is the “it’s actually live and working” signal; on-sync-succeeded is the “the apply finished” signal. Alert on the wrong one and you’ll page people before the pods are ready.

The when expression is written in expr, a small, safe expression language. You’ll only ever need a handful of its features:

oncePer — the anti-spam mechanism

Here is the single most important operational fact about triggers: a when condition stays true for as long as the state is true. An app that is Degraded is Degraded on this reconcile, the next reconcile ~180 seconds later, and every reconcile after that until someone fixes it. Without protection, on-health-degraded would deliver the same alert every three minutes, all night. oncePer is the fix. It computes a value per notification and tells the controller “only send again when this value changes”:

# A custom "degraded" trigger that alerts once per bad revision, not once per reconcile
trigger.on-health-degraded: |
  - when: app.status.health.status == 'Degraded'
    oncePer: app.status.sync.revision
    send: [app-health-degraded]

With oncePer: app.status.sync.revision, the app can sit Degraded for an hour and you get one message. Push a new commit (a fix, or another bad one), the revision changes, and you’re eligible for exactly one more. The controller records the last-sent value in the app’s notified.notifications.argoproj.io annotation; the de-dup is stateful and survives controller restarts because the state lives on the app object, not in the controller’s memory.

Omitting oncePer is occasionally correct — for a truly edge-triggered condition that can’t stay true — but treat a missing oncePer on a state trigger (Degraded, OutOfSync) as a bug. It is the number-one cause of “Argo CD spammed our channel 400 times.”

Custom triggers and defaultTriggers

You are not limited to the catalog. A custom trigger is just another trigger.<name> key. This one alerts only for production apps that have drifted out of sync:

trigger.on-prod-outofsync: |
  - when: app.status.sync.status == 'OutOfSync' and app.metadata.labels.env == 'prod'
    oncePer: app.status.sync.revision
    send: [app-sync-status-unknown]

defaultTriggers lets a subscription omit the trigger name and inherit a standard set — handy when you want every subscribed app to get the same baseline without repeating yourself:

defaultTriggers: |
  - on-sync-failed
  - on-health-degraded

---

Templates — deciding WHAT

A template is the message. It is a Go text/template rendered against a data model — the Application object plus some context. The plain message field is the fallback text every service can use; per-service blocks (slack, teams, email, webhook) override it with channel-specific formatting.

Start with the data model, because every template variable comes from here:

The .app object is the workhorse, and a few paths do most of the work. Get these exactly right — a typo renders an empty string, never an error:

Multi-source gotcha: apps that use spec.sources (a list) instead of spec.source (a single object) will render .app.spec.source.repoURL as blank. For multi-source apps, index the list: {{ (index .app.spec.sources 0).repoURL }}.

The template body can call functions — the engine registers the sprig library plus a few notification-specific helpers. The most useful ones:

That last one is the pattern the built-in templates use to render one message differently per channel. Here is the shape of the built-in app-sync-failed message — note how it emits a Slack :exclamation: emoji only when Slack is the service, and builds a deep link from .context.argocdUrl:

template.app-sync-failed: |
  message: |
    {{if eq .serviceType "slack"}}:exclamation:{{end}} The sync operation of application {{.app.metadata.name}}
    has failed at {{.app.status.operationState.finishedAt}} with the following error:
    {{.app.status.operationState.message}}
    Sync operation details: {{.context.argocdUrl}}/applications/{{.app.metadata.name}}?operation=true

Here are the per-service blocks a template can carry. You pick the ones for the channels you actually use; a single template can hold all of them at once:

---

Services and subscriptions — WHERE and WHO

A service (Argo CD calls them notifiers) is a delivery channel. It is defined once in the cm and reads its credentials from the secret. Argo CD ships a broad catalog; you’ll meet most of them here:

Every service follows the same rule: the secret value is never inlined. You write $slack-token in the cm and Argo CD looks up the key slack-token in argocd-notifications-secret. This is what keeps a plaintext bot token out of your GitOps repo — the cm is safe to commit; the secret is not.

A subscription is the wiring that says “send this trigger, delivered by this service, to this recipient.” There are three ways to declare one, and they compose:

The annotation form is the one you’ll use most. On an Application:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: checkout
  namespace: argocd
  annotations:
    # subscribe.<trigger>.<service>: <recipient>
    notifications.argoproj.io/subscribe.on-sync-failed.slack: platform-alerts
    notifications.argoproj.io/subscribe.on-health-degraded.slack: platform-alerts
spec:
  # ...

Two recipients for the same trigger+service? Separate them with a semicolon: platform-alerts;payments-oncall. Want every app in a project to alert without annotating each one? Put the same annotation on the AppProject. Want a fleet-wide default with a label filter? Use the cm’s subscriptions key with a selector:

subscriptions: |
  - recipients:
      - slack:platform-alerts
    triggers:
      - on-sync-failed
      - on-health-degraded
  - recipients:
      - slack:payments-oncall
    triggers:
      - on-health-degraded
    selector: team=payments        # only apps labelled team=payments

The subscription’s scope is where “the wrong app got paged” bugs live: an annotation on the AppProject fans out to every app in it, and a cm subscriptions entry without a selector fans out to the whole cluster. When in doubt, subscribe on the individual Application and widen deliberately.

---

The on-sync-failed alert, end to end (Slack)

This is the single most valuable notification a platform can have, so we’ll build it completely: secret, service, template, trigger, subscription — the full length of the pipeline. Slack first because it’s the most common; Teams and webhook variants follow in the next section.

Step 1 — Create a Slack bot and store its token

In Slack, create an app, add the bot OAuth scopes, install it to the workspace, copy the bot token (xoxb-…), and invite the bot into every channel you’ll notify. That last step is the one everyone forgets, and it produces channel_not_found even when the token is perfect.

The token goes into the secret — never the cm, never Git in plaintext:

apiVersion: v1
kind: Secret
metadata:
  name: argocd-notifications-secret
  namespace: argocd
  labels:
    app.kubernetes.io/part-of: argocd
type: Opaque
stringData:
  slack-token: xoxb-REPLACE-WITH-YOUR-BOT-TOKEN   # ⚠️ placeholder — never commit a real token

⚠️ In a real GitOps repo you would not commit this file at all. Source the token from your cloud secret store via External Secrets Operator, or seal it — see Managing Secrets: Sealed Secrets, ESO, SOPS & Vault and the per-cloud table later in this lesson.

Step 2 — Define the Slack service and shared context

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-notifications-cm
  namespace: argocd
  labels:
    app.kubernetes.io/part-of: argocd
data:
  # WHERE: the Slack notifier reads its token from the secret
  service.slack: |
    token: $slack-token
    username: argocd-bot          # optional; needs chat:write.customize
    icon: ":argo:"                # optional

  # Shared values every template can read — the UI base URL for deep links
  context: |
    argocdUrl: https://argocd.example.com

Step 3 — Write the template (WHAT)

A good failure message answers “which app, which cluster, what state, and take me there.” The plain message covers non-Slack services; the slack.attachments block gives Slack a coloured, structured card:

  template.app-sync-failed: |
    message: |
      {{if eq .serviceType "slack"}}:exclamation:{{end}} Sync FAILED — {{.app.metadata.name}}
      Cluster: {{.app.spec.destination.server}}
      Phase: {{.app.status.operationState.phase}} — {{.app.status.operationState.message}}
      Open: {{.context.argocdUrl}}/applications/{{.app.metadata.name}}?operation=true
    slack:
      attachments: |
        [{
          "title": "{{.app.metadata.name}} — sync failed",
          "title_link": "{{.context.argocdUrl}}/applications/{{.app.metadata.name}}?operation=true",
          "color": "#E96D76",
          "fields": [
            { "title": "Sync Status",  "value": "{{.app.status.sync.status}}",              "short": true },
            { "title": "Phase",        "value": "{{.app.status.operationState.phase}}",      "short": true },
            { "title": "Repository",   "value": "{{.app.spec.source.repoURL}}",              "short": false },
            { "title": "Revision",     "value": "{{.app.status.sync.revision}}",             "short": true }
          ]
        }]

Step 4 — Confirm the trigger (WHEN)

The catalog on-sync-failed already does what we want, but declaring it explicitly documents intent and lets you tune oncePer:

  trigger.on-sync-failed: |
    - when: app.status.operationState != nil and app.status.operationState.phase in ['Error', 'Failed']
      oncePer: app.status.operationState.syncResult.revision
      send: [app-sync-failed]

Step 5 — Subscribe an Application (WHO)

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: checkout
  namespace: argocd
  annotations:
    notifications.argoproj.io/subscribe.on-sync-failed.slack: platform-alerts
spec:
  project: default
  source:
    repoURL: https://github.com/acme/app-config.git
    targetRevision: main
    path: apps/checkout
  destination:
    server: https://kubernetes.default.svc
    namespace: checkout
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

That’s the whole pipeline. Apply the cm, secret, and app, and the moment checkout’s sync operation enters Failed, the controller renders app-sync-failed and posts it to #platform-alerts — once per failed revision. Because the notification tells you the automated sync failed, it pairs directly with Sync Policies: Automated, Self-Heal & Prune: auto-sync is safe to leave on precisely because its failures are no longer silent.

Step 6 — Test it without waiting for a real failure

You don’t have to break an app to prove the wiring. The argocd admin notifications CLI renders and delivers a template against a real app’s current state:

# Render app-sync-failed for the 'checkout' app and deliver to Slack channel platform-alerts.
# This uses the live cm + secret in the argocd namespace.
argocd admin notifications template notify app-sync-failed checkout \
  --recipient slack:platform-alerts

Representative result — the command exits 0 and a card lands in Slack. A rendered payload looks like this (labelled representative; formatting is what Slack shows):

[ representative Slack message ]
❗ checkout — sync failed
Sync Status: OutOfSync   Phase: Failed
Repository: https://github.com/acme/app-config.git
Revision: 4f2a9c1
→ title links to https://argocd.example.com/applications/checkout?operation=true

You can also ask whether a trigger would fire against an app right now:

# Evaluate the on-sync-failed trigger against checkout's current state.
argocd admin notifications trigger run on-sync-failed checkout

[ representative output ]
NAME             CONDITION                                                              TRIGGERED
on-sync-failed   app.status.operationState != nil and ...phase in ['Error','Failed']   false

false here is correct — a Synced/Healthy app should not fire the failure trigger. Break something (a bad image tag) and the same command returns true. This is how you develop triggers and templates without a cluster full of deliberately broken apps.

---

Microsoft Teams and generic webhooks

Slack is one delivery path. The same trigger and template feed Teams and any HTTP endpoint just by adding a service and a per-service template block.

Microsoft Teams

Teams historically receives Argo CD notifications through an Incoming Webhook connector — a per-channel URL that accepts an Office 365 MessageCard. You store the URL in the secret and map it to a name in service.teams:

  service.teams: |
    recipientUrls:
      payments-channel: $teams-payments-url

# in argocd-notifications-secret
stringData:
  teams-payments-url: https://REPLACE.webhook.office.com/webhookb2/xxxx-xxxx  # ⚠️ placeholder

Add a teams block to the template. Teams cards are built from facts (labelled key/values), a themeColor, and potentialAction buttons:

  template.app-sync-failed: |
    teams:
      themeColor: "#E96D76"
      title: "Sync FAILED — {{.app.metadata.name}}"
      text: "The sync of {{.app.metadata.name}} failed at {{.app.status.operationState.finishedAt}}."
      summary: "Argo CD sync failed for {{.app.metadata.name}}"
      facts: |
        [
          { "name": "Sync Status", "value": "{{.app.status.sync.status}}" },
          { "name": "Phase",       "value": "{{.app.status.operationState.phase}}" },
          { "name": "Repository",  "value": "{{.app.spec.source.repoURL}}" }
        ]
      potentialAction: |
        [{
          "@type": "OpenUri",
          "name": "Open in Argo CD",
          "targets": [
            { "os": "default", "uri": "{{.context.argocdUrl}}/applications/{{.app.metadata.name}}" }
          ]
        }]

Subscribe to the named connector, not to a raw URL:

notifications.argoproj.io/subscribe.on-sync-failed.teams: payments-channel

⚠️ Currency — the connector is being retired. Microsoft is deprecating Office 365 Incoming Webhook connectors; they stop working on 31 March 2026. The forward path is Teams Workflows (Power Automate), which issues a new webhook URL that accepts an Adaptive Card. Operationally, most teams keep using the service.teams connector config with a Workflows-issued URL, or switch to the generic webhook service (below) posting an Adaptive Card payload. Check Microsoft’s current guidance before standing up a new Teams integration, and don’t build anything new on a bare Office 365 connector.

Generic webhook — wire into anything

The generic webhook is the escape hatch: it POSTs (or any method) to a URL you control, with headers and a body you define. Use it for an internal change-management system, a CMDB, a Lambda/Cloud Function, an incident tool without a first-class notifier — anything that speaks HTTP.

Define the service once. Auth travels in a header sourced from the secret; the retry knobs bound how hard it tries:

  service.webhook.internal-cmdb: |
    url: https://cmdb.internal.example.com/api/deployments
    headers:
      - name: Content-Type
        value: application/json
      - name: Authorization
        value: Bearer $cmdb-webhook-token     # resolved from the secret
    insecureSkipVerify: false                  # true only for internal self-signed TLS
    retryWaitMin: 1s
    retryWaitMax: 5s
    retryMax: 3

The template supplies the request shape under a webhook.<name> block — method, an optional path appended to the service url, and a body:

  template.app-sync-failed: |
    webhook:
      internal-cmdb:
        method: POST
        body: |
          {
            "event": "argocd.sync.failed",
            "app": "{{.app.metadata.name}}",
            "cluster": "{{.app.spec.destination.server}}",
            "namespace": "{{.app.spec.destination.namespace}}",
            "syncStatus": "{{.app.status.sync.status}}",
            "phase": "{{.app.status.operationState.phase}}",
            "revision": "{{.app.status.sync.revision}}",
            "message": {{ .app.status.operationState.message | toRawJson }}
          }

Note {{ .app.status.operationState.message | toRawJson }} — piping a free-text field through toRawJson (a sprig function) safely quotes and escapes it so a stray " in the error message can’t produce invalid JSON. That one habit prevents a whole class of “the webhook returned 400” failures.

The webhook subscription names the service; the recipient is usually empty because the URL is fixed in the service config:

notifications.argoproj.io/subscribe.on-sync-failed.internal-cmdb: ""

---

Where the token lives: the notifications secret across AKS, EKS, GKE

Notifications themselves are cloud-neutral — the controller, triggers, templates, and subscriptions are identical whether you run on AKS, EKS, or GKE. The one genuine cloud edge is where the Slack token, Teams URL, and webhook credentials come from. Committing argocd-notifications-secret to Git in plaintext is a rotation and leak liability; the right pattern is to keep the secret’s values in your cloud secret store and project them into the Kubernetes secret with External Secrets Operator (ESO) (or seal them). The mechanics differ per cloud:

The ESO object is the same shape everywhere; only the secretStoreRef and the remote keys change. This ExternalSecret assembles the Argo CD notifications secret from three entries in your cloud store:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: argocd-notifications-secret
  namespace: argocd
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: cloud-secret-store        # points at azurekv / aws / gcpsm
    kind: SecretStore
  target:
    name: argocd-notifications-secret   # the exact name the controller reads
    creationPolicy: Owner
    template:
      metadata:
        labels:
          app.kubernetes.io/part-of: argocd
  data:
    - secretKey: slack-token
      remoteRef:
        key: argocd/slack-bot-token
    - secretKey: teams-payments-url
      remoteRef:
        key: argocd/teams-payments-url
    - secretKey: cmdb-webhook-token
      remoteRef:
        key: argocd/cmdb-webhook-token

The payoff: rotating a leaked Slack token is a change in Key Vault / Secrets Manager / Secret Manager, and ESO re-projects it within refreshInterval — no Git commit, no plaintext token anywhere in your repo. The deeper treatment of all four secret strategies (ESO, Sealed Secrets, SOPS, Vault) lives in Managing Secrets: Sealed Secrets, ESO, SOPS & Vault.

---

Hands-on lab

This lab wires on-sync-failed → Slack end to end at the config level and tests it with argocd admin notifications — no live failure required. You need kubectl context on a cluster running Argo CD 2.13+/3.x (a free local kind/minikube install is fine) and a Slack bot token. Every credential below is a placeholder.

⚠️ Never commit a real token. The secret in Step 2 is applied directly with kubectl, not stored in Git. In production, use the ESO pattern above.

Step 1 — Confirm the controller is running.

kubectl -n argocd get deploy argocd-notifications-controller

[ representative output ]
NAME                             READY   UP-TO-DATE   AVAILABLE   AGE
argocd-notifications-controller  1/1     1            1           42d

What just happened: the notifications controller ships with Argo CD; if it’s 1/1 you have nothing to install.

Step 2 — Load the Slack token into the secret.

kubectl -n argocd patch secret argocd-notifications-secret \
  --type merge \
  -p '{"stringData":{"slack-token":"xoxb-REPLACE-WITH-YOUR-BOT-TOKEN"}}'

What just happened: you added the slack-token key the cm will reference as $slack-token. patch --type merge leaves any existing keys intact.

Step 3 — Configure the service, context, template, and trigger. Save as notifications-cm.yaml and apply:

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-notifications-cm
  namespace: argocd
  labels:
    app.kubernetes.io/part-of: argocd
data:
  service.slack: |
    token: $slack-token
  context: |
    argocdUrl: https://argocd.example.com
  template.app-sync-failed: |
    message: |
      {{if eq .serviceType "slack"}}:exclamation:{{end}} Sync FAILED — {{.app.metadata.name}} ({{.app.status.operationState.phase}})
      {{.context.argocdUrl}}/applications/{{.app.metadata.name}}?operation=true
    slack:
      attachments: |
        [{
          "title": "{{.app.metadata.name}} — sync failed",
          "title_link": "{{.context.argocdUrl}}/applications/{{.app.metadata.name}}?operation=true",
          "color": "#E96D76",
          "fields": [
            { "title": "Sync Status", "value": "{{.app.status.sync.status}}", "short": true },
            { "title": "Revision",    "value": "{{.app.status.sync.revision}}", "short": true }
          ]
        }]
  trigger.on-sync-failed: |
    - when: app.status.operationState != nil and app.status.operationState.phase in ['Error', 'Failed']
      oncePer: app.status.operationState.syncResult.revision
      send: [app-sync-failed]

kubectl apply -f notifications-cm.yaml

What just happened: one apply installed WHERE (service.slack), WHAT (template.app-sync-failed), and WHEN (trigger.on-sync-failed), plus the shared argocdUrl for deep links.

Step 4 — Subscribe an app (WHO).

kubectl -n argocd annotate application checkout \
  notifications.argoproj.io/subscribe.on-sync-failed.slack=platform-alerts --overwrite

What just happened: you bound the on-sync-failed trigger, delivered by Slack, to #platform-alerts for the checkout app only.

Step 5 — Validate the config parses, from files, with no cluster calls.

kubectl -n argocd get cm argocd-notifications-cm -o yaml > cm.yaml
argocd admin notifications trigger get \
  --config-map ./cm.yaml --secret :empty

[ representative output ]
NAME             ENABLED
on-sync-failed   true

What just happened: --secret :empty tells the CLI to parse without needing real credentials, so you can lint triggers/templates in CI before they ever hit a cluster.

Step 6 — Test-fire the notification against the live app.

argocd admin notifications template notify app-sync-failed checkout \
  --recipient slack:platform-alerts

What just happened: the CLI rendered app-sync-failed from checkout’s current state and delivered it to Slack. A card in #platform-alerts proves all four blocks are wired. If it errors, jump to troubleshooting — the error names which block failed (not_authed = service/secret; empty fields = template path; channel_not_found = bot not in channel).

Step 7 — Watch the controller do it for real (optional). Change checkout to an image tag that will never pull, commit, let it sync and fail, and tail the controller:

kubectl -n argocd logs deploy/argocd-notifications-controller --tail=20

[ representative output ]
level=info msg="Trigger on-sync-failed result: [{[app-sync-failed] true}]" app=argocd/checkout
level=info msg="Notification about condition 'on-sync-failed.[0]' was sent" app=argocd/checkout recipient=slack:platform-alerts

What just happened: the controller observed the Failed transition, evaluated the trigger to true, and sent exactly one message. Fix the tag and it will not re-notify for that same failed revision — oncePer at work.

Teardown.

# Remove the subscription
kubectl -n argocd annotate application checkout \
  notifications.argoproj.io/subscribe.on-sync-failed.slack- 

# Blank the token (or delete/rotate it in Slack)
kubectl -n argocd patch secret argocd-notifications-secret \
  --type merge -p '{"stringData":{"slack-token":""}}'

# Optionally reset the cm to its shipped (empty) state
kubectl -n argocd get cm argocd-notifications-cm -o yaml   # inspect before editing

Leaving the cm in place is harmless (no subscriptions means no traffic), but always revoke or rotate the Slack token if it was a real one.

---

Common mistakes and troubleshooting

Every notification bug is a break in one of the four blocks. This table is organised by symptom; the “block” column tells you where to look first.

Three of these deserve extra words, because they burn the most hours:

1. “It never fires” is almost always the trigger or the subscription, never the template. People rewrite the template message when the real problem is that when evaluated false (the app was Progressing, not Failed, when they looked) or the subscription annotation has a typo (subscribe.on-sync-fail.slack instead of on-sync-failed). Debug in order: run argocd admin notifications trigger run <trigger> <app> to prove the condition; then kubectl get app <app> -o yaml | grep subscribe to prove the subscription. Only touch the template once you’ve seen the trigger return true and the subscription is correct.

2. Spam is a oncePer bug, and it’s a state-vs-event confusion. A trigger’s when describes a state (Degraded), and states persist. The controller re-evaluates every reconcile (~180s by default), so a persistent bad state without oncePer re-sends forever. The mental correction: you don’t want a message while the app is degraded; you want a message when it becomes degraded. oncePer: app.status.sync.revision encodes exactly that — one alert per bad revision. If you genuinely want a reminder every N minutes, that’s a Prometheus for: alert on the metric, not a notification.

3. Secret references are string-exact and silent on failure. $slack-token in the cm resolves to the key slack-token in argocd-notifications-secret. If the secret key is slackToken, slack_token, or lives in the wrong secret, the reference resolves to empty and the service fails to authenticate with a generic error — the controller does not tell you “key not found.” When any service returns an auth error, verify the exact key name in the secret first: kubectl -n argocd get secret argocd-notifications-secret -o jsonpath='{.data}' | tr ',' '\n'.

---

Going deeper

The pipeline above is the 90%. The remaining 10% is where production platforms live: how the controller de-dups statefully, how it updates a message in place, what it does not guarantee, and how it behaves at fleet scale. None of this changes the four-block model — it sharpens it.

A trigger is a list of conditions, not a single rule

The value under trigger.<name> is a YAML list. Each list item is an independent when / oncePer / send unit, evaluated in order, and any number of them may fire on the same reconcile. That lets one trigger name cover a whole lifecycle:

trigger.on-sync-lifecycle: |
  - when: app.status.operationState != nil and app.status.operationState.phase in ['Error','Failed']
    oncePer: app.status.operationState.syncResult.revision
    send: [app-sync-failed]
  - when: app.status.operationState != nil and app.status.operationState.phase in ['Succeeded']
    oncePer: app.status.operationState.syncResult.revision
    send: [app-sync-succeeded]

A single condition can also send several templates (send: [app-sync-failed, page-oncall]) — one event, two differently-formatted deliveries. Keep each condition’s when cheap; see scale below.

oncePer is stored on the app, in the notified annotation

The de-dup state that makes oncePer work is not held in controller memory — it is written back onto the Application as notified.notifications.argoproj.io, a JSON map keyed by trigger + condition index + template + service + the hashed oncePer value, with the send timestamp as the value:

# representative — Argo CD's own dedup bookkeeping, written onto the Application
metadata:
  annotations:
    notified.notifications.argoproj.io: >-
      {"on-sync-failed:[0]:app-sync-failed:slack:4f2a9c1":1719830400}

Two operational consequences. First, dedup survives controller restarts and re-elections because the memory is on the object, not the pod. Second, if you ever want to force a re-notify (say, to re-test after fixing a template), delete that annotation — kubectl -n argocd annotate app <name> notified.notifications.argoproj.io- — and the next matching reconcile is eligible to send again.

Updating a message in place: deliveryPolicy, groupingKey, notifyBroadcast

By default every send POSTs a new Slack message. For a status that evolves — Progressing → Degraded → Healthy — you often want one message that edits itself rather than three stacked cards. The Slack template block controls this:

template.app-health: |
  message: "{{.app.metadata.name}} is {{.app.status.health.status}}"
  slack:
    groupingKey: "{{.app.metadata.name}}-health"   # which messages count as "the same"
    deliveryPolicy: PostAndUpdate                   # post once, then edit that message
    notifyBroadcast: false                          # true = also push a threaded reply to the channel

deliveryPolicy takes Post (default — always a new message), PostAndUpdate (post the first, then update the message that shares this groupingKey), or Update (only update, never post a new one). This is the difference between a channel that reads like a live status board and one that reads like a spam folder.

Reliability: notifications are best-effort awareness, not an audit log

The controller reacts to observed Application status. It is at-most-once and has no durable event queue: if the controller is down during a brief transition that then reverts, that edge can be missed and is not replayed on restart — on restart it evaluates current state, and oncePer suppresses anything already marked in notified. The design implication is important: use notifications for human awareness (“someone should look”), and keep anything that must never be lost — compliance events, hard SLO breaches, real paging — on a durable path (PagerDuty/Opsgenie with its own retries, or a Prometheus for: alert on the metric, covered in Observability: Metrics with Prometheus & Grafana). A dropped Slack card should never be the only record that production broke.

Scale and cost: one controller, every trigger × every app

Unlike the application-controller — which shards across replicas — the notifications controller is a single, unsharded Deployment. On every reconcile of every app it evaluates every enabled trigger’s when against that app. That is cheap per-evaluation but multiplies: thousands of apps × dozens of conditions is real CPU. Practical guidance: keep when expressions simple (field comparisons and in, not heavy matches regex on every app), don’t enable catalog triggers nobody subscribes to, and if evaluation latency shows up, treat it as part of the same capacity conversation as repo-server and controller sizing in Scaling & Sharding: Repo-Server, Monorepo & Performance.

Observability of the controller itself

When a notification doesn’t arrive, your first question is “did the controller even try?” Two instruments answer it. The controller exposes its own Prometheus metrics (default :9001/metrics) — most usefully a delivery counter and a trigger-evaluation counter:

kubectl -n argocd port-forward deploy/argocd-notifications-controller 9001:9001
# argocd_notifications_deliveries_total{service,trigger,succeeded="true|false"}
# argocd_notifications_trigger_eval_total{name,triggered="true|false"}

A deliveries_total{succeeded="false"} climbing tells you the service is failing (auth, channel); an eval_total{triggered="false"} with no successes tells you the trigger never matched. For a line-by-line trace, raise verbosity with --loglevel debug on the controller and read the per-app evaluation log — it prints every trigger result, which is the fastest way to see a when returning false you expected to be true.

Security: least privilege for the secret and the controller

The controller reads argocd-notifications-secret, so that secret should hold only notification credentials — a leaked Slack token is bad, but a shared “everything” secret turns one leak into many. The controller’s ServiceAccount needs get/list/watch on Application objects and read on its own cm and secret, and nothing more; don’t grant it broad cluster read. Treat the notification bot token, Teams URL, and webhook credentials as rotatable secrets sourced from your cloud store (the ESO pattern earlier), and fold them into the same least-privilege posture as the rest of your install in Security Hardening: Least Privilege & Network Policy.

---

Common beginner mistakes

The troubleshooting table above is symptom → fix. These are the mental-model errors that produce those symptoms — the wrong idea first, the right one beside it.

---

Practice challenges

Work these in order — beginner to advanced. Each reveals a worked solution and a one-line reason it’s the right move. All credentials are placeholders; nothing here needs a broken app.

1. (Beginner) Prove the engine is already installed and name its two config objects.

<details> <summary>Solution</summary>

kubectl -n argocd get deploy argocd-notifications-controller
kubectl -n argocd get cm argocd-notifications-cm
kubectl -n argocd get secret argocd-notifications-secret

The controller ships with Argo CD 2.13+/3.x; the cm holds triggers/templates/services, the secret holds credentials. </details>

Why: you never install notifications separately — you configure objects that already exist.

2. (Beginner) Subscribe the checkout app to on-health-degraded, delivered to Slack #platform-alerts, using only an annotation.

<details> <summary>Solution</summary>

kubectl -n argocd annotate application checkout \
  notifications.argoproj.io/subscribe.on-health-degraded.slack=platform-alerts --overwrite

</details>

Why: the subscribe.<trigger>.<service> annotation is the WHO — no cm edit needed to wire one app.

3. (Intermediate) Your channel got 200 identical “degraded” messages overnight. Write the trigger that stops it.

<details> <summary>Solution</summary>

trigger.on-health-degraded: |
  - when: app.status.health.status == 'Degraded'
    oncePer: app.status.sync.revision
    send: [app-health-degraded]

</details>

Why: Degraded is a persistent state; oncePer collapses “every reconcile” into “once per bad revision.”

4. (Intermediate) Alert only when a production app drifts OutOfSync, ignoring dev and staging. Write the custom trigger.

<details> <summary>Solution</summary>

trigger.on-prod-drift: |
  - when: app.status.sync.status == 'OutOfSync' and app.metadata.labels.env == 'prod'
    oncePer: app.status.sync.revision
    send: [app-sync-status-unknown]

Then subscribe prod apps with notifications.argoproj.io/subscribe.on-prod-drift.slack: platform-alerts. </details>

Why: scoping inside the when keeps the condition self-documenting; a subscriptions selector is the alternative if you’d rather keep triggers generic.

5. (Advanced) Make repeated health alerts for one app update a single Slack message instead of stacking new cards.

<details> <summary>Solution</summary>

template.app-health-degraded: |
  message: "{{.app.metadata.name}} is {{.app.status.health.status}}"
  slack:
    groupingKey: "{{.app.metadata.name}}-health"
    deliveryPolicy: PostAndUpdate

</details>

Why: groupingKey marks which messages are “the same” and PostAndUpdate edits that one in place — a live status line per app instead of a wall of duplicates.

6. (Advanced) Wire on-sync-failed to an internal HTTP endpoint that needs a bearer token, with a body that survives quotes in the error text — then dry-run whether it would fire.

<details> <summary>Solution</summary>

# service (in argocd-notifications-cm) — auth token pulled from the secret
service.webhook.cmdb: |
  url: https://cmdb.internal.example.com/api/deployments
  headers:
    - name: Authorization
      value: Bearer $cmdb-webhook-token
# per-service template block (in argocd-notifications-cm)
template.app-sync-failed: |
  webhook:
    cmdb:
      method: POST
      body: |
        {"app":"{{.app.metadata.name}}","message":{{ .app.status.operationState.message | toRawJson }}}

# does the trigger match checkout right now? (no delivery, no broken app needed)
argocd admin notifications trigger run on-sync-failed checkout

Subscribe with notifications.argoproj.io/subscribe.on-sync-failed.cmdb: "" — the recipient is empty because the URL is fixed in the service. </details>

Why: | toRawJson quotes and escapes free text so a stray " can’t produce an invalid body, and trigger run proves the condition without touching a live deploy.

---

Cheat-sheet

Everything you reach for, in one place.

The four blocks → the ConfigMap/annotation keys

Built-in triggers

Most-used template variables

Service configs (in the cm; secrets via $key)

CLI (argocd admin notifications)

Subscription forms

---

Interview and exam questions

Q: What are the four building blocks of an Argo CD notification, and what does each decide? A: Triggers (WHEN — a when condition on the app’s state), templates (WHAT — the message body and per-service formatting), services/notifiers (WHERE — Slack, Teams, webhook, email…), and subscriptions (WHO — which recipient gets which trigger on which app, via a subscribe.<trigger>.<service> annotation or a cm subscriptions entry). Debugging a broken notification means identifying which of the four is wrong.

Q: Do you install argocd-notifications separately in current Argo CD? A: No. It was a standalone argoproj-labs project but was merged into Argo CD in 2.3. On 2.13+/3.x it’s the built-in argocd-notifications-controller Deployment; you configure it through the existing argocd-notifications-cm ConfigMap and argocd-notifications-secret.

Q: Your Slack channel got 400 identical “degraded” messages overnight. What happened and how do you fix it? A: The on-health-degraded trigger has no oncePer, so it re-sent on every ~180-second reconcile while the app stayed Degraded. States persist; the controller re-evaluates each cycle. Add oncePer: app.status.sync.revision so it fires once per revision. The conceptual fix is realising you want an edge (became degraded), not a level (is degraded) — levels belong to Prometheus for: alerts, not notifications.

Q: What’s the difference between on-sync-succeeded and on-deployed? A: on-sync-succeeded fires when the sync operation finished (phase == Succeeded). on-deployed additionally requires the app to be Healthy and is keyed oncePer the sync revision — it’s the “actually live and passing health checks” signal. Alerting on on-sync-succeeded can page people before pods are ready; on-deployed waits for health.

Q: A template field renders as an empty string. Why, and how do you find the bug? A: A wrong path in a Go template renders empty rather than erroring — e.g. .app.spec.source.repoURL on a multi-source app (which uses spec.sources, a list), or a simple typo. Compare the path against a real kubectl get app <name> -o yaml, and for multi-source use (index .app.spec.sources 0).repoURL. Test with argocd admin notifications template notify before shipping.

Q: How does a secret get referenced by a service, and what’s the classic failure? A: The cm uses $key (e.g. token: $slack-token) and the controller resolves key from argocd-notifications-secret. The classic failure is a name mismatch — the secret key is slackToken but the cm says $slack-token — which resolves to empty and yields a generic auth error with no “key not found” hint. Verify the exact key name in the secret first.

Q: How do you test a notification without breaking an application? A: argocd admin notifications template notify <template> <app> --recipient <service>:<recipient> renders the template against the app’s live state and delivers it; argocd admin notifications trigger run <trigger> <app> reports whether the when matches right now. You can also lint from files with --config-map ./cm.yaml --secret :empty, which is CI-friendly.

Q: You need to alert on production apps only. Two ways to scope that? A: (1) A custom trigger whose when includes a label check: ... and app.metadata.labels.env == 'prod'. (2) A cm subscriptions entry with a selector: env=prod, or per-app annotations only on prod apps. Scoping at the subscription keeps triggers generic; scoping in the trigger keeps the condition self-documenting.

Q: What is .serviceType used for in a template? A: It names the service currently rendering the message, so one template can format per channel — e.g. {{if eq .serviceType "slack"}}:exclamation:{{end}} emits a Slack emoji but nothing for email. It’s how the built-in templates keep a single body that renders sensibly on Slack, Teams, and email.

Q: Why prefer a bot token over an incoming webhook for Slack, and what scope does it need? A: A bot token (xoxb-…) with chat:write posts through the Slack API, supports multiple channels from one config, and gives structured attachments/blocks; a webhook is bound to a single channel. The catch is the bot must be invited to each channel or you get channel_not_found. chat:write.customize is needed only to override username/icon.

Q: A Teams integration you built last year suddenly returns 400. Most likely cause in 2026? A: The Office 365 Incoming Webhook connector it used has been retired (connectors stop working 31 March 2026). Move to a Teams Workflows (Power Automate) URL that accepts an Adaptive Card, or post an Adaptive Card via the generic webhook service. It’s an infrastructure deprecation, not a template bug.

Q: Where would you store the Slack token in a real GitOps setup, and why not in Git? A: In the cloud secret store — Azure Key Vault (AKS), AWS Secrets Manager (EKS), or Google Secret Manager (GKE) — projected into argocd-notifications-secret by External Secrets Operator (or sealed with Sealed Secrets). Plaintext in Git can’t be rotated without a commit and leaks permanently in history; ESO makes rotation a change in the store with no repo change.

---

Glossary

Plain-language definitions for every moving part in this lesson.

---

Key takeaways

• Notifications turn silent reconcile results into events your team actually sees. They complement metrics (continuous signals) by carrying rich, per-object context for discrete transitions like Failed and Degraded.
• Four blocks, four question words: triggers (WHEN), templates (WHAT), services (WHERE), subscriptions (WHO). Every bug is a break in one of them — diagnose by asking which.
• It’s built in. On Argo CD 2.13+/3.x the argocd-notifications-controller already runs; you configure argocd-notifications-cm and argocd-notifications-secret, you don’t install a separate project.
• oncePer is not optional on state triggers. A when over a persistent state (Degraded, OutOfSync) without oncePer spams every reconcile; key it to app.status.sync.revision for one alert per revision.
• Secrets are referenced, never inlined. $slack-token resolves from argocd-notifications-secret; keep the cm in Git and source the secret from your cloud store via ESO or seal it. The only real cloud edge here is where the token lives — Key Vault, Secrets Manager, or Secret Manager.
• Slack needs a bot invited to the channel; Teams connectors are being retired (31 Mar 2026 → Workflows); the generic webhook wires Argo CD into anything that speaks HTTP, with | toRawJson to keep free text from breaking the body.
• Test without breaking anything: argocd admin notifications template notify and trigger run render and evaluate against a live app, and --secret :empty lets you lint the config from files in CI.