Metrics-Driven Promotion: AnalysisTemplates, Prometheus Queries & Automatic Rollback

In a nutshell

A canary deployment is only useful if something decides whether the canary is healthy. This lesson replaces the human who normally makes that call — the one squinting at a Grafana panel deciding “looks fine, promote” — with a rule you write down once. You tell Argo Rollouts what “healthy” means as a metric query and a threshold (“success rate stays above 95%”), and the controller checks it automatically on every rollout: it promotes when the numbers hold and rewinds to the last working version, on its own, when they don’t.

Think of the new version as a new hire on a probationary period. It’s handed a slice of the real work (a fraction of live traffic), measured against clear targets for a fixed window, and then either kept on (promoted to 100%) or let go automatically when it misses those targets too many times — and because the previous hire never actually left the building (the stable version keeps running the whole time), reversing the decision is instant. You wrote the job description; the controller does the watching and the firing.

Four moving parts carry the whole lesson. An AnalysisTemplate is the rule (“query this, pass if it clears this bar”). An AnalysisRun is one live evaluation of that rule during a deploy. A provider is where the number comes from (usually Prometheus). The abort is the automatic rewind when the run fails. Everything else here is detail on those four.

Level: Expert · Time: ~55 min · Builds directly on Argo Rollouts: Canary & Blue-Green — you should already know the Rollout resource, canary steps, and setWeight, and have Prometheus scraping your workloads. After it you can write a metric-gated AnalysisTemplate, wire it into a Rollout as an inline gate or a background watcher, reason clearly about Failed vs Error, and prove a bad version auto-reverts to stable with nobody watching.

---

A canary deploy is only as good as the thing deciding whether the canary is healthy. In most shops that “thing” is a human: someone pushes a new version to 10% of traffic, opens Grafana, squints at an error-rate panel for ten minutes, and either clicks promote or clicks rollback. That works right up until it doesn’t — the deploy goes out at 2 a.m., or the reviewer gets pulled into a meeting, or the panel they’re watching is the wrong one. The whole promise of progressive delivery is that a bad version should never reach 100% of users. A human with a dashboard is a fragile way to keep that promise.

This lesson is about removing the human from the promotion decision — not the human judgement (you still write the rules), but the human staring. You will teach the canary to query its own vital signs, evaluate them against a threshold you defined in Git, and act: promote when the numbers are good, automatically abort and revert to stable when they are not. Argo Rollouts calls this analysis, and it is the single feature that turns “canary” from a traffic trick into a real safety system.

We assume you already know the Rollout resource and the canary/blue-green strategies from Argo Rollouts: Canary & Blue-Green, and that you have Prometheus scraping your workloads (if not, Observability: Metrics with Prometheus & Grafana gets you there). Everything here targets Argo Rollouts 1.7+ on Kubernetes 1.29+. There is no live cluster behind this page, so every command output below is labelled representative — the shapes, states, and field names are exactly what a real run produces; only the hashes and timestamps will differ on yours.

---

Why this matters

The core problem is a mismatch of speed. A deployment can go from “10% of traffic” to “100% of traffic” in the time it takes to click a button, but knowing whether that 10% is healthy takes minutes of observation and a clear definition of “healthy.” When the two are done by the same tired person under deadline pressure, you get one of two failure modes: you promote too fast (and ship the outage), or you promote too slow (and every deploy becomes an afternoon of babysitting). Neither scales past a handful of services.

Metric-driven promotion fixes the mismatch by writing the definition of “healthy” down, once, as code, and letting the controller enforce it on every rollout forever. The definition lives in an AnalysisTemplate — a small CRD that says “query this metric, this often, and the new version passes only if the number clears this bar.” The Rollout references the template at a specific point in the canary, and the controller creates an AnalysisRun that does the watching for you. If the numbers hold, the rollout proceeds. If they don’t, the rollout aborts itself and shifts all traffic back to the stable version — no page, no click, no Grafana.

The mental model to hold for the rest of this lesson is a control loop, not a pipeline: observe → decide → act, repeated on an interval. Observe the metric. Decide against the condition. Act by continuing or aborting. That loop is what an AnalysisRun is.

The right way to read that table: analysis does not replace your judgement, it encodes it. You still decide that 95% success rate is the line. You just stop being the sensor and the switch.

---

The idea: a canary that watches its own vitals

Picture the canary as a patient in an ICU and the AnalysisRun as the monitor wired to it. The monitor takes a reading every so often (the interval), compares each reading to an alarm threshold (the successCondition / failureCondition), and tolerates a set number of bad readings before it pulls the alarm (the failureLimit). While the readings are good, the surgery continues — traffic keeps shifting to the new version. The instant too many readings go bad, the alarm fires and the controller rolls the patient back to the last known-good state — the stable ReplicaSet.

Three properties make this trustworthy rather than a gimmick:

• It is periodic, not a single snapshot. A one-off check right after a deploy catches nothing — the JVM hasn’t warmed up, caches are cold, the first request always looks weird. Analysis measures count times at interval, so a transient blip doesn’t fail the run and a slow-burning regression doesn’t slip through.
• It tolerates noise deliberately. Real metrics are jittery. failureLimit lets you say “three bad readings out of ten is noise; the fourth is a signal,” so the gate is neither hair-trigger nor asleep.
• The failure action is automatic and safe. Because the stable ReplicaSet is still running (a canary never deletes stable until it’s fully promoted), reverting is instant: scale stable back to full weight, scale the canary to zero. There is nothing to rebuild.

Read the loop left to right below. The Rollout is mid-canary; the controller spawns an AnalysisRun; the run queries Prometheus on an interval; each result is checked against the successCondition; if the condition holds the step promotes, and if failures pile past the limit the Rollout aborts and traffic snaps back to stable. Every numbered badge is a decision point or a failure mode — the places where this either saves you or, misconfigured, bites you.

The two nodes worth tattooing on your brain are the amber gate and the red outcome. The gate (badges 3-4) is where your SLO becomes an executable expression — result[0] >= 0.95 — with a tolerance (failureLimit) that decides how jumpy it is. The red outcome (badge 6) is the payoff of the entire feature: when analysis fails, you do nothing and the right thing happens. That is the difference between “we have canaries” and “we are actually protected by them.”

---

AnalysisTemplate and ClusterAnalysisTemplate, field by field

Analysis is defined by two nearly identical CRDs. An AnalysisTemplate is namespaced — it lives next to the Rollout that uses it. A ClusterAnalysisTemplate is cluster-scoped — define your organisation’s standard SLO checks once and reference them from Rollouts in any namespace. The spec schema is identical between the two; only the scope and how you reference them differ.

Here is a complete, schema-correct AnalysisTemplate with every commonly used field present. Read it once, then we’ll dissect the metric block.

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
  namespace: demo
spec:
  args:
  - name: service-name                     # required at use-time (no default)
  - name: prometheus-address               # has a default, can be overridden
    value: http://prometheus-operated.monitoring.svc.cluster.local:9090
  metrics:
  - name: success-rate                     # identifier for this measurement
    interval: 30s                          # measure every 30s...
    count: 5                               # ...five times, then finish
    initialDelay: 60s                      # wait 60s before the first measurement
    successCondition: result[0] >= 0.95    # pass when success rate >= 95%
    failureLimit: 3                        # tolerate 3 bad readings; the 4th fails the run
    consecutiveErrorLimit: 4               # 4 provider errors in a row = Error (default)
    provider:
      prometheus:
        address: "{{args.prometheus-address}}"
        timeout: 20                        # seconds to wait for the query
        query: |
          sum(rate(
            http_requests_total{app="{{args.service-name}}", code!~"5.."}[2m]
          ))
          /
          sum(rate(
            http_requests_total{app="{{args.service-name}}"}[2m]
          ))

The metric fields, exactly

Every field under spec.metrics[] is confirmed against the Argo Rollouts Metric type. These are the knobs that decide how jumpy, how patient, and how strict your gate is.

Two subtleties beginners miss. First, failureLimit counts total failed measurements across the run, while consecutiveErrorLimit counts consecutive provider errors — a failed measurement (your app is genuinely returning 500s) and an errored measurement (Prometheus timed out) are different things with different budgets. Second, count: 0 is not “measure zero times” — it means “measure forever,” which is exactly what you want for a background analysis that should run for the whole duration of the canary.

Namespaced vs cluster-scoped

A healthy pattern for a platform team: publish ClusterAnalysisTemplates named success-rate, latency-p95, and error-budget that encode the house SLOs, and have every app Rollout reference them with clusterScope: true, passing only the app-specific args (the service name, the namespace). One definition of “healthy,” enforced fleet-wide, owned by the people who own the SLOs.

Arguments: parameterising a template

spec.args turns a template into a reusable function. Each arg is either declared with no value (the caller must supply it), given a default value, or sourced from a Secret with valueFrom.secretKeyRef — the right place for an API token so it never appears in the manifest.

spec:
  args:
  - name: service-name                     # caller must provide this
  - name: prometheus-address
    value: http://prometheus-operated.monitoring.svc:9090   # default
  - name: api-token
    valueFrom:
      secretKeyRef:
        name: datadog-creds
        key: api-token                     # secret, never inline

Inside metrics[], reference an arg anywhere a string is expected with the double-brace syntax: {{args.service-name}}. The controller substitutes the value before sending the query. Miss a required arg at use-time and the AnalysisRun never starts — you’ll see it in the Rollout events, covered in troubleshooting.

---

The metric providers

A provider is the source the analysis queries. Argo Rollouts ships twelve built-in providers, and you pick exactly one per metric. There is deliberately no generic “Kubernetes” provider — when people ask for one they mean the job provider, which runs an arbitrary Kubernetes Job as the check (run any container, any script, any in-cluster probe, and let its exit code be the verdict). For anything truly custom there is the plugin provider, which loads a metric plugin you supply. Everything else is a specific observability backend.

Prometheus — the one you’ll use most

The Prometheus provider runs a PromQL query and exposes the outcome to your condition as result. For an instant query returning a vector, result is an array of the scalar values, and result[0] is the first (usually only) series’ value. The whole art is writing a query that collapses to a single meaningful number — a ratio, a quantile, a rate.

provider:
  prometheus:
    address: http://prometheus-operated.monitoring.svc.cluster.local:9090
    timeout: 20                  # seconds; a slow query shouldn't hang the run
    insecure: false              # set true only to skip TLS verification (dev)
    query: |
      sum(rate(
        http_requests_total{app="checkout", code!~"5.."}[2m]
      ))
      /
      sum(rate(
        http_requests_total{app="checkout"}[2m]
      ))

The result shape is the number-one source of broken conditions, so be precise:

A bare instant vector gives you result[0] as a float; multiple series give you result[0], result[1], …; and a query that matches nothing gives an empty result, where result[0] is an error — the dreaded no-data trap we handle below.

Web — hit any HTTP endpoint

When your source of truth is a REST API (a home-grown gate service, a third-party SLO tool), the web provider fetches a URL and evaluates a jsonPath from the response.

provider:
  web:
    url: https://gate.example.com/api/v1/canary?svc={{args.service-name}}
    method: GET                  # or POST/PUT with a body
    timeoutSeconds: 20
    headers:
    - key: Authorization
      value: "Bearer {{args.api-token}}"
    jsonPath: "{$.approved}"     # pull one field out of the JSON response
# successCondition: "result == true"

Job — run a check instead of querying one

The job provider runs a Kubernetes Job. The measurement is Successful if the Job completes with exit code 0 and Failed otherwise — no successCondition needed. This is your escape hatch for anything you can script: integration tests, synthetic transactions, a load-test-then-check.

provider:
  job:
    metadata:
      labels:
        role: canary-check
    spec:
      backoffLimit: 1
      template:
        spec:
          containers:
          - name: test
            image: ghcr.io/acme/canary-tests:1.4.0
            command: ["pytest", "-q", "--target", "{{args.service-name}}"]
          restartPolicy: Never

Because the verdict is just an exit code, the job provider is how you run any Kubernetes-native check — the closest thing to the “Kubernetes provider” people expect, and far more flexible.

Cloud-managed metric backends — where the multi-cloud edge appears

The analysis mechanics are entirely cloud-neutral: the same AnalysisTemplate, successCondition, and abort behaviour run identically on AKS, EKS, and GKE. The one place a cloud edge appears is which metric store you query and how the Rollouts controller authenticates to it. If you run a plain in-cluster Prometheus, there is no cloud edge at all — skip this table. The moment you point analysis at a managed metric service, each cloud authenticates its own way, and you grant that identity to the argo-rollouts controller ServiceAccount (namespace argo-rollouts), because the controller is the process making the query.

Two paired examples make the difference concrete. On EKS with Amazon Managed Prometheus, SigV4 signing is the whole story — no address-embedded credentials:

# EKS + Amazon Managed Prometheus
provider:
  prometheus:
    address: https://aps-workspaces.us-east-1.amazonaws.com/workspaces/ws-abc123/
    authentication:
      sigv4:
        region: us-east-1
        roleArn: arn:aws:iam::111122223333:role/argo-rollouts-amp-query
    query: |
      sum(rate(http_requests_total{app="checkout",code!~"5.."}[2m]))
      / sum(rate(http_requests_total{app="checkout"}[2m]))

On AKS with Azure Monitor managed Prometheus, OAuth2 client-credentials against the query endpoint, with the client secret pulled from a Kubernetes Secret via an arg:

# AKS + Azure Monitor managed Prometheus
provider:
  prometheus:
    address: https://my-workspace.eastus.prometheus.monitor.azure.com
    authentication:
      oauth2:
        tokenUrl: https://login.microsoftonline.com/<tenant-id>/oauth2/v2.0/token
        clientId: "{{args.entra-client-id}}"
        clientSecret: "{{args.entra-client-secret}}"
        scopes:
        - https://prometheus.monitor.azure.com/.default
    query: |
      sum(rate(http_requests_total{app="checkout",code!~"5.."}[2m]))
      / sum(rate(http_requests_total{app="checkout"}[2m]))

On GKE, the cleanest path avoids auth in the template entirely: deploy the Managed Service for Prometheus frontend in-cluster (it runs under Workload Identity and adds Google credentials for you), and point the analysis at it like any ordinary Prometheus — address: http://frontend.gmp-system.svc:9090. The cloud edge collapses back into a plain Prometheus query.

For deeper coverage of registering these identities — IRSA vs EKS Pod Identity, Entra Workload Identity, GKE Workload Identity — the per-cloud lessons on running Argo CD on each platform go field by field; here the only new fact is that the Rollouts controller SA is the identity that needs the read permission, not Argo CD’s.

---

Writing the success and failure conditions

The condition is where your SLO becomes executable. Conditions are expressions in expr evaluated against result. The rules that decide a measurement’s phase are simple but easy to get backwards:

So successCondition: result[0] >= 0.95 alone means “anything under 95% is a failure.” If you’d rather express the bad case directly — often clearer for error budgets — use failureCondition: result[0] > 0.05 (“more than 5% errors is a failure”). Using both creates a deliberate dead-band in the middle that is neither pass nor fail — an Inconclusive zone that pauses the rollout for a human instead of promoting or aborting. That’s the tool for “I’m not confident enough to auto-decide in this range.”

The condition language gives you a small, reliable toolkit — learn these five pieces and you can express almost any gate:

The limit fields decide how jumpy the gate is

A single measurement almost never decides a run — the limits do. Each phase has its own budget:

The interaction to internalise: with the defaults (failureLimit: 0), the very first failed measurement fails the whole run. That is far too jumpy for real metrics — one slow scrape and your good deploy aborts. Production templates almost always raise failureLimit to tolerate a few blips over a count of several measurements. A common, sane shape is count: 5, interval: 1m, failureLimit: 2 — “measure five times a minute apart; abort only if three of them are bad.”

Timing: count, interval, initialDelay

initialDelay is the field people forget and then blame the tool. A brand-new pod has cold caches, an unwarmed JVM, and near-zero traffic; measured immediately, its “success rate” is noise. Give it a warm-up. And interval should never be shorter than your metric’s freshness — if Prometheus aggregates over a 2-minute window and scrapes every 30s, an interval: 10s just re-reads the same stale number four times.

The no-data trap — the failure mode that fools everyone

Here is the gotcha that costs teams a weekend. Your successCondition: result[0] >= 0.95 looks airtight. Then a canary goes out during a quiet hour, receives almost no traffic, and your success-rate query — a ratio with near-zero denominator — returns empty or NaN. Now result[0] either throws (an Error measurement, eating your consecutiveErrorLimit) or is NaN (the comparison is false, a Failed measurement). Either way a perfectly healthy deploy gets aborted because there wasn’t enough traffic to judge it — or, if you “fixed” it the wrong way with a permissive default, a genuinely broken deploy sails through because no-data was treated as pass.

There is no free lunch here; you choose a risk posture:

• Guarantee traffic during analysis. Run a small load generator against the canary so the query always has data. This is the most robust and the approach the lab uses.
• Guard the expression with a default. Providers like Datadog support default(result, 0) to substitute a value when the result is missing. Choosing the fallback is the risk decision: default(result[0], 0) treats no-data as failure (fail-safe, may abort good deploys in quiet windows); default(result[0], 1) treats it as success (fail-open, dangerous — a broken build with no traffic promotes).
• Widen the window or raise failureLimit. A longer rate window and more tolerance ride out brief data gaps instead of reacting to them.

The instinct to reach for default(..., 1) “so my quiet-hour deploys stop failing” is exactly how a real outage promotes itself. Prefer guaranteeing traffic. If you must default, default to the fail-safe value and accept the occasional false abort.

---

AnalysisRun: the instance

An AnalysisTemplate is a class; an AnalysisRun is the object instantiated from it when analysis actually runs. The controller creates it, records every measurement in its status, and drives it to a terminal phase. You rarely write one by hand — you read them to understand what a rollout decided and why.

There are two levels of “phase,” and confusing them is a common source of muddled debugging. Each individual measurement gets a phase; the accumulated measurements, weighed against the limits, decide the run’s phase. First, the per-measurement phases:

And then the run’s overall status.phase, which is what the Rollout actually reacts to:

The distinction that matters operationally: Failed and Error both abort, but they mean opposite things. Failed = “your app is genuinely bad” (the metric cleared the failure bar). Error = “I couldn’t measure your app” (Prometheus was down, the query timed out, credentials expired). Chasing a code bug when the real problem was an unreachable Prometheus is a classic wasted afternoon — always read the run’s message to see which it was.

AnalysisRuns are named after the Rollout, the ReplicaSet’s pod-template-hash, and a counter (e.g. rollouts-demo-687d76d795-2). Inspect them with standard kubectl:

# List the runs for a namespace
kubectl -n demo get analysisrun
# NAME                          STATUS      AGE
# rollouts-demo-687d76d795-2    Running     45s

# The full story: every measurement, its value, its phase
kubectl -n demo get analysisrun rollouts-demo-687d76d795-2 -o yaml

The status.metricResults[].measurements[] array is the receipt — each entry has a phase, a value (the actual number the query returned), and timestamps. When someone asks “why did the deploy roll back?”, that array is the answer, and it’s in the cluster, not in anyone’s memory.

---

Wiring analysis into a Rollout

A template that no Rollout references never runs. There are four placements, and choosing the right one is a real design decision. The table first, then each in code.

Inline: a gate between canary steps

The most common shape. analysis is a step, so the rollout stops at it and won’t advance until the run reaches Successful.

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: rollouts-demo
  namespace: demo
spec:
  replicas: 5
  revisionHistoryLimit: 3
  selector:
    matchLabels:
      app: rollouts-demo
  template:
    metadata:
      labels:
        app: rollouts-demo
    spec:
      containers:
      - name: rollouts-demo
        image: argoproj/rollouts-demo:blue
        ports:
        - name: http
          containerPort: 8080
        resources:
          requests: { cpu: 5m, memory: 32Mi }
  strategy:
    canary:
      steps:
      - setWeight: 20
      - pause: { duration: 30s }
      - analysis:                      # <-- the gate
          templates:
          - templateName: success-rate
          args:
          - name: service-name
            value: rollouts-demo
      - setWeight: 50
      - pause: { duration: 30s }
      - setWeight: 80
      - pause: { duration: 30s }

Background: watch the whole rollout

strategy.canary.analysis starts a run that lives for the entire rollout and can abort it at any step, not just at a gate. Pair it with a template using count: 0 (continuous). startingStep delays the watch until a given step index, so you don’t judge the version before any traffic reaches it.

  strategy:
    canary:
      analysis:
        templates:
        - templateName: success-rate
        args:
        - name: service-name
          value: rollouts-demo
        startingStep: 2               # don't start analysing until step index 2
      steps:
      - setWeight: 20
      - pause: { duration: 1m }
      - setWeight: 50                  # from here, background analysis is watching
      - pause: { duration: 1m }
      - setWeight: 80
      - pause: { duration: 1m }

Inline and background are complementary, not exclusive: inline gates make discrete promotion decisions (“is 20% healthy enough to earn 50%?”), while background analysis is a continuous safety net that catches a regression that only shows up at higher weight. Many production Rollouts use both.

Blue-green: pre- and post-promotion

For blue-green, the two hooks bracket the cutover. prePromotionAnalysis runs against the preview Service before flipping traffic; postPromotionAnalysis runs against the active Service after the flip and can roll the cutover back.

  strategy:
    blueGreen:
      activeService: rollouts-demo-active
      previewService: rollouts-demo-preview
      autoPromotionEnabled: false
      prePromotionAnalysis:            # gate BEFORE traffic cutover
        templates:
        - templateName: smoke-tests
        args:
        - name: service-name
          value: rollouts-demo-preview.demo.svc.cluster.local
      postPromotionAnalysis:           # verify AFTER cutover; revert if bad
        templates:
        - templateName: success-rate
        args:
        - name: service-name
          value: rollouts-demo-active.demo.svc.cluster.local

Passing args at use-time, including the canary’s own hash

Notice the args under each analysis reference — that’s where the caller fills in the template’s declared arguments. Beyond static values, valueFrom can inject dynamic facts about the rollout itself, which is how you scope a query to only the new pods:

        args:
        - name: service-name
          value: rollouts-demo
        - name: canary-hash
          valueFrom:
            podTemplateHashValue: Latest     # measure ONLY the new pods

Then your PromQL filters on that hash — ...{rollouts_pod_template_hash="{{args.canary-hash}}"}... — so the success rate reflects the canary alone, not a blend of canary and stable that hides the regression. (This assumes your metrics carry the pod-template-hash as a label, e.g. via kube-state-metrics or pod relabeling; without traffic routing that splits canary and stable, scoping the query is the only way to isolate the new version’s health.)

---

Automatic rollback: the payoff

Everything so far exists to make one sentence true: when analysis fails, you do nothing and traffic returns to the last good version. Here is exactly what the controller does the moment an inline or background AnalysisRun reaches Failed (or Error):

1. It sets the Rollout’s status.abort: true and stamps status.abortedAt.
2. It scales the stable ReplicaSet back to full and the canary to zero — traffic snaps back to the known-good version, immediately, because stable was never torn down.
3. The Rollout’s phase becomes Degraded with a message naming the metric that failed.
4. It stays aborted. It does not thrash — it won’t retry the bad revision on its own.

To move forward from an aborted rollout you have two levers. Push a new image (a fix), and the new revision clears the abort and starts a fresh canary. Or, if you believe the abort was a fluke (a flaky metric, a transient Prometheus outage), retry the same revision:

# Re-run the aborted rollout from step 0 (same revision)
kubectl argo rollouts retry rollout rollouts-demo -n demo

# Manually abort a rollout yourself (e.g. you spotted trouble first)
kubectl argo rollouts abort rollouts-demo -n demo

# Manually promote past the current step / to full
kubectl argo rollouts promote rollouts-demo -n demo

The metric that matters — tie the gate to an SLO

A rollback system is only as trustworthy as the metric it watches. Gate on user-facing SLIs, not on infrastructure vanity metrics. CPU being high doesn’t mean users are unhappy; a rising 5xx rate does. The three that belong in almost every gate:

The error-budget framing is the SRE-grade version: instead of a static “95% is fine,” you gate on how fast the new version is spending this month’s allowed downtime. A fast-burn condition (e.g. burn rate > 14.4× over an hour, the multiwindow-multiburn convention) aborts a version that would exhaust the budget in hours, even if its instantaneous success rate still looks “okay-ish.” If your org runs formal SLOs, gate the canary on burn rate and the promotion decision and the reliability target become the same number.

---

Best practices

Analysis is powerful enough to hurt you if you deploy it aggressively on day one. The path that works:

The last one deserves a pointer: the auto-abort is deliberately quiet — that’s the feature — but a version that rolled itself back is still a signal a human should see. Route it with Argo CD Notifications: Slack, Teams & Webhooks so a failed analysis posts to the channel that owns the service. And if your canary uses real traffic shifting (not just replica-weighting) to route that 20%, the mechanics are in Rollouts Traffic Management: Istio, NGINX, ALB & Gateway API — analysis and traffic routing are the two halves of a real canary.

---

Going deeper

The core of this lesson — templates, providers, conditions, wiring, abort — is the 90% you touch every day. This section is the other 10%: the machinery that separates a working gate from a statistically honest one, plus the CRDs and fields the basic path never mentions.

Experiments: baseline-vs-canary, the honest comparison

Every gate so far compares the canary against a fixed number (>= 0.95). But “95%” is an assumption about what normal looks like, and normal drifts — Monday-morning traffic, a downstream dependency having a bad hour, a noisy-neighbour node. A canary that dips to 94% might be genuinely broken, or the whole cluster might be at 94% right now and the canary is perfectly fine. A static threshold cannot tell those two apart.

The Experiment CRD answers this by running a concurrent baseline. An experiment spins up short-lived ReplicaSets side by side — typically one from the stable spec (“baseline”) and one from the new spec (“canary”) — sends them comparable traffic, and analyses both over a bounded duration. Now you compare the canary against a baseline living through the exact same conditions, so cluster-wide weather cancels out. It is the idea Spinnaker’s Kayenta popularised, done natively in Rollouts.

In a canary Rollout you drop it in as an experiment step:

strategy:
  canary:
    steps:
    - setWeight: 20
    - experiment:
        duration: 10m
        templates:
        - name: baseline
          specRef: stable          # a fresh pod from the STABLE spec
        - name: canary
          specRef: canary          # a fresh pod from the NEW spec
        analyses:
        - name: compare
          templateName: compare-baseline-canary
          args:
          - name: baseline-hash
            valueFrom: { podTemplateHashValue: Stable }
          - name: canary-hash
            valueFrom: { podTemplateHashValue: Latest }

The referenced template compares the two directly — one PromQL that divides canary success rate by baseline success rate, so the gate is relative, not absolute:

metrics:
- name: canary-vs-baseline
  interval: 1m
  count: 8
  failureLimit: 2
  successCondition: result[0] >= 0.95     # canary within 5% of the live baseline
  provider:
    prometheus:
      address: http://prom-prometheus-server.monitoring.svc.cluster.local
      query: |
        (
          sum(rate(http_requests_total{rollouts_pod_template_hash="{{args.canary-hash}}",code!~"5.."}[2m]))
          / sum(rate(http_requests_total{rollouts_pod_template_hash="{{args.canary-hash}}"}[2m]))
        )
        /
        (
          sum(rate(http_requests_total{rollouts_pod_template_hash="{{args.baseline-hash}}",code!~"5.."}[2m]))
          / sum(rate(http_requests_total{rollouts_pod_template_hash="{{args.baseline-hash}}"}[2m]))
        )

What makes an experiment different from plain analysis: it creates its own pods (they don’t serve canary weight unless you also set weight on the template), it has a hard duration after which its ReplicaSets are torn down, and if any of its analyses fail the experiment fails — which aborts the Rollout exactly like an inline analysis failure. Standalone Experiment objects also exist for A/B/n tests unrelated to a rollout; there each template carries an inline template: pod spec, because specRef: stable|canary only has meaning inside a Rollout’s experiment step.

dryRun and measurementRetention: observe-only, done properly

Best Practices above said “start observe-only” and reached for a generous failureLimit. That works, but Rollouts has a purpose-built field for it: dryRun. A metric listed under spec.dryRun is still queried and recorded on every interval, but its result cannot fail the run — it is a shadow metric. This is how you roll a new SLI into production analysis and watch it behave for a week before it is allowed to abort anything, without weakening the real gate the way a loose failureLimit does.

Its companion measurementRetention controls how many of a metric’s measurements are kept in the AnalysisRun status — raise it when you want the full history of a long background run visible in kubectl get analysisrun -o yaml rather than just the most recent readings.

spec:
  dryRun:
  - metricName: latency-p99          # measured & recorded, but never fails the run
  measurementRetention:
  - metricName: success-rate
    limit: 25                         # retain 25 measurements in status
  metrics:
  - name: success-rate
    interval: 1m
    count: 10
    successCondition: result[0] >= 0.99
    failureLimit: 1
    provider: { prometheus: { address: "{{args.addr}}", query: "..." } }
  - name: latency-p99                 # this one is in dryRun — informational only
    interval: 1m
    count: 10
    successCondition: result[0] <= 0.5
    provider: { prometheus: { address: "{{args.addr}}", query: "..." } }

metricName accepts a regex, so metricName: ".*" puts an entire template in dry-run — the cleanest way to trial a whole new gate. Both fields can also be set on the Rollout’s analysis reference block, so one Rollout can dry-run a metric that fails the run for everyone else.

Composing templates: many metrics, one AnalysisRun

A Rollout’s analysis.templates is a list on purpose. Reference several templates at one gate and the controller merges their metrics, args, dryRun, and measurementRetention into a single AnalysisRun. This is how a platform team composes house SLOs — keep success-rate, latency-p95, and error-budget-burn as separate ClusterAnalysisTemplates, and an app gate pulls in all three at once:

- analysis:
    templates:
    - templateName: success-rate
      clusterScope: true
    - templateName: latency-p95
      clusterScope: true
    args:
    - name: service-name
      value: checkout

Two rules make or break this. Metric names must be unique across every merged template — two metrics both named success-rate is a hard error and the run never starts. And an arg declared in more than one template must resolve to the same value. The merged run passes only if every metric passes within its own limits: the gate is an AND across all of them.

When a static threshold isn’t enough: variance and statistical scoring

Everything above still compares a number to a bar. The deeper problem seasoned teams hit is variance. A success-rate ratio computed over a one-minute window during low traffic is a ratio of small integers, and those jump around violently: 99 good out of 100 is 99%, but a single extra error drops it to 98%. Tighten the threshold to catch real regressions and it trips on noise; loosen it to survive noise and real regressions slip through. failureLimit over a count of measurements is a crude smoother, not a cure.

Three levers make a gate statistically honest rather than lucky:

That last lever is Automated Canary Analysis (ACA). The kayenta provider hands the canary and baseline metric streams to a Spinnaker Kayenta service, which runs a non-parametric test (the Mann-Whitney U test) per metric and returns a single canary score from 0–100. You gate on the score — say > 75 pass, < 60 fail, in-between marginal — instead of on each raw metric.

The win is that the verdict accounts for how much data and how much natural variation there is: a 1% success-rate gap over a million requests is damning; the same gap over 200 requests is noise, and a statistical test knows the difference where result[0] >= 0.95 never can. Most teams don’t need Kayenta — a widened window plus an Experiment baseline captures the bulk of the benefit with no extra service to run. Reach for ACA when you have high request volume, formal SLOs, and a mandate that no promotion happens without a defensible statistical verdict.

Housekeeping: AnalysisRun garbage collection

Every rollout that runs analysis leaves an AnalysisRun object behind, and background runs on a busy fleet accumulate fast. Two controls keep the cluster tidy. On the Rollout, spec.analysis caps how many completed runs are retained:

spec:
  analysis:
    successfulRunHistoryLimit: 5       # keep the last 5 successful AnalysisRuns
    unsuccessfulRunHistoryLimit: 5     # and the last 5 failed/errored ones — keep these!

Keep a healthy unsuccessfulRunHistoryLimit: those are exactly the runs you read in a post-incident review to see which measurement aborted a deploy. For time-based cleanup, an AnalysisTemplate can carry a ttlStrategy:

spec:
  ttlStrategy:
    secondsAfterCompletion: 3600       # delete the AnalysisRun 1h after it finishes

On a large multi-cluster estate this matters more than it sounds — unbounded AnalysisRun accumulation is a real cause of etcd bloat and a laggy Rollouts dashboard.

---

Hands-on lab

You’ll build a metric-gated canary end to end on a free local cluster, then watch a healthy version promote and a bad version auto-abort and revert to stable. Everything runs on kind — no cloud, no bill. The two rollout outcomes are shown as representative output (this page has no live cluster), but every manifest is real and applies cleanly, and the states/fields are exactly what you’ll see.

⚠️ Nothing in this lab bills. It is entirely local (kind + in-cluster Prometheus). The cloud-managed-Prometheus variants from the providers section would incur cost (AMP query units, Azure Monitor ingestion) — don’t point this lab at them.

Step 1 — Cluster, controller, and the kubectl plugin

# A throwaway local cluster
kind create cluster --name analysis-lab

# Install the Argo Rollouts controller
kubectl create namespace argo-rollouts
kubectl apply -n argo-rollouts \
  -f https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml

# Install the kubectl plugin (macOS/Linux via krew, or grab the binary)
kubectl krew install argo-rollouts
kubectl argo rollouts version
# kubectl-argo-rollouts: v1.7.x

What just happened: you have the controller that reconciles Rollout and AnalysisRun objects, plus the kubectl argo rollouts plugin that renders them far more readably than raw kubectl get.

Step 2 — A minimal Prometheus

Install a small Prometheus (the community chart is fine; kube-prometheus-stack works too). The only requirement is that it scrapes our demo app’s /metrics.

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm install prom prometheus-community/prometheus \
  --namespace monitoring --create-namespace
# Its in-cluster address will be:
#   http://prom-prometheus-server.monitoring.svc.cluster.local

What just happened: you now have a metric store the AnalysisRun can query. In a real cluster this is your existing Prometheus; here it’s a throwaway.

Step 3 — The AnalysisTemplate

Save and apply the success-rate template. Note the guarded, canary-tolerant settings: a warm-up, five measurements, and room for two blips.

# analysis-template.yaml
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: success-rate
  namespace: demo
spec:
  args:
  - name: service-name
  - name: prometheus-address
    value: http://prom-prometheus-server.monitoring.svc.cluster.local
  metrics:
  - name: success-rate
    interval: 20s
    count: 5
    initialDelay: 30s
    successCondition: result[0] >= 0.95
    failureLimit: 2
    provider:
      prometheus:
        address: "{{args.prometheus-address}}"
        timeout: 15
        query: |
          sum(rate(
            http_requests_total{app="{{args.service-name}}", code!~"5.."}[1m]
          ))
          /
          sum(rate(
            http_requests_total{app="{{args.service-name}}"}[1m]
          ))

kubectl create namespace demo
kubectl apply -f analysis-template.yaml

What just happened: the definition of “healthy” for this service now lives in the cluster (and, in real life, in Git). It measures success rate every 20s, five times, after a 30s warm-up, and fails only if three of five readings drop below 95%.

Step 4 — The Rollout that references it, plus traffic

# rollout.yaml
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: rollouts-demo
  namespace: demo
spec:
  replicas: 5
  revisionHistoryLimit: 3
  selector:
    matchLabels:
      app: rollouts-demo
  template:
    metadata:
      labels:
        app: rollouts-demo
      annotations:
        prometheus.io/scrape: "true"     # let the community Prometheus scrape it
        prometheus.io/port: "8080"
    spec:
      containers:
      - name: rollouts-demo
        image: argoproj/rollouts-demo:blue
        ports:
        - name: http
          containerPort: 8080
        resources:
          requests: { cpu: 5m, memory: 32Mi }
  strategy:
    canary:
      steps:
      - setWeight: 20
      - pause: { duration: 20s }
      - analysis:
          templates:
          - templateName: success-rate
          args:
          - name: service-name
            value: rollouts-demo
      - setWeight: 60
      - pause: { duration: 20s }
      - setWeight: 100
---
apiVersion: v1
kind: Service
metadata:
  name: rollouts-demo
  namespace: demo
spec:
  selector:
    app: rollouts-demo
  ports:
  - port: 80
    targetPort: 8080

kubectl apply -f rollout.yaml
# Generate steady traffic so the query always has data (dodges the no-data trap):
kubectl -n demo run loadgen --image=busybox --restart=Never -- \
  /bin/sh -c 'while true; do wget -q -O- http://rollouts-demo/ >/dev/null 2>&1; done'

What just happened: the initial revision comes up as stable, the Service gives the loadgen something to hit, and steady traffic keeps the success-rate query populated. The gate sits at step 3. (The PromQL here assumes your app exports http_requests_total{app,code}; adapt the metric name in the template to whatever your workload actually exposes — the mechanism is identical regardless of the metric’s name.)

Step 5 — A healthy update (analysis passes → promotes)

Trigger a canary by changing the image to another good build, then watch:

kubectl argo rollouts set image rollouts-demo \
  rollouts-demo=argoproj/rollouts-demo:green -n demo

kubectl argo rollouts get rollout rollouts-demo -n demo --watch

Representative output while the gate is evaluating, then after it passes:

Name:            rollouts-demo
Namespace:       demo
Status:          ॥ Paused
Message:         CanaryPauseStep
Strategy:        Canary
  Step:          3/6
  SetWeight:     20
  ActualWeight:  20
Images:          argoproj/rollouts-demo:blue (stable)
                 argoproj/rollouts-demo:green (canary)

NAME                                        KIND         STATUS        AGE  INFO
⟳ rollouts-demo                             Rollout      ॥ Paused      6m
├──# revision:2
│  ├──⧉ rollouts-demo-687d76d795            ReplicaSet   ✔ Healthy     40s  canary
│  │  └──□ rollouts-demo-687d76d795-abc     Pod          ✔ Running     40s  ready:1/1
│  └──α rollouts-demo-687d76d795-2          AnalysisRun  ◌ Running     20s  ✔ 2
└──# revision:1
   └──⧉ rollouts-demo-6cf78c96b5            ReplicaSet   ✔ Healthy     6m   stable

The AnalysisRun line (α) shows ◌ Running ✔ 2 — two successful measurements so far. Once it reaches five successes, the run flips to ✔ Successful, the gate opens, and the rollout walks the remaining steps to setWeight: 100, at which point revision 2 becomes stable:

Status:          ✔ Healthy
  Step:          6/6
  SetWeight:     100
Images:          argoproj/rollouts-demo:green (stable)
   └──α rollouts-demo-687d76d795-2          AnalysisRun  ✔ Successful  3m   ✔ 5

What just happened: nobody watched Grafana. The canary measured itself five times, cleared 95% every time, and promoted on its own.

Step 6 — A bad update (analysis Failed → auto-abort → revert to stable)

Now ship a “bad” version — one that returns 5xx. The rollouts-demo app has a built-in error slider in its web UI you can crank up to make the new pods return errors; in your own service this is simply a build with a regression. We change the image tag to trigger a fresh canary revision, then drive its error rate up. The success-rate query collapses below 0.95, three of five readings fail, and the run fails.

# New tag = new revision = new canary; the ':red' tag is just a distinct build to roll out
kubectl argo rollouts set image rollouts-demo \
  rollouts-demo=argoproj/rollouts-demo:red -n demo
# ...then raise the error rate on the canary pods (demo UI slider, or deploy a genuinely bad build)

kubectl argo rollouts get rollout rollouts-demo -n demo --watch

Representative output as the gate fails:

Name:            rollouts-demo
Status:          ✖ Degraded
Message:         RolloutAborted: Rollout aborted update to revision 3: metric "success-rate" assessed Failed: failed (3) > failureLimit (2)
Strategy:        Canary
  Step:          3/6
  ActualWeight:  0
Images:          argoproj/rollouts-demo:green (stable)
                 argoproj/rollouts-demo:red   (canary, scaled down)

NAME                                        KIND         STATUS        AGE  INFO
⟳ rollouts-demo                             Rollout      ✖ Degraded    9m
├──# revision:3
│  ├──⧉ rollouts-demo-7c9fbb6d84            ReplicaSet   •             30s  canary
│  └──α rollouts-demo-7c9fbb6d84-3          AnalysisRun  ✖ Failed      25s  ✖ 3
└──# revision:2
   └──⧉ rollouts-demo-6b5c8d7f9c            ReplicaSet   ✔ Healthy     6m   stable

Inspect the run to see the receipt:

kubectl -n demo get analysisrun rollouts-demo-7c9fbb6d84-3 \
  -o jsonpath='{.status.phase} {.status.metricResults[0].failed}{"\n"}'
# Failed 3

kubectl -n demo describe analysisrun rollouts-demo-7c9fbb6d84-3 | grep -A2 Message
# Message: metric "success-rate" assessed Failed:
#          failed (3) > failureLimit (2)

What just happened — the payoff: three of five measurements came back under 95%, the run hit Failed, and the controller aborted the rollout and scaled the bad canary to zero — traffic is entirely back on the green stable version. ActualWeight: 0, revision 3 canary scaled down, revision 2 still Healthy and serving. No human touched it. To move on you’d set image to a fix (new revision) or retry if you thought the metric lied.

Step 7 — Teardown

kubectl -n demo delete pod loadgen --ignore-not-found
kind delete cluster --name analysis-lab
# One command removes the cluster, the controller, Prometheus, and everything above.

---

Practice challenges

Work these in order on the kind cluster from the lab (or any cluster with the Rollouts controller and a Prometheus). Each builds on the last. Try it before opening the solution.

1 (Beginner) — Read a live AnalysisRun. During a canary, find the AnalysisRun and report its phase and how many measurements have succeeded and failed.

<details> <summary>Show solution</summary>

kubectl -n demo get analysisrun
kubectl -n demo get analysisrun <name> \
  -o jsonpath='{.status.phase} succ={.status.metricResults[0].successful} fail={.status.metricResults[0].failed}{"\n"}'

Why: the AnalysisRun status is the single source of truth for why a rollout promoted or aborted — reading it fluently is the core operational skill. </details>

2 (Beginner → Intermediate) — Stop a good deploy aborting on one blip. A template has count: 5, interval: 20s and no failureLimit, and healthy deploys occasionally abort. Fix it to tolerate two bad readings and warm up first.

<details> <summary>Show solution</summary>

metrics:
- name: success-rate
  interval: 20s
  count: 5
  initialDelay: 30s      # let the new pods warm up before judging them
  failureLimit: 2        # was absent → defaulted to 0 (hair-trigger)
  successCondition: result[0] >= 0.95

Why: failureLimit defaults to 0, so the very first jittery scrape fails the whole run; real metrics need a noise budget spread over several measurements. </details>

3 (Intermediate) — Add a latency gate. Add a second metric that fails the run when p95 request latency exceeds 300 ms.

<details> <summary>Show solution</summary>

- name: latency-p95
  interval: 1m
  count: 5
  failureLimit: 1
  successCondition: result[0] <= 0.3       # seconds
  provider:
    prometheus:
      address: "{{args.prometheus-address}}"
      query: |
        histogram_quantile(0.95,
          sum(rate(http_request_duration_seconds_bucket{app="{{args.service-name}}"}[5m])) by (le)
        )

Why: histogram_quantile over a _bucket metric is the canonical latency SLI; the value is in seconds, so 0.3 is 300 ms. </details>

4 (Intermediate → Advanced) — Measure only the canary. Your success rate looks healthy but hides a canary regression because the query includes stable pods. Scope it to the new ReplicaSet.

<details> <summary>Show solution</summary>

Pass the canary’s pod-template-hash as an arg, then filter the query on it:

# in the Rollout's analysis.args
- name: canary-hash
  valueFrom:
    podTemplateHashValue: Latest

sum(rate(http_requests_total{rollouts_pod_template_hash="{{args.canary-hash}}",code!~"5.."}[2m]))
/ sum(rate(http_requests_total{rollouts_pod_template_hash="{{args.canary-hash}}"}[2m]))

Why: a blended canary+stable metric dilutes the canary’s errors with stable’s health, so a real regression can clear the bar; scoping isolates the version under test. </details>

5 (Advanced) — Inline gate plus background safety net. Give one Rollout an inline gate at 20% and a separate background analysis that watches continuously from step 2 onward. Say what each one catches.

<details> <summary>Show solution</summary>

strategy:
  canary:
    analysis:                         # background: continuous safety net
      templates: [{ templateName: success-rate-bg }]   # a count: 0 variant
      args: [{ name: service-name, value: checkout }]
      startingStep: 2
    steps:
    - setWeight: 20
    - pause: { duration: 30s }
    - analysis:                       # inline: hard gate at this step
        templates: [{ templateName: success-rate }]
        args: [{ name: service-name, value: checkout }]
    - setWeight: 60
    - pause: { duration: 30s }
    - setWeight: 100

Why: the inline gate makes a discrete promote-or-hold decision (“earn 60%”), while the background run (a count: 0 template) is a continuous net that catches a regression only showing up at higher weight, and can abort at any step. </details>

6 (Advanced → Expert) — Baseline-vs-canary. Replace the absolute 95% gate with an Experiment step that compares the canary to a concurrently-running baseline, passing only if the canary is within 5% of it.

<details> <summary>Show solution</summary>

- experiment:
    duration: 10m
    templates:
    - { name: baseline, specRef: stable }
    - { name: canary,   specRef: canary }
    analyses:
    - name: compare
      templateName: compare-baseline-canary
      args:
      - { name: baseline-hash, valueFrom: { podTemplateHashValue: Stable } }
      - { name: canary-hash,   valueFrom: { podTemplateHashValue: Latest } }

The compare-baseline-canary template’s condition is successCondition: result[0] >= 0.95 over the canary ÷ baseline success-rate ratio (see Going deeper).

Why: a concurrent baseline cancels out cluster-wide weather — a bad-hour dependency, a traffic swing — so the gate reflects the canary’s relative health instead of an absolute guess. </details>

---

Common beginner mistakes

These are misconceptions about the model, not symptoms to look up — the troubleshooting table below handles symptom → cause → fix. Correct the mental model and most of those symptoms never appear.

• “Writing the AnalysisTemplate protects the rollout.” It does nothing on its own. A template is an unused function until a Rollout references it in a steps[].analysis or canary.analysis block. The protection lives in the reference, not the definition. Prove it on every new Rollout: during a canary, kubectl get analysisrun must show a run — no run, no gate.

• “count: 0 means don’t measure.” It means measure forever. Zero is the value you give a background analysis meant to watch for the entire rollout; a finite count is for inline gates that must finish so the step can advance. Reading count: 0 as “disabled” is exactly backwards.

• “Failed and Error are the same — both roll back.” They abort identically but mean opposite things. Failed = your app genuinely missed the SLO (the metric returned a bad number). Error = the controller couldn’t get a number (Prometheus unreachable, query timeout, expired credentials). Treating an Error as an app bug sends you debugging code when the fix is in your observability plumbing.

• “A higher failureLimit makes the gate weaker.” Backwards. failureLimit: 0 isn’t strong, it’s fragile — one slow scrape aborts a perfectly good deploy, and after two false aborts your team stops trusting the gate and switches it off. Tolerating a few bad readings over a count of several is what makes the gate trustworthy enough to leave on. Noise tolerance is a feature, not a loophole.

• “successCondition was true, so it promotes.” One true measurement is not a pass. The run’s verdict comes from the accumulated measurements weighed against the limits — a pass needs enough good readings (and, if set, consecutiveSuccessLimit) across the whole count. Analysis is a control loop sampled over time, not a single if statement.

• “Analysis replaces my alerts and dashboards.” It replaces the human staring, not the human knowing. The auto-abort is deliberately silent, so a version can roll itself back at 3 a.m. and nobody learns of it until morning unless you wired a notification. Keep alerting on AnalysisRun Failed: the gate is the seatbelt, the alert is still the dashboard warning light.

• “Gate on CPU and memory to be safe.” For promotion, those are vanity metrics — users don’t feel CPU, they feel errors and slowness. A canary can peg a core and serve every request perfectly, or sit at 10% CPU while returning 500s. Gate on user-facing SLIs (success rate, latency percentile, error-budget burn); keep resource metrics for capacity planning, not for the promotion decision.

---

Common mistakes and troubleshooting

Three gotchas cost the most hours:

1. Confusing Failed with Error. Both abort, so people treat them the same and go hunting for an application bug — when half the time the app was fine and Prometheus was the problem. Failed means the metric cleared the failure bar (real regression). Error means the controller couldn’t get a measurement (unreachable provider, timeout, bad auth). Always read status.metricResults[].phase and the message. If it’s Error, fix your observability plumbing, not your code.

2. The no-data trap, again. It is the single most common reason a healthy deploy aborts. A ratio query with no traffic returns empty/NaN, which becomes Error or Failed, and your good canary rolls back at 3 a.m. The instinct to “just default no-data to pass” (default(result[0], 1)) is worse than the disease — now a broken build with no traffic promotes itself. Guarantee traffic during analysis; if you must default, default to fail-safe and raise your tolerance.

3. Forgetting the reference. The most anticlimactic failure: you wrote a beautiful AnalysisTemplate, watched it apply cleanly, and shipped a bad version that promoted straight to 100% — because the Rollout never mentioned the template. An AnalysisTemplate on its own does nothing. Prove the wiring on every new Rollout: during a canary, kubectl -n <ns> get analysisrun must show a run. No run, no gate, no protection.

---

Cheat-sheet

AnalysisTemplate metric fields

Metric providers (pick one per metric)

PromQL patterns for gates (collapse to one number, use result[0])

kubectl argo rollouts commands

---

Interview and exam questions

Q: What is the difference between an AnalysisTemplate and an AnalysisRun? A: The AnalysisTemplate (or ClusterAnalysisTemplate) is the reusable definition — which metric to query, how often, and the pass/fail condition. The AnalysisRun is the instance the controller creates when analysis actually runs during a rollout; it records every measurement and drives to a terminal phase (Successful, Failed, Error, Inconclusive). Template is the class; run is the object.

Q: A canary’s inline analysis reaches Failed. Walk through what happens next. A: The controller sets status.abort: true on the Rollout, scales the stable ReplicaSet back to full and the canary to zero (traffic returns to the known-good version instantly, because stable was never removed), and marks the Rollout Degraded with a message naming the failed metric. It stays aborted — no auto-retry. You move forward by pushing a fixed image (new revision) or kubectl argo rollouts retry rollout (same revision).

Q: Explain failureLimit vs consecutiveErrorLimit. Why are they separate? A: failureLimit budgets Failed measurements — the app genuinely missed the SLO (query returned a bad number). consecutiveErrorLimit budgets consecutive provider Errors — the controller couldn’t get a measurement at all (Prometheus unreachable, timeout, bad auth). They’re separate because “my app is bad” and “I can’t measure my app” are different problems: one is your code, the other is your observability plumbing, and you want different tolerances for each.

Q: Your success-rate condition is result[0] >= 0.95. A deploy during a quiet hour aborts even though the app is fine. Why, and how do you fix it? A: The no-data trap. With almost no traffic, the ratio query returns empty or NaN; result[0] then errors (an Error measurement) or the comparison is false (a Failed measurement), so a healthy deploy aborts. Fix by guaranteeing traffic during analysis (a load generator against the canary), widening the rate window, or guarding with a fail-safe default — never default no-data to “pass,” which would let a broken low-traffic build promote.

Q: When would you use background analysis over inline (per-step) analysis? A: Inline analysis is a hard gate at a specific step — the rollout blocks until it passes, good for “prove 20% is healthy before earning 50%.” Background analysis (strategy.canary.analysis, usually count: 0) runs continuously for the whole rollout and can abort at any point, catching a regression that only appears at higher weight. They’re complementary; many Rollouts use inline gates plus a background safety net.

Q: How do you make an analysis measure only the new (canary) pods rather than a blend of canary and stable? A: Pass the canary’s pod-template-hash as an arg with valueFrom.podTemplateHashValue: Latest, then filter the PromQL on that hash label (rollouts_pod_template_hash="{{args.canary-hash}}"). Alternatively, when using traffic routing, point the query at a canary-only Service. A blended metric dilutes the canary’s errors with stable’s health and can hide the regression.

Q: What does count: 0 mean in a metric, and where is it appropriate? A: count: 0 means measure continuously (not zero times). It’s used for background analysis, which should keep watching for the entire duration of the rollout rather than finishing after a fixed number of measurements. Inline gates, by contrast, use a finite count so the run completes and the step advances.

Q: Which metric providers exist, and how do you run a check that isn’t a metric query at all? A: Twelve built-ins: prometheus, datadog, newRelic, cloudWatch, wavefront, graphite, influxdb, skywalking, kayenta, web, job, and plugin. There is no generic “Kubernetes” provider — to run a check (integration test, smoke test, synthetic probe) you use the job provider, which runs a Kubernetes Job and treats exit code 0 as success. For anything custom, plugin loads a metric plugin you supply.

Q (multi-cloud): You point analysis at Amazon Managed Prometheus on EKS and every run is Error with 403. What’s wrong? A: The identity making the query — the argo-rollouts controller ServiceAccount — lacks permission on the AMP workspace. AMP requires SigV4-signed requests; configure provider.prometheus.authentication.sigv4 with the region and a role ARN, and give the controller SA that role via IRSA or EKS Pod Identity with aps:QueryMetrics. The equivalent on AKS is Entra Workload Identity + oauth2 to the Azure Monitor endpoint; on GKE, Workload Identity to the managed-Prometheus frontend.

Q: How do you introduce analysis to a nervous team without risking aborts on day one? A: Start observe-only: a permissive condition or a high failureLimit so the AnalysisRun records measurements but rarely fails. Watch several real rollouts to learn how the metric behaves (warm-up noise, quiet-hour data gaps), then tighten the condition and lower the tolerance once you trust it. It turns analysis into a monitoring dashboard first and a gate second.

Q: You set both successCondition and failureCondition. What happens when a measurement satisfies neither? A: It’s Inconclusive. If inconclusiveLimit is exceeded, the run is Inconclusive and the Rollout pauses for a human decision rather than promoting or aborting. Using both conditions deliberately creates a dead-band — a “not confident enough to auto-decide” zone — which is exactly when you want a person in the loop instead of an automatic promote or rollback.

Q: Why gate on error-budget burn rate instead of a static success-rate threshold? A: A static “95%” is disconnected from your reliability target. Burn rate expresses how fast the new version is spending the month’s allowed error budget; a fast-burn condition aborts a version that would exhaust the budget in hours even if its instantaneous success rate looks marginally acceptable. It makes the promotion decision and the SLO the same number, which is the SRE-grade way to gate.

---

Key takeaways

• Analysis removes the human from the staring, not the judgement. You still define “healthy” as a condition in Git; the controller becomes the sensor and the switch, on every rollout, forever.
• The loop is observe → decide → act: an AnalysisRun measures a metric count times at interval, checks each result against successCondition/failureCondition, and tolerates noise up to failureLimit before it decides.
• Failed and Error both abort but mean opposite things — a genuinely bad app vs. an unmeasurable one. Read the run’s phase and message before you debug the wrong layer.
• The payoff is automatic and safe: a failed run sets abort, scales the canary to zero, and returns all traffic to the still-running stable ReplicaSet — no page, no click, no rebuild.
• Wire it deliberately: inline steps[].analysis is a hard gate between weights; background canary.analysis is a continuous safety net; blue-green uses prePromotionAnalysis/postPromotionAnalysis. An unreferenced template does nothing.
• Beware the no-data trap: low-traffic ratio queries return empty/NaN and abort good deploys. Guarantee traffic, and never default no-data to “pass.”
• Gate on user-facing SLIs — success rate, latency percentile, error-budget burn — not on CPU or other vanity metrics, and scope the query to the canary so its errors aren’t diluted by stable.
• The mechanics are cloud-neutral; only the metric-store auth is a cloud edge. Grant the argo-rollouts controller SA read access via IRSA/Pod Identity (EKS), Entra Workload Identity (AKS), or GKE Workload Identity — the same template runs on all three.

---

Glossary

AnalysisTemplate — The reusable rule for judging a canary: which metric to query, how often, and the pass/fail condition. Namespaced. The “job description.”

ClusterAnalysisTemplate — The same rule, but cluster-scoped so Rollouts in any namespace can reference it. Where a platform team keeps the house SLOs.

AnalysisRun — One live instance of a template, created by the controller during a rollout. It records every measurement and drives to a terminal verdict. Template is the class; run is the object.

Experiment — A CRD that runs short-lived ReplicaSets (typically a baseline from the stable spec and a canary from the new spec) side by side and analyses both, so you compare the canary against a concurrent baseline instead of a fixed number.

Provider — The source a metric queries: prometheus, datadog, cloudWatch, web, job, and the rest. Exactly one per metric.

Measurement — A single reading an AnalysisRun takes at each interval. Each gets a phase: Successful, Failed, Inconclusive, or Error.

successCondition / failureCondition — Expressions (in the expr language) evaluated against result that decide a single measurement’s phase.

result — The value a provider hands your condition. For a Prometheus instant vector it is an array; result[0] is the first series’ number.

failureLimit — How many Failed measurements the run tolerates before it fails and the Rollout aborts. Defaults to 0 (too jumpy for production).

inconclusiveLimit — How many Inconclusive measurements are tolerated before the run is Inconclusive and the Rollout pauses for a human.

consecutiveErrorLimit — How many back-to-back provider errors (Prometheus down, timeout) are tolerated before the run Errors and aborts. Defaults to 4.

interval / count / initialDelay — How often to measure, how many times, and how long to let the new pods warm up before the first measurement.

Inline analysis — An analysis entry in steps[]: a hard gate that blocks a canary step until the run passes.

Background analysis — strategy.canary.analysis: a run that watches for the whole rollout in parallel with the steps and can abort at any point. Usually count: 0.

startingStep — Delays a background analysis until a given step index, so you don’t judge the version before traffic reaches it.

podTemplateHashValue — An arg source (Latest / Stable) that injects a ReplicaSet’s pod-template-hash, letting a query target only the canary (or baseline) pods.

dryRun — Marks a metric as observe-only: it is measured and recorded but can never fail the run. The proper way to trial a new SLI in production.

measurementRetention — Controls how many of a metric’s measurements are kept in the AnalysisRun status — raise it to keep the full history of a long background run.

Abort — The automatic rewind: on a Failed/Error run the controller scales stable back to full and the canary to zero, and marks the Rollout Degraded. Reversible because stable was never removed.

SLI / SLO — Service Level Indicator (a measured signal such as success rate) and Objective (its target, such as 99.9%). Gate canaries on SLIs, tied to your SLOs.

Error-budget burn rate — How fast a version is spending the month’s allowed unreliability. A fast-burn gate aborts a version that would exhaust the budget in hours, even when its instantaneous success rate looks acceptable.

ACA (Automated Canary Analysis) — Statistically scoring the canary against a baseline (for example Kayenta’s Mann-Whitney test → a 0–100 score) rather than comparing one number to a bar.

No-data trap — When a low-traffic ratio query returns empty/NaN, turning a healthy deploy into an Error/Failed and a needless abort. Guarantee traffic; never default no-data to “pass.”