Skip to main content
Argo Rollouts: Canary Analysis That Auto-Aborts on Bad Metrics

Argo Rollouts: Canary Analysis That Auto-Aborts on Bad Metrics

KU
· Founder & Chief Editor
• 6 min read•Updated •44 views

A bad deploy used to mean a pager at 2am and a manual rollback. Now Argo Rollouts watches the error rate and aborts the canary itself before anyone wakes up.

Key takeaways

  • Argo Rollouts aborts a canary automatically when an AnalysisTemplate fails.
  • Add analysis steps between setWeight steps, query Prometheus for the canary's error ratio, and keep failureLimit at 1 or 2 so one bad scrape is tolerated.
  • Compare canary to stable as a ratio instead of an absolute threshold to catch real regressions.

Argo Rollouts auto-aborts a canary when an AnalysisTemplate step fails: put analysis steps between setWeight steps, query Prometheus for the canary's 5xx ratio, and set failureLimit so a sustained regression fails the run. A failed analysis shifts traffic back to stable and marks the Rollout Degraded, with no human in the loop.

Verified against

Argo Rollouts v1.10.1 (released 2026-10-12), against the argo-rollouts.readthedocs.io analysis, canary and Prometheus provider docs. Checked October 2026.

The worst deploy I remember shipped a serialization bug that only fired on 3% of requests, the ones with a rare payload shape. It passed CI, passed staging, and the canary "looked healthy" to the human watching the dashboard for five minutes before promoting. The errors compounded overnight. We caught it at 2am from a customer email, not our own monitoring. After that we wired Argo Rollouts to judge the canary itself, against Prometheus, and abort without asking a human.

The Rollout replaces your Deployment

Argo Rollouts swaps kind: Deployment for kind: Rollout and adds a strategy (if you are new to the controller, start with progressive delivery with Argo Rollouts). The canary strategy steps traffic up in stages, and between steps it can run an AnalysisTemplate that queries your metrics and votes on whether to continue.

k8s/checkout-api/rollout.yaml
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: checkout-api
spec:
  replicas: 10
  strategy:
    canary:
      canaryService: checkout-api-canary
      stableService: checkout-api-stable
      trafficRouting:
        nginx:
          stableIngress: checkout-api
      steps:
        - setWeight: 10
        - pause: {duration: 2m}
        - analysis:
            templates:
              - templateName: error-rate
        - setWeight: 50
        - pause: {duration: 5m}
        - analysis:
            templates:
              - templateName: error-rate
        - setWeight: 100

At 10% weight it pauses two minutes, then runs the analysis. If the analysis fails, the whole rollout aborts and traffic snaps back to the stable version. No promotion, no page.

The analysis is a Prometheus query with a threshold

The AnalysisTemplate is where the judgment lives. This one measures the canary's 5xx rate and fails if it climbs above 1%:

k8s/checkout-api/analysis-error-rate.yaml
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: error-rate
spec:
  metrics:
    - name: error-rate
      interval: 30s
      count: 4
      successCondition: result[0] < 0.01
      failureLimit: 2
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            sum(rate(http_requests_total{
              service="checkout-api-canary", code=~"5.."}[1m]))
            /
            sum(rate(http_requests_total{
              service="checkout-api-canary"}[1m]))

The mechanics that matter: interval: 30s and count: 4 mean it samples four times over two minutes. successCondition: result[0] < 0.01 is the pass rule; the Prometheus provider returns a vector, so you index element 0. failureLimit: 2 is the number of failed measurements tolerated, so a third bad sample fails the analysis (and aborts the rollout). A single blip doesn't nuke a good deploy, but a sustained problem does. That failureLimit is the single knob people get wrong. Leave it unset and it defaults to 0, so one scrape gap aborts everything; set it too high and a real regression rides through.

Measure the canary against the baseline, not an absolute

An absolute "5xx below 1%" threshold breaks the moment your service has a normally-noisy endpoint. Better is to compare canary against stable, so you catch a relative regression even if the absolute number looks fine. Pass both queries and fail when the canary is meaningfully worse:

YAML
# k8s/checkout-api/analysis-error-ratio.yaml (metric excerpt)
successCondition: len(result) == 0 || result[0] <= 1.5
query: |
  (sum(rate(http_requests_total{service="checkout-api-canary",code=~"5.."}[2m]))
   / sum(rate(http_requests_total{service="checkout-api-canary"}[2m])))
  /
  (sum(rate(http_requests_total{service="checkout-api-stable",code=~"5.."}[2m]))
   / sum(rate(http_requests_total{service="checkout-api-stable"}[2m])))

This fails when the canary's error ratio is more than 1.5x the stable version's. A service that always runs at 0.4% errors won't false-alarm, but a canary that jumps to 0.9% while stable holds at 0.4% (a 2.25x regression) gets caught even though 0.9% is under any sane absolute threshold. We also added a second metric on p99 latency from the same template, so a deploy that's correct but slow also aborts.

Watch it actually work

You drive and observe rollouts with the plugin:

Bash
kubectl argo rollouts get rollout checkout-api --watch
Name:            checkout-api
Status:          ✖ Degraded
Message:         RolloutAborted: metric "error-rate" assessed Failed
...

That Degraded with RolloutAborted is the outcome we wanted from the start: the system caught the bad version at 10% traffic, held it there for about 90 seconds, decided it was worse than stable, and rolled back. Blast radius was one in ten requests for a minute and a half, not everyone for eight hours.

Where this breaks

The ratio query has a sharp edge: when stable serves zero 5xx responses, the denominator is zero and Prometheus returns NaN or +Inf, which fails result[0] <= 1.5 on a perfectly healthy canary. Guard it with a floor such as clamp_min(..., 0.001) on the stable ratio, or pair the ratio with an absolute ceiling. Low-traffic services are the other trap: ten requests a minute at 10% weight is one canary request, and no threshold means anything at that volume. Lengthen the pause or skip analysis there. Finally, an Inconclusive result pauses the rollout instead of aborting it, so someone still has to look. If your rollouts are managed by Argo CD, the abort shows up as a degraded app, which fits the GitOps model of Git as desired state and the cluster reporting reality.

Questions people ask

How does Argo Rollouts decide to abort a canary?

Each metric in an AnalysisTemplate takes count measurements every interval. A measurement fails when successCondition is false or failureCondition is true. When failures exceed failureLimit (default 0), the analysis run fails. For an inline analysis step, that aborts the rollout: canary weight goes back to zero, traffic returns to stable, and the Rollout reports Degraded.

What should failureLimit be for canary analysis?

We use 1 or 2 with interval: 30s and count: 4. The default of 0 means a single missed scrape or a one-off spike kills a good deploy, which trains people to ignore aborts. Set it close to count and a regression has to fail nearly every sample before anything happens. Setting failureLimit: -1 disables the limit entirely, which defeats the purpose.

Can Argo Rollouts compare canary metrics to the stable version?

Yes, by writing a PromQL query that divides the canary's error ratio by the stable's, and asserting the result stays under a factor such as 1.5. This catches relative regressions that absolute thresholds miss on noisy services. Protect the denominator against zero errors on stable, or the query returns NaN and fails healthy canaries.

The call we'd make

Don't ship canary analysis with absolute thresholds only; they either false-alarm on noisy endpoints or miss real regressions on quiet ones. Compare canary to stable as a ratio, gate on both error rate and p99 latency, and set failureLimit to 1 or 2 (against a count of 4) so one bad scrape doesn't abort a healthy deploy. Fancy math is not the point. A machine sampling every 30 seconds beats a human staring at Grafana for five minutes and then going to lunch.

Sources

Explore topics:KubernetesDevOps
React
Share this post
Cite this article
Plain text

Kiril Urbonas (2026). Argo Rollouts: Canary Analysis That Auto-Aborts on Bad Metrics. DevOpsNess. Updated 2026-10-31. https://www.devopsness.com/blog/argo-rollouts-canary-analysis-auto-abort

Markdown

[Argo Rollouts: Canary Analysis That Auto-Aborts on Bad Metrics](https://www.devopsness.com/blog/argo-rollouts-canary-analysis-auto-abort) (Kiril Urbonas, DevOpsNess, updated 2026-10-31)

KU

Kiril Urbonas

Founder & Chief Editor

604 articles
View all articles by Kiril Urbonas

You might have missed

Evergreen posts worth revisiting.