Argo Rollouts: Canary Analysis That Auto-Aborts on Bad Metrics
A bad deploy used to mean a pager at 2am and a manual rollback. Now Argo Rollouts watches the error rate and aborts the canary itself before anyone wakes up.
Key takeaways
- Argo Rollouts aborts a canary automatically when an AnalysisTemplate fails.
- Add analysis steps between setWeight steps, query Prometheus for the canary's error ratio, and keep failureLimit at 1 or 2 so one bad scrape is tolerated.
- Compare canary to stable as a ratio instead of an absolute threshold to catch real regressions.
On this page
Argo Rollouts auto-aborts a canary when an AnalysisTemplate step fails: put analysis steps between setWeight steps, query Prometheus for the canary's 5xx ratio, and set failureLimit so a sustained regression fails the run. A failed analysis shifts traffic back to stable and marks the Rollout Degraded, with no human in the loop.
Verified against
Argo Rollouts v1.10.1 (released 2026-10-12), against the argo-rollouts.readthedocs.io analysis, canary and Prometheus provider docs. Checked October 2026.
The worst deploy I remember shipped a serialization bug that only fired on 3% of requests, the ones with a rare payload shape. It passed CI, passed staging, and the canary "looked healthy" to the human watching the dashboard for five minutes before promoting. The errors compounded overnight. We caught it at 2am from a customer email, not our own monitoring. After that we wired Argo Rollouts to judge the canary itself, against Prometheus, and abort without asking a human.
The Rollout replaces your Deployment
Argo Rollouts swaps kind: Deployment for kind: Rollout and adds a strategy (if you are new to the controller, start with progressive delivery with Argo Rollouts). The canary strategy steps traffic up in stages, and between steps it can run an AnalysisTemplate that queries your metrics and votes on whether to continue.
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: checkout-api
spec:
replicas: 10
strategy:
canary:
canaryService: checkout-api-canary
stableService: checkout-api-stable
trafficRouting:
nginx:
stableIngress: checkout-api
steps:
- setWeight: 10
- pause: {duration: 2m}
- analysis:
templates:
- templateName: error-rate
- setWeight: 50
- pause: {duration: 5m}
- analysis:
templates:
- templateName: error-rate
- setWeight: 100
At 10% weight it pauses two minutes, then runs the analysis. If the analysis fails, the whole rollout aborts and traffic snaps back to the stable version. No promotion, no page.
The analysis is a Prometheus query with a threshold
The AnalysisTemplate is where the judgment lives. This one measures the canary's 5xx rate and fails if it climbs above 1%:
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
name: error-rate
spec:
metrics:
- name: error-rate
interval: 30s
count: 4
successCondition: result[0] < 0.01
failureLimit: 2
provider:
prometheus:
address: http://prometheus.monitoring:9090
query: |
sum(rate(http_requests_total{
service="checkout-api-canary", code=~"5.."}[1m]))
/
sum(rate(http_requests_total{
service="checkout-api-canary"}[1m]))
The mechanics that matter: interval: 30s and count: 4 mean it samples four times over two minutes. successCondition: result[0] < 0.01 is the pass rule; the Prometheus provider returns a vector, so you index element 0. failureLimit: 2 is the number of failed measurements tolerated, so a third bad sample fails the analysis (and aborts the rollout). A single blip doesn't nuke a good deploy, but a sustained problem does. That failureLimit is the single knob people get wrong. Leave it unset and it defaults to 0, so one scrape gap aborts everything; set it too high and a real regression rides through.
Measure the canary against the baseline, not an absolute
An absolute "5xx below 1%" threshold breaks the moment your service has a normally-noisy endpoint. Better is to compare canary against stable, so you catch a relative regression even if the absolute number looks fine. Pass both queries and fail when the canary is meaningfully worse:
# k8s/checkout-api/analysis-error-ratio.yaml (metric excerpt)
successCondition: len(result) == 0 || result[0] <= 1.5
query: |
(sum(rate(http_requests_total{service="checkout-api-canary",code=~"5.."}[2m]))
/ sum(rate(http_requests_total{service="checkout-api-canary"}[2m])))
/
(sum(rate(http_requests_total{service="checkout-api-stable",code=~"5.."}[2m]))
/ sum(rate(http_requests_total{service="checkout-api-stable"}[2m])))
This fails when the canary's error ratio is more than 1.5x the stable version's. A service that always runs at 0.4% errors won't false-alarm, but a canary that jumps to 0.9% while stable holds at 0.4% (a 2.25x regression) gets caught even though 0.9% is under any sane absolute threshold. We also added a second metric on p99 latency from the same template, so a deploy that's correct but slow also aborts.
Watch it actually work
You drive and observe rollouts with the plugin:
kubectl argo rollouts get rollout checkout-api --watch
Name: checkout-api
Status: ✖ Degraded
Message: RolloutAborted: metric "error-rate" assessed Failed
...
That Degraded with RolloutAborted is the outcome we wanted from the start: the system caught the bad version at 10% traffic, held it there for about 90 seconds, decided it was worse than stable, and rolled back. Blast radius was one in ten requests for a minute and a half, not everyone for eight hours.
Where this breaks
The ratio query has a sharp edge: when stable serves zero 5xx responses, the denominator is zero and Prometheus returns NaN or +Inf, which fails result[0] <= 1.5 on a perfectly healthy canary. Guard it with a floor such as clamp_min(..., 0.001) on the stable ratio, or pair the ratio with an absolute ceiling. Low-traffic services are the other trap: ten requests a minute at 10% weight is one canary request, and no threshold means anything at that volume. Lengthen the pause or skip analysis there. Finally, an Inconclusive result pauses the rollout instead of aborting it, so someone still has to look. If your rollouts are managed by Argo CD, the abort shows up as a degraded app, which fits the GitOps model of Git as desired state and the cluster reporting reality.
Questions people ask
How does Argo Rollouts decide to abort a canary?
Each metric in an AnalysisTemplate takes count measurements every interval. A measurement fails when successCondition is false or failureCondition is true. When failures exceed failureLimit (default 0), the analysis run fails. For an inline analysis step, that aborts the rollout: canary weight goes back to zero, traffic returns to stable, and the Rollout reports Degraded.
What should failureLimit be for canary analysis?
We use 1 or 2 with interval: 30s and count: 4. The default of 0 means a single missed scrape or a one-off spike kills a good deploy, which trains people to ignore aborts. Set it close to count and a regression has to fail nearly every sample before anything happens. Setting failureLimit: -1 disables the limit entirely, which defeats the purpose.
Can Argo Rollouts compare canary metrics to the stable version?
Yes, by writing a PromQL query that divides the canary's error ratio by the stable's, and asserting the result stays under a factor such as 1.5. This catches relative regressions that absolute thresholds miss on noisy services. Protect the denominator against zero errors on stable, or the query returns NaN and fails healthy canaries.
The call we'd make
Don't ship canary analysis with absolute thresholds only; they either false-alarm on noisy endpoints or miss real regressions on quiet ones. Compare canary to stable as a ratio, gate on both error rate and p99 latency, and set failureLimit to 1 or 2 (against a count of 4) so one bad scrape doesn't abort a healthy deploy. Fancy math is not the point. A machine sampling every 30 seconds beats a human staring at Grafana for five minutes and then going to lunch.
Sources
Cite this article
Kiril Urbonas (2026). Argo Rollouts: Canary Analysis That Auto-Aborts on Bad Metrics. DevOpsNess. Updated 2026-10-31. https://www.devopsness.com/blog/argo-rollouts-canary-analysis-auto-abort
[Argo Rollouts: Canary Analysis That Auto-Aborts on Bad Metrics](https://www.devopsness.com/blog/argo-rollouts-canary-analysis-auto-abort) (Kiril Urbonas, DevOpsNess, updated 2026-10-31)
More from DevOps
Explore more articles in this category
KYAML: Kubernetes YAML Without the Norway Problem
Kubernetes is promoting KYAML, a strict subset of YAML that any parser accepts. It kills type coercion and indentation bugs without a migration.
Dead-Letter Queues: Handling Messages That Won't Process
A practical guide to dead-letter queues, the pattern that isolates poison messages so one bad payload can't stall your whole pipeline.
Multi-Arch Docker Builds with Buildx: One Image, Every Platform
Our M-series laptops built arm64, our CI built amd64, and prod pulled whichever tag won the race. Buildx and a manifest list ended the chaos.
You might have missed
Evergreen posts worth revisiting.