A bad deploy used to mean a pager at 2am and a manual rollback. Now Argo Rollouts watches the error rate and aborts the canary itself before anyone wakes up.
Argo Rollouts: Canary Analysis That Auto-Aborts on Bad Metrics
The worst deploy I remember shipped a serialization bug that only fired on 3% of requests, the ones with a rare payload shape. It passed CI, passed staging, and the canary "looked healthy" to the human watching the dashboard for five minutes before promoting. The errors compounded overnight. We caught it at 2am from a customer email, not our own monitoring. After that we wired Argo Rollouts to judge the canary itself, against Prometheus, and abort without asking a human.
Argo Rollouts swaps kind: Deployment for kind: Rollout and adds a strategy. The canary strategy steps traffic up in stages, and between steps it can run an AnalysisTemplate that queries your metrics and votes on whether to continue.
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: checkout-api
spec:
replicas: 10
strategy:
canary:
canaryService: checkout-api-canary
stableService: checkout-api-stable
trafficRouting:
nginx:
stableIngress: checkout-api
steps:
- setWeight: 10
- pause: {duration: 2m}
- analysis:
templates:
- templateName: error-rate
- setWeight: 50
- pause: {duration: 5m}
- analysis:
templates:
- templateName: error-rate
- setWeight: 100
At 10% weight it pauses two minutes, then runs the analysis. If the analysis fails, the whole rollout aborts and traffic snaps back to the stable version. No promotion, no page.
The AnalysisTemplate is where the judgment lives. This one measures the canary's 5xx rate and fails if it climbs above 1%:
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
name: error-rate
spec:
metrics:
- name: error-rate
interval: 30s
count: 4
successCondition: result < 0.01
failureLimit: 2
provider:
prometheus:
address: http://prometheus.monitoring:9090
query: |
sum(rate(http_requests_total{
service="checkout-api-canary", code=~"5.."}[1m]))
/
sum(rate(http_requests_total{
service="checkout-api-canary"}[1m]))
The mechanics that matter: interval: 30s and count: 4 mean it samples four times over two minutes. successCondition: result < 0.01 is the pass rule. failureLimit: 2 means two bad samples abort the analysis (and the rollout), so a single blip doesn't nuke a good deploy but a sustained problem does. That failureLimit is the single knob people get wrong. Set it to 0 and one scrape gap aborts everything; set it too high and a real regression rides through.
An absolute "5xx below 1%" threshold breaks the moment your service has a normally-noisy endpoint. Better is to compare canary against stable, so you catch a relative regression even if the absolute number looks fine. Pass both queries and fail when the canary is meaningfully worse:
successCondition: len(result) == 0 || result[0] <= 1.5
query: |
(sum(rate(http_requests_total{service="checkout-api-canary",code=~"5.."}[2m]))
/ sum(rate(http_requests_total{service="checkout-api-canary"}[2m])))
/
(sum(rate(http_requests_total{service="checkout-api-stable",code=~"5.."}[2m]))
/ sum(rate(http_requests_total{service="checkout-api-stable"}[2m])))
This fails when the canary's error ratio is more than 1.5x the stable version's. A service that always runs at 0.4% errors won't false-alarm, but a canary that jumps to 0.9% while stable holds at 0.4% (a 2.25x regression) gets caught even though 0.9% is under any sane absolute threshold. We also added a second metric on p99 latency from the same template, so a deploy that's correct but slow also aborts.
You drive and observe rollouts with the plugin:
$ kubectl argo rollouts get rollout checkout-api --watch
Name: checkout-api
Status: ✖ Degraded
Message: RolloutAborted: metric "error-rate" assessed Failed
...
That Degraded with RolloutAborted is the outcome we wanted from the start: the system caught the bad version at 10% traffic, held it there for about 90 seconds, decided it was worse than stable, and rolled back. Blast radius was one in ten requests for a minute and a half, not everyone for eight hours.
Don't ship canary analysis with absolute thresholds only; they either false-alarm on noisy endpoints or miss real regressions on quiet ones. Compare canary to stable as a ratio, gate on both error rate and p99 latency, and set failureLimit to 2 or 3 so one bad scrape doesn't abort a healthy deploy. The point isn't fancy math, it's that a machine sampling every 30 seconds beats a human staring at Grafana for five minutes and then going to lunch.
