Argo Rollouts: Canary Analysis That Auto-Aborts on Bad Metrics

Argo Rollouts: Canary Analysis That Auto-Aborts on Bad Metrics#

The worst deploy I remember shipped a serialization bug that only fired on 3% of requests, the ones with a rare payload shape. It passed CI, passed staging, and the canary "looked healthy" to the human watching the dashboard for five minutes before promoting. The errors compounded overnight. We caught it at 2am from a customer email, not our own monitoring. After that we wired Argo Rollouts to judge the canary itself, against Prometheus, and abort without asking a human.

The Rollout replaces your Deployment#

Argo Rollouts swaps kind: Deployment for kind: Rollout and adds a strategy. The canary strategy steps traffic up in stages, and between steps it can run an AnalysisTemplate that queries your metrics and votes on whether to continue.

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: checkout-api
spec:
  replicas: 10
  strategy:
    canary:
      canaryService: checkout-api-canary
      stableService: checkout-api-stable
      trafficRouting:
        nginx:
          stableIngress: checkout-api
      steps:
        - setWeight: 10
        - pause: {duration: 2m}
        - analysis:
            templates:
              - templateName: error-rate
        - setWeight: 50
        - pause: {duration: 5m}
        - analysis:
            templates:
              - templateName: error-rate
        - setWeight: 100

At 10% weight it pauses two minutes, then runs the analysis. If the analysis fails, the whole rollout aborts and traffic snaps back to the stable version. No promotion, no page.

The analysis is a Prometheus query with a threshold#

The AnalysisTemplate is where the judgment lives. This one measures the canary's 5xx rate and fails if it climbs above 1%:

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: error-rate
spec:
  metrics:
    - name: error-rate
      interval: 30s
      count: 4
      successCondition: result < 0.01
      failureLimit: 2
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            sum(rate(http_requests_total{
              service="checkout-api-canary", code=~"5.."}[1m]))
            /
            sum(rate(http_requests_total{
              service="checkout-api-canary"}[1m]))

The mechanics that matter: interval: 30s and count: 4 mean it samples four times over two minutes. successCondition: result < 0.01 is the pass rule. failureLimit: 2 means two bad samples abort the analysis (and the rollout), so a single blip doesn't nuke a good deploy but a sustained problem does. That failureLimit is the single knob people get wrong. Set it to 0 and one scrape gap aborts everything; set it too high and a real regression rides through.

Measure the canary against the baseline, not an absolute#

An absolute "5xx below 1%" threshold breaks the moment your service has a normally-noisy endpoint. Better is to compare canary against stable, so you catch a relative regression even if the absolute number looks fine. Pass both queries and fail when the canary is meaningfully worse:

successCondition: len(result) == 0 || result[0] <= 1.5
query: |
  (sum(rate(http_requests_total{service="checkout-api-canary",code=~"5.."}[2m]))
   / sum(rate(http_requests_total{service="checkout-api-canary"}[2m])))
  /
  (sum(rate(http_requests_total{service="checkout-api-stable",code=~"5.."}[2m]))
   / sum(rate(http_requests_total{service="checkout-api-stable"}[2m])))

This fails when the canary's error ratio is more than 1.5x the stable version's. A service that always runs at 0.4% errors won't false-alarm, but a canary that jumps to 0.9% while stable holds at 0.4% (a 2.25x regression) gets caught even though 0.9% is under any sane absolute threshold. We also added a second metric on p99 latency from the same template, so a deploy that's correct but slow also aborts.

Watch it actually work#

You drive and observe rollouts with the plugin:

$ kubectl argo rollouts get rollout checkout-api --watch
Name:            checkout-api
Status:          ✖ Degraded
Message:         RolloutAborted: metric "error-rate" assessed Failed
...

That Degraded with RolloutAborted is the outcome we wanted from the start: the system caught the bad version at 10% traffic, held it there for about 90 seconds, decided it was worse than stable, and rolled back. Blast radius was one in ten requests for a minute and a half, not everyone for eight hours.

The call we'd make#

Don't ship canary analysis with absolute thresholds only; they either false-alarm on noisy endpoints or miss real regressions on quiet ones. Compare canary to stable as a ratio, gate on both error rate and p99 latency, and set failureLimit to 2 or 3 so one bad scrape doesn't abort a healthy deploy. The point isn't fancy math, it's that a machine sampling every 30 seconds beats a human staring at Grafana for five minutes and then going to lunch.