# Argo Rollouts: Canary Analysis That Auto-Aborts on Bad Metrics

URL: https://www.devopsness.com/blog/argo-rollouts-canary-analysis-auto-abort
Author: Kiril Urbonas
Published: 2026-06-30
Updated: 2026-10-31
Categories: DevOps
Tags: Argo CD, Canary Releases, Deployment, Kubernetes, Reliability

Summary: A bad deploy used to mean a pager at 2am and a manual rollback. Now Argo Rollouts watches the error rate and aborts the canary itself before anyone wakes up.

---
# Argo Rollouts: Canary Analysis That Auto-Aborts on Bad Metrics

Argo Rollouts auto-aborts a canary when an `AnalysisTemplate` step fails: put `analysis` steps between `setWeight` steps, query Prometheus for the canary's 5xx ratio, and set `failureLimit` so a sustained regression fails the run. A failed analysis shifts traffic back to stable and marks the Rollout `Degraded`, with no human in the loop.

> [!VERIFIED]
> Argo Rollouts v1.10.1 (released 2026-10-12), against the argo-rollouts.readthedocs.io analysis, canary and Prometheus provider docs. Checked October 2026.

The worst deploy I remember shipped a serialization bug that only fired on 3% of requests, the ones with a rare payload shape. It passed CI, passed staging, and the canary "looked healthy" to the human watching the dashboard for five minutes before promoting. The errors compounded overnight. We caught it at 2am from a customer email, not our own monitoring. After that we wired Argo Rollouts to judge the canary itself, against Prometheus, and abort without asking a human.

## The Rollout replaces your Deployment

Argo Rollouts swaps `kind: Deployment` for `kind: Rollout` and adds a strategy (if you are new to the controller, start with [progressive delivery with Argo Rollouts](/blog/argo-rollouts-progressive-delivery)). The canary strategy steps traffic up in stages, and between steps it can run an `AnalysisTemplate` that queries your metrics and votes on whether to continue.

```yaml
# k8s/checkout-api/rollout.yaml
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: checkout-api
spec:
  replicas: 10
  strategy:
    canary:
      canaryService: checkout-api-canary
      stableService: checkout-api-stable
      trafficRouting:
        nginx:
          stableIngress: checkout-api
      steps:
        - setWeight: 10
        - pause: {duration: 2m}
        - analysis:
            templates:
              - templateName: error-rate
        - setWeight: 50
        - pause: {duration: 5m}
        - analysis:
            templates:
              - templateName: error-rate
        - setWeight: 100
```

At 10% weight it pauses two minutes, then runs the analysis. If the analysis fails, the whole rollout aborts and traffic snaps back to the stable version. No promotion, no page.

## The analysis is a Prometheus query with a threshold

The `AnalysisTemplate` is where the judgment lives. This one measures the canary's 5xx rate and fails if it climbs above 1%:

```yaml
# k8s/checkout-api/analysis-error-rate.yaml
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: error-rate
spec:
  metrics:
    - name: error-rate
      interval: 30s
      count: 4
      successCondition: result[0] < 0.01
      failureLimit: 2
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            sum(rate(http_requests_total{
              service="checkout-api-canary", code=~"5.."}[1m]))
            /
            sum(rate(http_requests_total{
              service="checkout-api-canary"}[1m]))
```

The mechanics that matter: `interval: 30s` and `count: 4` mean it samples four times over two minutes. `successCondition: result[0] < 0.01` is the pass rule; the Prometheus provider returns a vector, so you index element 0. `failureLimit: 2` is the number of failed measurements tolerated, so a third bad sample fails the analysis (and aborts the rollout). A single blip doesn't nuke a good deploy, but a sustained problem does. That `failureLimit` is the single knob people get wrong. Leave it unset and it defaults to 0, so one scrape gap aborts everything; set it too high and a real regression rides through.

## Measure the canary against the baseline, not an absolute

An absolute "5xx below 1%" threshold breaks the moment your service has a normally-noisy endpoint. Better is to compare canary against stable, so you catch a relative regression even if the absolute number looks fine. Pass both queries and fail when the canary is meaningfully worse:

```yaml
# k8s/checkout-api/analysis-error-ratio.yaml (metric excerpt)
successCondition: len(result) == 0 || result[0] <= 1.5
query: |
  (sum(rate(http_requests_total{service="checkout-api-canary",code=~"5.."}[2m]))
   / sum(rate(http_requests_total{service="checkout-api-canary"}[2m])))
  /
  (sum(rate(http_requests_total{service="checkout-api-stable",code=~"5.."}[2m]))
   / sum(rate(http_requests_total{service="checkout-api-stable"}[2m])))
```

This fails when the canary's error ratio is more than 1.5x the stable version's. A service that always runs at 0.4% errors won't false-alarm, but a canary that jumps to 0.9% while stable holds at 0.4% (a 2.25x regression) gets caught even though 0.9% is under any sane absolute threshold. We also added a second metric on p99 latency from the same template, so a deploy that's correct but slow also aborts.

## Watch it actually work

You drive and observe rollouts with the plugin:

```bash
$ kubectl argo rollouts get rollout checkout-api --watch
Name:            checkout-api
Status:          ✖ Degraded
Message:         RolloutAborted: metric "error-rate" assessed Failed
...
```

That `Degraded` with `RolloutAborted` is the outcome we wanted from the start: the system caught the bad version at 10% traffic, held it there for about 90 seconds, decided it was worse than stable, and rolled back. Blast radius was one in ten requests for a minute and a half, not everyone for eight hours.

## Where this breaks

The ratio query has a sharp edge: when stable serves zero 5xx responses, the denominator is zero and Prometheus returns `NaN` or `+Inf`, which fails `result[0] <= 1.5` on a perfectly healthy canary. Guard it with a floor such as `clamp_min(..., 0.001)` on the stable ratio, or pair the ratio with an absolute ceiling. Low-traffic services are the other trap: ten requests a minute at 10% weight is one canary request, and no threshold means anything at that volume. Lengthen the pause or skip analysis there. Finally, an `Inconclusive` result pauses the rollout instead of aborting it, so someone still has to look. If your rollouts are managed by Argo CD, the abort shows up as a degraded app, which fits the [GitOps model](/blog/gitops-explained-introduction) of Git as desired state and the cluster reporting reality.

## Questions people ask

### How does Argo Rollouts decide to abort a canary?

Each metric in an `AnalysisTemplate` takes `count` measurements every `interval`. A measurement fails when `successCondition` is false or `failureCondition` is true. When failures exceed `failureLimit` (default 0), the analysis run fails. For an inline analysis step, that aborts the rollout: canary weight goes back to zero, traffic returns to stable, and the Rollout reports `Degraded`.

### What should failureLimit be for canary analysis?

We use 1 or 2 with `interval: 30s` and `count: 4`. The default of 0 means a single missed scrape or a one-off spike kills a good deploy, which trains people to ignore aborts. Set it close to `count` and a regression has to fail nearly every sample before anything happens. Setting `failureLimit: -1` disables the limit entirely, which defeats the purpose.

### Can Argo Rollouts compare canary metrics to the stable version?

Yes, by writing a PromQL query that divides the canary's error ratio by the stable's, and asserting the result stays under a factor such as 1.5. This catches relative regressions that absolute thresholds miss on noisy services. Protect the denominator against zero errors on stable, or the query returns `NaN` and fails healthy canaries.

## The call we'd make

Don't ship canary analysis with absolute thresholds only; they either false-alarm on noisy endpoints or miss real regressions on quiet ones. Compare canary to stable as a ratio, gate on both error rate and p99 latency, and set `failureLimit` to 1 or 2 (against a `count` of 4) so one bad scrape doesn't abort a healthy deploy. Fancy math is not the point. A machine sampling every 30 seconds beats a human staring at Grafana for five minutes and then going to lunch.

## Sources

- [Argo Rollouts docs: Analysis and progressive delivery](https://argo-rollouts.readthedocs.io/en/stable/features/analysis/)
- [Argo Rollouts docs: Canary deployment strategy](https://argo-rollouts.readthedocs.io/en/stable/features/canary/)
- [Argo Rollouts docs: Prometheus metric provider](https://argo-rollouts.readthedocs.io/en/stable/analysis/prometheus/)
- [Argo Rollouts releases on GitHub](https://github.com/argoproj/argo-rollouts/releases)

