# Argo Rollouts: Canary Deployments That Caught a $40k Bug URL: https://www.devopsness.com/blog/argo-rollouts-canary-deployments-that-caught-a-40k-bug-2026-04-19 Author: Kiril Urbonas Published: 2026-04-19 Updated: 2026-04-19 Categories: DevOps Tags: Kubernetes, CI/CD, Monitoring Summary: A two-line config change to an Argo Rollouts analysis template caught a regression that would have cost ~$40k in API spend before we noticed. Here's the pattern. --- # Argo Rollouts: Canary Deployments That Caught a $40k Bug Last quarter, an automated canary rollback on Argo Rollouts caught a regression that would have cost us ~$40k in extra LLM API spend before any human noticed. The detection took 11 minutes; rollback completed in 90 seconds. The bug fix took an afternoon. Here's exactly how that pattern works in production, and the analysis templates that make it possible. ## The Bug That Almost Shipped A senior engineer refactored our retrieval pipeline. The change passed unit tests, integration tests, and staging soak. The bug: a missing early-return on a cache hit caused every query to **also** call the upstream LLM in a "shadow mode" check that someone added during another refactor weeks earlier. In staging, traffic was low and the extra call was unmeasurable. In production, it doubled our LLM API calls. ## What Argo Rollouts Did Our Rollout config has a canary step that ramps from 5% → 20% → 50% → 100% with **automated analysis** between steps. The analysis template runs three Prometheus queries against the canary pods. ```yaml apiVersion: argoproj.io/v1alpha1 kind: AnalysisTemplate metadata: name: ai-service-canary-analysis spec: metrics: - name: error-rate interval: 1m successCondition: result[0] < 0.005 failureLimit: 3 provider: prometheus: address: http://prometheus.monitoring.svc:9090 query: | sum(rate(http_requests_total{ service="{{args.service}}", version="canary", status=~"5.." }[2m])) / sum(rate(http_requests_total{ service="{{args.service}}", version="canary" }[2m])) - name: latency-p95 interval: 1m successCondition: result[0] < 1.5 failureLimit: 3 provider: prometheus: query: | histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket{ service="{{args.service}}", version="canary" }[2m])) ) # The metric that caught the bug - name: llm-cost-per-request interval: 1m successCondition: result[0] < {{args.cost-baseline}} failureLimit: 2 provider: prometheus: query: | sum(rate(llm_tokens_total{ service="{{args.service}}", version="canary" }[2m])) / sum(rate(http_requests_total{ service="{{args.service}}", version="canary" }[2m])) ``` The third metric, **llm-cost-per-request**, is the one that caught it. ### What happened in real time ``` T+0:00 Canary deploys to 5% of traffic T+2:00 First analysis run - error-rate: 0.001 ✅ - latency-p95: 0.9s ✅ - llm-cost-per-request: 2.4× baseline ❌ T+3:00 Second analysis run (still failing) T+4:00 Third — failureLimit (2) hit on cost metric T+4:30 Argo Rollouts paused the rollout, sent Slack alert T+11:00 On-call decided to roll back (no human review needed for revert) T+12:30 Rollback complete; canary pods drained ``` If we'd ramped to 100% naively, we'd have paid the doubled cost on every request. The analysis caught it at 5%, when extra spend was ~$30/hour. We saved ~$40k worst-case. ## The Rollout Spec ```yaml apiVersion: argoproj.io/v1alpha1 kind: Rollout metadata: name: ai-service spec: replicas: 12 strategy: canary: steps: - setWeight: 5 - pause: { duration: 5m } - analysis: templates: - templateName: ai-service-canary-analysis args: - { name: service, value: ai-service } - { name: cost-baseline, value: "1500" } # tokens per request - setWeight: 20 - pause: { duration: 10m } - analysis: templates: - templateName: ai-service-canary-analysis args: - { name: service, value: ai-service } - { name: cost-baseline, value: "1500" } - setWeight: 50 - pause: { duration: 15m } - analysis: templates: - templateName: ai-service-canary-analysis args: - { name: service, value: ai-service } - { name: cost-baseline, value: "1500" } - setWeight: 100 ``` Three analysis gates. Each has to pass. If any fails, Rollouts auto-aborts and reverts. ## Patterns That Took Us a While to Learn ### Pattern 1: Compare Canary to Baseline, Not Absolute Our first analysis used absolute thresholds: "error rate < 0.005". This bit us when prod was naturally noisy: a small per-deploy bump in baseline error pushed us over the line constantly. The fix: compare canary against the **stable** version's contemporaneous metric. ```yaml successCondition: result[0] < (result[1] * 1.10) # canary < 110% of stable provider: prometheus: query: | [ sum(rate(http_requests_total{service="my-svc", version="canary", status=~"5.."}[2m])) / sum(rate(http_requests_total{service="my-svc", version="canary"}[2m])), sum(rate(http_requests_total{service="my-svc", version="stable", status=~"5.."}[2m])) / sum(rate(http_requests_total{service="my-svc", version="stable"}[2m])) ] ``` Canary needs to be no more than 10% worse than stable on the same metric, in real time. This eliminates false positives from baseline drift. ### Pattern 2: Multiple Failure Counts, Not Single `failureLimit: 1` causes flaky rollbacks. `failureLimit: 5` is too forgiving. We landed on **3 consecutive failures within a 15-min analysis window**. The analysis template runs every minute; 3 in a row is real signal. ### Pattern 3: Cost Metrics Are Underrated Most canary analyses look at error rate and latency. Both are necessary; both miss expensive bugs that don't error or slow down. We now include at least one **cost-shaped** metric in every analysis template: - LLM token cost per request (for AI services) - DB query count per request (for backend services that use Postgres) - Cache hit rate (regression = expensive cold path traffic) - Outbound API calls per request (regression = expensive third-party traffic) These are the metrics that catch "the code is working but spending way more than it should" bugs. The $40k save came from this category. ### Pattern 4: Don't Forget Saturation We hit a near-miss where canary CPU usage was 30% higher per request than stable, but error and latency stayed within spec because we had headroom. The rollout completed; CPU went red two days later under peak load. ```yaml - name: cpu-per-request interval: 1m successCondition: result[0] < (result[1] * 1.15) provider: prometheus: query: | [ rate(container_cpu_usage_seconds_total{pod=~".*canary.*"}[2m]) / rate(http_requests_total{version="canary"}[2m]), rate(container_cpu_usage_seconds_total{pod=~".*stable.*"}[2m]) / rate(http_requests_total{version="stable"}[2m]) ] ``` Saturation regressions don't fire alerts now; they fire rollbacks. ## The Other Win: Faster Reviews Before canary analysis, every PR with infrastructure changes got "extra eyes" review. After: we know that if the change passes CI *and* the canary analysis, it's at least no worse on our metrics than what's running. PR review time dropped 22% on average for normal feature work because the safety net moved from "human eyes" to "automated comparison against running production." ## What We Run On Every Service Below is our standard analysis template. We've copy-pasted some variant of this onto every Rollout in production: ```yaml metrics: - { name: error-rate-vs-stable, ... } # canary < 110% stable error rate - { name: latency-p95-vs-stable, ... } # canary < 110% stable p95 - { name: latency-p99-vs-stable, ... } # canary < 120% stable p99 - { name: cpu-per-req-vs-stable, ... } # canary < 115% stable CPU/req - { name: mem-per-req-vs-stable, ... } # canary < 115% stable mem/req - { name: cost-per-req-vs-stable, ... } # service-specific cost metric ``` Six metrics, all "canary vs stable" ratios. The thresholds are tuned per service but the *shape* is the same everywhere. ## What's Still Hard - **Choosing the metric for "cost".** It varies per service. We have a runbook entry for "what's the cost-shaped metric for service X" — every new service needs to answer this. - **Initial calibration.** A new service has no stable baseline; the first few canaries gate on absolute thresholds, then switch to ratios after 7 days of healthy data. - **Long-tail bugs.** A bug that only fires on 0.1% of traffic at peak hours might not show in 5% canary traffic. We mitigate by holding at 50% for 30 minutes during peak windows. ## Best Practices We'd Recommend 1. **Use `AnalysisTemplate`s, not inline analysis.** Reusable, versioned, easier to PR-review. 2. **Compare canary to stable, not to fixed thresholds.** Eliminates baseline-drift false positives. 3. **Always include a cost-shaped metric.** Errors and latency miss the expensive bugs. 4. **Hold at 50% during a peak window.** Catches load-dependent regressions. 5. **Keep `failureLimit` ≥ 2.** One blip shouldn't roll you back; sustained signal should. 6. **Send analysis failures to Slack.** Even if Argo auto-rolls back, humans should know. 7. **Don't ramp 0% → 100% in two steps.** 5/20/50/100 catches issues much earlier than 50/100. ## Numbers Worth Sharing Across our production rollouts in the last 90 days: | Metric | Value | |---|---:| | Rollouts deployed | 412 | | Analysis-triggered auto-rollbacks | 9 | | Of those, true regressions (not flakes) | 7 | | Average time to rollback | 6m 40s | | Estimated incidents avoided | 7 (incl. the $40k one) | A 1.7% rollback rate on real production deploys, with most of those being legitimate regressions caught early. That's the pattern paying for itself. ## When This Is Overkill - **Internal tools with < 10 users.** Canary the manual way; the automation overhead isn't worth it. - **Rare deploys.** If you ship to a service quarterly, the analysis template will rot. - **No good metrics.** Without observability, canary analysis is theater. Get metrics first. But for any production service with real traffic and real money on the line, the pattern is among the highest-leverage investments we've made in deployment safety.