Argo CD ApplicationSets: Managing Many Clusters Without Copy-Paste

Argo CD ApplicationSets: Managing Many Clusters Without Copy-Paste#

We had a platform-agent DaemonSet that needed to run on every cluster we operated: 23 of them across three regions, plus a rotating set of ephemeral preview clusters. Each cluster had its own Argo CD Application manifest checked into Git. When we bumped the agent image, someone had to touch 23 files. Inevitably two got missed, and we'd find a stale agent version three weeks later during an incident review. The fix was ApplicationSets.

One generator, N Applications#

An ApplicationSet is a controller that templates out Application resources from a generator. The cluster generator is the obvious starting point: it produces one Application per cluster registered in Argo CD (each cluster is a Secret labeled argocd.argoproj.io/secret-type: cluster).

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: platform-agent
  namespace: argocd
spec:
  goTemplate: true
  generators:
    - clusters:
        selector:
          matchLabels:
            agent: enabled
  template:
    metadata:
      name: 'agent-{{.name}}'
    spec:
      project: platform
      source:
        repoURL: https://github.com/acme/platform-charts
        targetRevision: main
        path: charts/agent
        helm:
          valueFiles:
            - 'values-{{.metadata.labels.region}}.yaml'
      destination:
        server: '{{.server}}'
        namespace: platform-system
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

Now the set of managed clusters is driven by a label on the cluster secret. Add agent: enabled to a new cluster's secret and Argo generates the Application within seconds. Remove the label and, because we set prune: true at the ApplicationSet level too, the Application is deleted and its workloads cleaned up.

Region-specific values without forking the chart#

Notice values-{{.metadata.labels.region}}.yaml. Each cluster secret carries a region label (us-east, eu-west, ap-south), and the template picks the matching values file. That's how we handle the real-world truth that clusters are not identical: eu-west needs a lower memory limit, ap-south points at a regional mirror registry. One chart, per-region overrides, zero forked manifests.

Matrix generator for the harder cases#

The cluster generator alone stops being enough when you deploy several apps to several clusters and the combinations matter. The matrix generator takes the Cartesian product of two generators. We use it to deploy a list of infra addons (defined in a Git directory) across the same labeled clusters:

generators:
  - matrix:
      generators:
        - git:
            repoURL: https://github.com/acme/platform-charts
            revision: main
            directories:
              - path: addons/*
        - clusters:
            selector:
              matchLabels:
                agent: enabled

The git generator yields one entry per directory under addons/ (say cert-manager, node-exporter, fluent-bit), the cluster generator yields the clusters, and the matrix produces one Application per pair. Adding a new addon is a new directory; adding a cluster is a label. Neither requires editing the ApplicationSet.

The guardrails you actually need#

Two settings saved us from a bad afternoon. The first is preserveResourcesOnDeletion. Without it, deleting the ApplicationSet cascades to every generated Application and prunes real workloads. If you're refactoring the ApplicationSet itself, that's terrifying. Set it while you experiment:

spec:
  syncPolicy:
    preserveResourcesOnDeletion: true

The second is the ApplicationSet progressive sync (strategy: RollingSync), which rolls changes out cluster group by cluster group instead of hitting all 23 at once. We label clusters env: canary vs env: prod and sync canary first, wait for health, then prod. A bad image bump now blast-radiuses to one cluster, not the fleet.

strategy:
  type: RollingSync
  rollingSync:
    steps:
      - matchExpressions:
          - key: env
            operator: In
            values: [canary]
      - matchExpressions:
          - key: env
            operator: In
            values: [prod]

What still bites#

Templating errors are opaque. A typo in a Go template field renders an Application with an empty path, and Argo happily tries to sync nothing, or worse, the repo root. Run argocd appset get platform-agent and check the generated Applications before trusting a sync. Also, the cluster generator only sees clusters Argo already knows about, so cluster registration has to happen first, usually via your Terraform or a cluster-bootstrap job.

The call we'd make#

If you manage more than a handful of clusters and find yourself copying Application YAML, switch to ApplicationSets now, starting with the cluster generator plus per-region values files. Add the matrix generator only when you have a genuine app-times-cluster grid. And turn on RollingSync before you point it at production, because "sync all 23 at once" is a great way to learn what a fleet-wide outage feels like.