GitOps with Argo CD: Best Practices for 2025

GitOps with Argo CD: Best Practices for 2025#

We moved our first cluster to Argo CD in 2021 because kubectl apply from a CI job had stopped being honest. Nobody could tell you what was actually running. Two engineers had hotfixed the same Deployment by hand, CI kept overwriting one of them on every merge, and the incident channel had a thread titled "why did the replica count change again." GitOps fixed the honesty problem, though not for free, and most of what we learned since is about the parts the tutorials skip.

GitOps, stripped of the marketing, is one rule: the desired state of your cluster lives in git, and a controller in the cluster makes reality match it. No human runs apply. Argo CD is the controller that watches the repo, diffs it against live objects, and either shows you the gap or closes it. The payoff isn't automation for its own sake — it's that every change is a reviewed commit, every rollback is a git revert, and the answer to "what's running" is a SHA, not a shrug.

Why Argo CD over Flux or a homegrown script? Two reasons that mattered to us. The UI is a real diff viewer your on-call engineer will actually open at 2am, and the Application/ApplicationSet model maps cleanly onto how teams already think about "an app in an environment." Flux is excellent, and if your team lives in the terminal and wants a lighter footprint it's a fair pick, but the visual sync status pulled more people into GitOps than any doc we wrote.

Repo structure: app-of-apps, then ApplicationSets#

The first mistake we made was one giant repo with every manifest flat in a folder. It worked for six services and fell apart at forty.

The pattern that scaled: a root Application pointing at a directory of other Applications. This is "app-of-apps." One Application defines the set, git owns the membership, and adding a service is a PR that adds a file. Here's the shape of a leaf Application:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: payments-prod
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: payments
  source:
    repoURL: https://github.com/acme/deploy.git
    targetRevision: main
    path: apps/payments/overlays/prod
  destination:
    server: https://kubernetes.default.svc
    namespace: payments
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

App-of-apps handles one cluster well. Once you have the same twelve apps across dev, staging, and four regional prod clusters, hand-writing Applications becomes the new flat folder: copy-paste with a typo waiting to bite. ApplicationSets were built for exactly this. One generator, many Applications:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: payments
  namespace: argocd
spec:
  generators:
    - matrix:
        generators:
          - clusters:
              selector:
                matchLabels:
                  env: prod
          - list:
              elements:
                - app: payments
                - app: ledger
  template:
    metadata:
      name: '{{.values.app}}-{{.name}}'
    spec:
      project: payments
      source:
        repoURL: https://github.com/acme/deploy.git
        targetRevision: main
        path: 'apps/{{.values.app}}/overlays/{{.metadata.labels.region}}'
      destination:
        server: '{{.server}}'
        namespace: payments
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

The matrix generator crosses your registered clusters against a list of apps, and Argo CD manages the fan-out. Register a new prod cluster with the right labels and the apps show up. We run one ApplicationSet per team, not per cluster, because ownership follows the team.

Sync policies, and where to slam the brakes#

automated with selfHeal and prune is the config people copy first and regret later. Turn all three on and Argo CD becomes a very fast, very literal robot. That's what you want in dev; in prod it depends on what you're syncing.

Our rule: automated sync everywhere, selfHeal everywhere, and prune on for stateless workloads but gated for anything with a PersistentVolumeClaim or a CRD other controllers depend on. prune deletes objects that left git. If someone refactors a directory and a StatefulSet's name changes, prune will happily delete the old one, PVC and all. We watched it happen in staging and were glad it wasn't Friday.

For the changes that scare you, like a database migration or a CRD version bump, use sync waves and hooks instead of reaching for manual sync. Annotate resources with argocd.argoproj.io/sync-wave so the migration Job runs and completes before the Deployment rolls, and a failing PreSync hook halts the sync with a clear status instead of a half-applied cluster. Manual sync as a safety gate sounds prudent and mostly means someone forgets to click the button while drift piles up.

Config and secrets#

Two ways to template config, and you'll probably use both. Kustomize overlays are our default: just YAML patches over a base, easy to diff, easy for a reviewer to reason about. We keep a base/ and per-environment overlays/, and let the overlay carry only what differs. The fuller argument is in Kustomize overlays that scale. Helm earns its place for third-party charts and genuinely dynamic config; for your own apps, a chart is often more machinery than the problem needs, as Helm chart anti-patterns covers.

Secrets have one rule and it isn't negotiable: never in git, not even encrypted-and-you'll-clean-it-up-later. Plaintext secrets in a repo outlive the repo. The two approaches we run:

• External Secrets Operator when a real secret manager (Vault, AWS Secrets Manager, GCP Secret Manager) is the source of truth. Git holds an ExternalSecret that references a key; the operator materializes the actual Secret. This is our default because rotation happens in one place. The cross-cloud setup is in External Secrets Operator across clouds.
• Sealed Secrets when you want the encrypted value in git and have no external manager to lean on. The controller holds the private key and decrypts in-cluster. Simpler to bootstrap, but rotation is more manual.

Either way, Argo CD only ever sees the reference or the ciphertext. The plaintext exists only inside the cluster.

Progressive delivery#

Argo CD syncs your manifests; it does not make a rollout gradual. A synced Deployment still swaps pods on the standard rolling strategy, and a bad image reaches every user in the time it takes to reschedule. Argo Rollouts fills that gap. It replaces the Deployment with a Rollout object that supports canary and blue-green, pauses at defined weights, and queries a metrics provider to promote or abort automatically.

The pattern that paid for itself: a canary that sends 10 percent of traffic to the new version, runs an AnalysisTemplate against Prometheus for error rate and p99 latency, and aborts on its own if either crosses the line. We've had it block releases at 3am with nobody watching. That's the whole point. The Argo Rollouts progressive delivery walkthrough has the analysis templates we actually run.

Drift, RBAC, and knowing what's happening#

Drift detection is Argo CD's quiet superpower. Every app is Synced, OutOfSync, or Unknown, and OutOfSync means live state diverged from git: someone ran kubectl edit, a mutating webhook added a sidecar. With selfHeal on, Argo CD reverts the drift within its reconciliation window. Even without it, the diff view tells you exactly what changed and who to ask. Drift that used to surface during an incident now surfaces on a dashboard. The same discipline applies to cloud infra as code; we cover the Terraform side in Terraform drift detection in CI.

RBAC and multi-tenancy run through AppProjects. A Project scopes which repos an app can pull from, which clusters and namespaces it can deploy to, and which resource kinds are allowed. We give each team a Project, deny cluster-scoped resources by default, and grant them explicitly. Map your SSO groups to Argo CD roles so a team can sync its own apps and see but not touch everyone else's. Without Projects, anyone with UI access can sync anything anywhere: a compliance finding waiting to be written up.

Observability matters more than a fresh Argo CD user expects. Scrape the argocd-application-controller metrics into Prometheus, alert on apps stuck OutOfSync past a threshold and on sync failures, and pipe sync events to Slack. A sync that fails silently is worse than a failed CI job: CI is red and loud, while a stuck app sits there looking fine in a list.

Anti-patterns we stopped repeating#

• Deploying from the same repo as your app code. Keep a separate deploy repo so the app repo's churn doesn't drag your cluster's history with it.
• Letting CI run kubectl apply "just for this one thing." The one thing becomes the exception that erodes the model. If it's in the cluster, it's in git.
• prune on globally without thinking about stateful resources. Covered above, learned the hard way.
• One enormous Application for a whole cluster. A single failed resource can wedge the sync for everything. Split by team or domain.
• Auto-sync with no progressive delivery on user-facing services. Fast propagation of a good change is great; fast propagation of a bad one is an outage.
• Secrets in git. Still no. Still never.

The call we'd make#

Start with app-of-apps on one cluster, automated sync with self-heal on from day one, and prune scoped to stateless workloads. Move to ApplicationSets the moment you're copy-pasting Applications across environments, not before — the matrix generator is worth learning but it's overhead you don't need at six services. Put secrets in External Secrets Operator, keep a deploy repo separate from app code, and give every team an AppProject with cluster-scoped resources denied. Add Argo Rollouts before you think you need it. The rest is discipline: if it's running in the cluster and it isn't in git, treat that as the bug.