Kustomize Overlays That Scale Across Environments

Kustomize Overlays That Scale Across Environments#

The overlay tree looked fine with two environments. By the time we hit seven (dev, three staging tiers, and three regional prod clusters) it had become a game of find-the-difference. Someone bumped the readiness probe timeout in prod-eu but forgot prod-us, and a slow-starting pod got killed in a loop during a Friday deploy. The base plus overlay model is right, but the naive version, one flat overlay per environment copying the same patches, drifts the second you have more than a handful of environments.

The base stays boring#

The base should be a deployable-but-generic version of the app. No environment specifics, no secrets, sensible defaults that would technically run anywhere.

# base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml
  - service.yaml
  - hpa.yaml
commonLabels:
  app.kubernetes.io/name: payments

The mistake teams make is stuffing production values into the base "because that's what we run most." Then every non-prod overlay is a pile of patches undoing prod. Keep the base neutral and let each overlay add, not subtract.

Components are the piece people miss#

Kustomize components are the feature that makes many environments manageable. A component is a reusable, mixin-style patch you can pull into any overlay. Instead of copying the "high availability" settings into all three prod overlays, you write it once as a component and reference it.

# components/high-availability/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component
patches:
  - target:
      kind: Deployment
      name: payments
    patch: |-
      - op: replace
        path: /spec/replicas
        value: 6
      - op: add
        path: /spec/template/spec/topologySpreadConstraints
        value:
          - maxSkew: 1
            topologyKey: topology.kubernetes.io/zone
            whenUnsatisfiable: DoNotSchedule
            labelSelector:
              matchLabels:
                app.kubernetes.io/name: payments

Now every production overlay just lists it:

# overlays/prod-eu/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../../base
components:
  - ../../components/high-availability
  - ../../components/otel-sidecar
patches:
  - path: region-patch.yaml
configMapGenerator:
  - name: payments-config
    literals:
      - REGION=eu-west-1
      - DB_HOST=payments.eu-west-1.rds.internal

The readiness probe timeout that bit us now lives in one component. Change it once and all three prod clusters move together. That single restructuring removed the entire class of "changed one region, forgot the others" bugs.

What goes in the overlay vs the component#

The rule we settled on: if a value differs per environment, it's a configMapGenerator literal or a small patch in the overlay. If a behavior is shared by a group of environments (all prod, all EU, all canary), it's a component. Region name is overlay-local. HA topology is a component. Don't put a one-off in a component and don't duplicate a shared behavior across overlays.

Keep it honest with a diff in CI#

Overlays lie if you never render them. We run a CI step that builds every overlay and fails on unexpected drift, so a broken patch is caught before GitOps applies it.

for env in overlays/*/; do
  echo "== building $env"
  kustomize build "$env" > "/tmp/$(basename $env).yaml" || exit 1
done
# fail if a prod overlay somehow has fewer than 6 replicas
grep -A2 "replicas:" /tmp/prod-*.yaml | grep -qE "replicas: [6-9]" \
  || { echo "prod replica count regressed"; exit 1; }

Argo CD then syncs from the rendered manifests. Because Argo shows the live-vs-desired diff, and because CI already validated the build, a merge that would change prod is visible in the PR, not discovered at 2am.

The version thing#

Pin your Kustomize version in CI and in Argo CD. Components graduated across API versions and patch merge behavior shifted subtly between releases. We got bitten when a developer's local kubectl kustomize (bundling an older Kustomize) produced different output than Argo's newer one. Same files, different manifests. Pin both to the same version and the "works on my machine" reports stop.

The call we'd make#

If you have one or two environments, plain overlays are fine and components are premature. Past three or four, or the moment you catch yourself copy-pasting the same patch between overlays, move the shared behavior into components immediately. Keep the base neutral, put per-env values in the overlay as generators, and render every overlay in CI so drift fails loudly. Our seven-environment tree went from a drift minefield to something a new engineer could reason about in an afternoon, and the Friday probe-timeout incident never came back.