Our overlay tree grew to seven environments and started copy-pasting the same patch into each. Here's the component-based layout that stopped the drift.
Kustomize Overlays That Scale Across Environments
The overlay tree looked fine with two environments. By the time we hit seven (dev, three staging tiers, and three regional prod clusters) it had become a game of find-the-difference. Someone bumped the readiness probe timeout in prod-eu but forgot prod-us, and a slow-starting pod got killed in a loop during a Friday deploy. The base plus overlay model is right, but the naive version, one flat overlay per environment copying the same patches, drifts the second you have more than a handful of environments.
The base should be a deployable-but-generic version of the app. No environment specifics, no secrets, sensible defaults that would technically run anywhere.
# base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deployment.yaml
- service.yaml
- hpa.yaml
commonLabels:
app.kubernetes.io/name: payments
The mistake teams make is stuffing production values into the base "because that's what we run most." Then every non-prod overlay is a pile of patches undoing prod. Keep the base neutral and let each overlay add, not subtract.
Kustomize components are the feature that makes many environments manageable. A component is a reusable, mixin-style patch you can pull into any overlay. Instead of copying the "high availability" settings into all three prod overlays, you write it once as a component and reference it.
# components/high-availability/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component
patches:
- target:
kind: Deployment
name: payments
patch: |-
- op: replace
path: /spec/replicas
value: 6
- op: add
path: /spec/template/spec/topologySpreadConstraints
value:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app.kubernetes.io/name: payments
Now every production overlay just lists it:
# overlays/prod-eu/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
components:
- ../../components/high-availability
- ../../components/otel-sidecar
patches:
- path: region-patch.yaml
configMapGenerator:
- name: payments-config
literals:
- REGION=eu-west-1
- DB_HOST=payments.eu-west-1.rds.internal
The readiness probe timeout that bit us now lives in one component. Change it once and all three prod clusters move together. That single restructuring removed the entire class of "changed one region, forgot the others" bugs.
The rule we settled on: if a value differs per environment, it's a configMapGenerator literal or a small patch in the overlay. If a behavior is shared by a group of environments (all prod, all EU, all canary), it's a component. Region name is overlay-local. HA topology is a component. Don't put a one-off in a component and don't duplicate a shared behavior across overlays.
Overlays lie if you never render them. We run a CI step that builds every overlay and fails on unexpected drift, so a broken patch is caught before GitOps applies it.
for env in overlays/*/; do
echo "== building $env"
kustomize build "$env" > "/tmp/$(basename $env).yaml" || exit 1
done
# fail if a prod overlay somehow has fewer than 6 replicas
grep -A2 "replicas:" /tmp/prod-*.yaml | grep -qE "replicas: [6-9]" \
|| { echo "prod replica count regressed"; exit 1; }
Argo CD then syncs from the rendered manifests. Because Argo shows the live-vs-desired diff, and because CI already validated the build, a merge that would change prod is visible in the PR, not discovered at 2am.
Pin your Kustomize version in CI and in Argo CD. Components graduated across API versions and patch merge behavior shifted subtly between releases. We got bitten when a developer's local kubectl kustomize (bundling an older Kustomize) produced different output than Argo's newer one. Same files, different manifests. Pin both to the same version and the "works on my machine" reports stop.
If you have one or two environments, plain overlays are fine and components are premature. Past three or four, or the moment you catch yourself copy-pasting the same patch between overlays, move the shared behavior into components immediately. Keep the base neutral, put per-env values in the overlay as generators, and render every overlay in CI so drift fails loudly. Our seven-environment tree went from a drift minefield to something a new engineer could reason about in an afternoon, and the Friday probe-timeout incident never came back.
