External Secrets Operator: One Secrets Workflow Across Clouds

External Secrets Operator: One Secrets Workflow Across Clouds#

We ran three clusters across two clouds plus on-prem, and each had its own secrets story. AWS workloads pulled from Secrets Manager with an init container. GCP used a sidecar hitting Secret Manager. The on-prem cluster read from HashiCorp Vault with yet another tool. Onboarding an engineer meant teaching three workflows, and every audit turned up a secret that had been copy-pasted into a plain ConfigMap because someone didn't want to learn the fourth way to do it.

External Secrets Operator (ESO) collapsed all of that into one Kubernetes-native pattern: a controller that syncs secrets from whatever backend into native Kubernetes Secrets, using the same two CRDs everywhere.

The two objects you actually work with#

ESO gives you a SecretStore (where secrets come from and how to auth) and an ExternalSecret (which secrets to pull and what to name them). The developer only ever writes ExternalSecret. The platform team owns SecretStore. That split is the reason it scales, app teams never touch credentials for the backend.

Here's an AWS store using IRSA so there are no static keys anywhere:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secrets
  namespace: payments
spec:
  provider:
    aws:
      service: SecretsManager
      region: eu-west-1
      auth:
        jwt:
          serviceAccountRef:
            name: payments-eso   # bound to an IAM role via IRSA

The GCP store and the Vault store are structurally identical, just a different provider block. That sameness is the whole value. An app team's ExternalSecret looks the same no matter which cloud the cluster runs in:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
  namespace: payments
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets
    kind: SecretStore
  target:
    name: db-credentials      # the k8s Secret ESO creates
  data:
    - secretKey: password
      remoteRef:
        key: prod/payments/db
        property: password

That remoteRef.key is the path in the backend; on GCP it'd be a resource name, in Vault a mount path. Same CRD, backend-specific reference. Developers learn one manifest shape.

Rotation without redeploys#

The feature that sold the security team: refreshInterval. ESO re-reads the backend on that cadence and updates the Kubernetes Secret in place. Pair it with a reloader so pods pick up the change without a manual rollout.

metadata:
  annotations:
    reloader.stakater.com/auto: "true"

When we rotate the database password in Secrets Manager, ESO syncs it within the hour and the reloader restarts the affected deployments. What used to be a coordinated change across three clusters is now a single write to the backend. We rotate the payments DB credential every 30 days now, which we simply never did before because the manual dance was too painful.

The gotchas that cost us time#

A synced Kubernetes Secret is still a base64 Secret sitting in etcd. ESO moves secrets around, it does not make them encrypted at rest by itself. You still need etcd encryption enabled and RBAC locked down. We caught a namespace where any pod could read the Secret because a broad Role had crept in. ESO is not a substitute for least privilege.

Second, refresh interval versus API cost. We set refreshInterval: 1m on a chatty cluster and got a bill surprise, plus AWS throttling, because ESO was calling the Secrets Manager API constantly across hundreds of ExternalSecrets. One minute times hundreds of secrets is a lot of GetSecretValue calls. We moved to 1h for stable secrets and reserved short intervals for the handful that actually rotate fast. Right-size per secret.

Third, a failed sync is quiet unless you watch for it. If the backend is unreachable, ESO keeps serving the last-good Secret, which is good for uptime and bad because you won't notice the sync broke. Alert on the condition:

external_secret_status_condition{condition="Ready",status="false"} == 1

We wire that into Prometheus so a store losing auth pages us before the stale secret expires.

Where it didn't fit#

We tried ESO for dynamic database credentials from Vault, the kind that are minted per-session and expire in 15 minutes. That's a poor fit; ESO's sync model wants relatively stable secrets, and 15-minute leases meant constant churn and race conditions on pod start. For that one case we kept the Vault Agent sidecar. Use the right tool, ESO is for syncing stored secrets, not for short-lived dynamic ones.

The call we'd make#

If you have more than one secrets backend, or more than one cluster, adopt ESO. The single-workflow payoff is worth it just for onboarding and audits, and rotation stops being a project. Keep the platform team owning SecretStore and let app teams own ExternalSecret. And keep a sidecar around for genuinely dynamic, short-lived credentials, that's the one case ESO shouldn't own.