# mTLS for Service-to-Service Auth — Beyond API Keys URL: https://www.devopsness.com/blog/mtls-service-to-service-auth-beyond-api-keys Author: Kiril Urbonas Published: 2026-07-11 Updated: 2026-07-11 Categories: Cloud Tags: Kubernetes, Security, Networking, Identity, Zero Trust Summary: A shared API key between two internal services proves nothing about who is calling. mTLS makes every service present a cryptographic identity instead. --- # mTLS for Service-to-Service Auth — Beyond API Keys We found a Kubernetes secret named `internal-api-key` mounted into eleven services. Nobody knew when it was created. Nobody knew which of the eleven actually needed it. Rotating it meant a coordinated redeploy of all eleven, so it hadn't been rotated in about two years. That secret was the entire authentication story for our east-west traffic, and it authenticated nothing about the caller. That's the core problem with shared API keys and bearer tokens between services. They're bearer credentials: whoever holds the string is trusted, full stop. They don't encode who is calling, only that the caller once got a copy of the string. They tend to be long-lived because rotating them is painful. And once one leaks — a log line, a crash dump, a compromised sidecar — every service that trusts it is exposed, and you have no way to tell legitimate use from abuse because every request looks identical. ## Certificates as identity, not just encryption Plain TLS gives the client confidence about the server. Mutual TLS makes the client prove itself too. During the handshake, the calling service presents its own X.509 certificate, and the receiving service validates it against a trusted CA before a single byte of application data moves. The cert's subject *is* the caller's identity, signed by an authority both sides trust. There is no shared string to leak, and the private key never leaves the workload that owns it. The identity naming that's winning here is SPIFFE. A SPIFFE ID is a URI like `spiffe://prod.example.com/ns/payments/sa/checkout`, carried in the certificate's SAN. The certificate that encodes it is an SVID (SPIFFE Verifiable Identity Document). Now authorization decisions read like sentences: "the checkout service account in the payments namespace may call the ledger." That's a real subject, not a secret. ## Where the certs come from You do not want to hand-roll a CA and copy PEM files around. Three common sources, roughly in order of how much you're already running: - **cert-manager** issues and renews `Certificate` resources in Kubernetes from an issuer you control. Good when you want explicit certs for specific workloads. - **SPIFFE/SPIRE** runs a control plane that attests workloads (by node + process characteristics) and hands each one a short-lived SVID over a local socket. It's the vendor-neutral option and integrates well beyond Kubernetes. We wrote up a full setup in [zero-trust service auth with SPIFFE and SPIRE](/blog/zero-trust-service-auth-spiffe-spire). - **A service mesh** — Istio or Linkerd — bootstraps a CA and injects sidecars that do the handshake for you. Auto-mTLS means your application code keeps talking plain HTTP to localhost and the proxy wraps it. This is the lowest-friction path if you're already on a mesh. ## Turning it on with Istio Two objects. First, require mTLS so plaintext is rejected inside the namespace: ```yaml apiVersion: security.istio.io/v1 kind: PeerAuthentication metadata: name: default namespace: payments spec: mtls: mode: STRICT ``` `STRICT` is the setting that matters. `PERMISSIVE` accepts both mTLS and plaintext, which is fine for a migration window and a trap if you forget to flip it. Then authorize by identity, not IP: ```yaml apiVersion: security.istio.io/v1 kind: AuthorizationPolicy metadata: name: ledger-allow-checkout namespace: payments spec: selector: matchLabels: app: ledger action: ALLOW rules: - from: - source: principals: - "prod.example.com/ns/payments/sa/checkout" to: - operation: methods: ["POST"] paths: ["/v1/entries"] ``` The `principals` field is the SPIFFE identity from the peer cert. Nothing about a source IP, a shared token, or a network location. If you'd rather manage certs directly, cert-manager's equivalent is a `Certificate` with a short lifetime: ```yaml apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: checkout-mtls namespace: payments spec: secretName: checkout-mtls-tls duration: 24h renewBefore: 8h issuerRef: name: internal-ca kind: ClusterIssuer uris: - "spiffe://prod.example.com/ns/payments/sa/checkout" privateKey: rotationPolicy: Always ``` ## Short TTLs and the rotation trap `duration: 24h` is not cosmetic. A leaked long-lived cert is a standing liability; a leaked SVID with an hour left on it is a much smaller window, and it revokes itself by expiring. Short TTLs also mean your renewal path gets exercised constantly, so it's never a rusty code path you discover during an incident. SPIRE hands out SVIDs measured in minutes for exactly this reason. `rotationPolicy: Always` above forces a fresh key on every renewal instead of reusing one. The gotchas are real, though. Rotation without downtime needs the workload to reload the new cert and key without dropping in-flight connections — meshes and SPIRE's socket handle this, but a naive app that reads a PEM once at boot will happily serve an expired cert until it restarts. Trust bundle distribution is the other one: every service needs the current CA bundle to validate peers, and when you rotate the root or add an intermediate, that bundle has to reach everyone *before* the new signer goes live, or handshakes fail cluster-wide. And be clear about what mTLS does not do. It proves *who* is calling. It does not decide *what* they may do. mTLS says "this is genuinely the checkout service"; your `AuthorizationPolicy` (or app-layer authz) says "checkout may POST to `/v1/entries` and nothing else." Skip that second layer and a compromised-but-legitimate service can reach everything its transport identity can connect to. ## The call we'd make If you're on a mesh already, turn on `STRICT` mTLS per namespace and start deleting shared keys the week you do it — the payoff is immediate and the code change is zero. If you're not on a mesh, don't adopt one just for this; run SPIRE or cert-manager and get SVIDs into your workloads. Either way, treat mTLS as the identity layer and keep an explicit authorization policy on top of it. Same principle as [keyless authentication](/blog/keyless-cloud-authentication-ending-static-credentials) for cloud APIs: stop shipping secrets that prove nothing, and let a short-lived cryptographic identity do the proving instead.