mTLS for Service-to-Service Auth — Beyond API Keys

We found a Kubernetes secret named internal-api-key mounted into eleven services. Nobody knew when it was created. Nobody knew which of the eleven actually needed it. Rotating it meant a coordinated redeploy of all eleven, so it hadn't been rotated in about two years. That secret was the entire authentication story for our east-west traffic, and it authenticated nothing about the caller.

That's the core problem with shared API keys and bearer tokens between services. They're bearer credentials: whoever holds the string is trusted, full stop. They don't encode who is calling, only that the caller once got a copy of the string. They tend to be long-lived because rotating them is painful. And once one leaks — a log line, a crash dump, a compromised sidecar — every service that trusts it is exposed, and you have no way to tell legitimate use from abuse because every request looks identical.

Certificates as identity, not just encryption#

Plain TLS gives the client confidence about the server. Mutual TLS makes the client prove itself too. During the handshake, the calling service presents its own X.509 certificate, and the receiving service validates it against a trusted CA before a single byte of application data moves. The cert's subject is the caller's identity, signed by an authority both sides trust. There is no shared string to leak, and the private key never leaves the workload that owns it.

The identity naming that's winning here is SPIFFE. A SPIFFE ID is a URI like spiffe://prod.example.com/ns/payments/sa/checkout, carried in the certificate's SAN. The certificate that encodes it is an SVID (SPIFFE Verifiable Identity Document). Now authorization decisions read like sentences: "the checkout service account in the payments namespace may call the ledger." That's a real subject, not a secret.

Where the certs come from#

You do not want to hand-roll a CA and copy PEM files around. Three common sources, roughly in order of how much you're already running:

• cert-manager issues and renews Certificate resources in Kubernetes from an issuer you control. Good when you want explicit certs for specific workloads.
• SPIFFE/SPIRE runs a control plane that attests workloads (by node + process characteristics) and hands each one a short-lived SVID over a local socket. It's the vendor-neutral option and integrates well beyond Kubernetes. We wrote up a full setup in zero-trust service auth with SPIFFE and SPIRE.
• A service mesh — Istio or Linkerd — bootstraps a CA and injects sidecars that do the handshake for you. Auto-mTLS means your application code keeps talking plain HTTP to localhost and the proxy wraps it. This is the lowest-friction path if you're already on a mesh.

Turning it on with Istio#

Two objects. First, require mTLS so plaintext is rejected inside the namespace:

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: payments
spec:
  mtls:
    mode: STRICT

STRICT is the setting that matters. PERMISSIVE accepts both mTLS and plaintext, which is fine for a migration window and a trap if you forget to flip it. Then authorize by identity, not IP:

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: ledger-allow-checkout
  namespace: payments
spec:
  selector:
    matchLabels:
      app: ledger
  action: ALLOW
  rules:
    - from:
        - source:
            principals:
              - "prod.example.com/ns/payments/sa/checkout"
      to:
        - operation:
            methods: ["POST"]
            paths: ["/v1/entries"]

The principals field is the SPIFFE identity from the peer cert. Nothing about a source IP, a shared token, or a network location. If you'd rather manage certs directly, cert-manager's equivalent is a Certificate with a short lifetime:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: checkout-mtls
  namespace: payments
spec:
  secretName: checkout-mtls-tls
  duration: 24h
  renewBefore: 8h
  issuerRef:
    name: internal-ca
    kind: ClusterIssuer
  uris:
    - "spiffe://prod.example.com/ns/payments/sa/checkout"
  privateKey:
    rotationPolicy: Always

Short TTLs and the rotation trap#

duration: 24h is not cosmetic. A leaked long-lived cert is a standing liability; a leaked SVID with an hour left on it is a much smaller window, and it revokes itself by expiring. Short TTLs also mean your renewal path gets exercised constantly, so it's never a rusty code path you discover during an incident. SPIRE hands out SVIDs measured in minutes for exactly this reason. rotationPolicy: Always above forces a fresh key on every renewal instead of reusing one.

The gotchas are real, though. Rotation without downtime needs the workload to reload the new cert and key without dropping in-flight connections — meshes and SPIRE's socket handle this, but a naive app that reads a PEM once at boot will happily serve an expired cert until it restarts. Trust bundle distribution is the other one: every service needs the current CA bundle to validate peers, and when you rotate the root or add an intermediate, that bundle has to reach everyone before the new signer goes live, or handshakes fail cluster-wide.

And be clear about what mTLS does not do. It proves who is calling. It does not decide what they may do. mTLS says "this is genuinely the checkout service"; your AuthorizationPolicy (or app-layer authz) says "checkout may POST to /v1/entries and nothing else." Skip that second layer and a compromised-but-legitimate service can reach everything its transport identity can connect to.

The call we'd make#

If you're on a mesh already, turn on STRICT mTLS per namespace and start deleting shared keys the week you do it — the payoff is immediate and the code change is zero. If you're not on a mesh, don't adopt one just for this; run SPIRE or cert-manager and get SVIDs into your workloads. Either way, treat mTLS as the identity layer and keep an explicit authorization policy on top of it. Same principle as keyless authentication for cloud APIs: stop shipping secrets that prove nothing, and let a short-lived cryptographic identity do the proving instead.