Zero-Trust Service-to-Service Auth with SPIFFE and SPIRE

We found a bearer token for our payments service sitting in a Grafana Loki index, in plaintext, because a debug log line printed the request headers. That token had been valid for 14 months. Nobody had rotated it because rotating it meant coordinating a redeploy across six services that all had it baked into a Kubernetes Secret. That incident is what pushed us to SPIFFE and SPIRE.

The idea is simple even if the implementation has sharp edges: every workload gets a cryptographic identity, that identity is proven with a short-lived X.509 certificate (an SVID), and no service ever holds a shared secret again. Certs live for an hour, not a year.

What SPIFFE actually gives you #

A SPIFFE ID is just a URI. Ours look like spiffe://prod.internal/ns/payments/sa/payments-api. That string names a workload by namespace and service account. SPIRE is the runtime that decides which process on which node is allowed to receive the SVID for that ID.

The trust model rests on attestation. When a pod asks the SPIRE agent for its identity, the agent doesn't take its word for it. It checks the kubelet, reads the pod's service account, verifies the node it's running on, and only then hands over a cert. That chain of checks is the whole point. A compromised pod can't request an identity it wasn't registered for.

Registering workloads without going insane #

The part that bites people is registration entries. Each one maps a set of selectors to a SPIFFE ID. Do this by hand and you'll have a config drift nightmare within a week. We drive it through the SPIRE Kubernetes Workload Registrar, but here's what a manual entry looks like so the mechanics are clear:

bash.bash

spire-server entry create \
  -spiffeID spiffe://prod.internal/ns/payments/sa/payments-api \
  -parentID spiffe://prod.internal/spire/agent/k8s_psat/prod-cluster/node-7 \
  -selector k8s:ns:payments \
  -selector k8s:sa:payments-api \
  -ttl 3600

The -ttl 3600 gives one-hour certs. The agent rotates them at roughly half-life, so around the 30-minute mark, with no restart and no human. The parentID ties the entry to a node that itself proved its identity via the PSAT (projected service account token) node attestor. If you skip node attestation and use join_token in production, you've recreated the shared-secret problem you were trying to kill.

Consuming the SVID in code #

Applications fetch their SVID over the Workload API, a Unix domain socket the agent exposes. You don't parse certs yourself. The go-spiffe library keeps an in-memory copy that auto-updates on rotation:

go.go

source, err := workloadapi.NewX509Source(ctx,
    workloadapi.WithClientOptions(
        workloadapi.WithAddr("unix:///run/spire/sockets/agent.sock")))
if err != nil {
    log.Fatalf("spiffe source: %v", err)
}
tlsConfig := tlsconfig.MTLSClientConfig(source, source,
    tlsconfig.AuthorizeID(spiffeid.RequireFromString(
        "spiffe://prod.internal/ns/ledger/sa/ledger-api")))

That AuthorizeID line is the authorization decision. The ledger client will only complete a handshake with a peer presenting the ledger-api identity. No allowlist of IPs, no API keys checked in the app. The mTLS handshake is the auth.

For services we couldn't recompile (a legacy Java billing app), we ran Envoy as a sidecar with the SPIRE Agent as its SDS provider. Envoy terminates and originates mTLS, the app speaks plain HTTP to localhost, and it never knows SPIFFE exists.

What broke in practice #

Two things cost us real time. First, clock skew. SVIDs are short-lived, so a node whose clock drifted 90 seconds started rejecting valid certs as not-yet-valid. We now run chrony with a hard alert if offset exceeds 250ms.

Second, the SPIRE server datastore. We started on SQLite, which is fine for a demo and terrible for an HA control plane. Under load the registrar hammered it and entry syncs lagged by minutes, so freshly scheduled pods sat without identities and crash-looped. Moving to a Postgres backend with two SPIRE server replicas fixed it:

hcl.hcl

DataStore "sql" {
  plugin_data {
    database_type = "postgres"
    connection_string = "dbname=spire host=spire-db.prod sslmode=require"
  }
}

The call we'd make #

If you're running more than a handful of services on Kubernetes and passing secrets between them, SPIFFE/SPIRE is worth the setup tax. The operational load is real: you're now running a PKI control plane and you need to babysit its datastore and the agents' clocks. For a three-service startup, that's overkill, and a service mesh with built-in mTLS (Linkerd issues its own identities) gets you 80% of the value with far less to run. But once you're past the point where a leaked long-lived token means an all-hands incident, short-lived attested identities stop being a nice-to-have.

Zero-Trust Service-to-Service Auth with SPIFFE and SPIRE

What SPIFFE actually gives you#

Registering workloads without going insane#

Consuming the SVID in code#

What broke in practice#

The call we'd make#

What SPIFFE actually gives you #

Registering workloads without going insane #

Consuming the SVID in code #

What broke in practice #

The call we'd make #

#Zero Trust

Zero-Trust Service-to-Service Auth with SPIFFE and SPIRE