GCP Workload Identity Federation: Replacing Service Account Keys

GCP Workload Identity Federation: Replacing Service Account Keys#

Six weeks ago we had 34 GCP service account JSON keys scattered across CI systems, developer laptops, and one regrettable Slack DM. Today we have zero. Every workload — CI, on-prem agents, third-party SaaS — authenticates via Workload Identity Federation. Here's the migration plan, the friction we hit, and the policies that now keep this from regressing.

Why Static Keys Are the Problem#

Three near-misses in the last year:

1. A CI runner image accidentally bundled a creds.json and was pushed to a public ECR mirror. Caught in 23 minutes by GCP's automated detection — but still.
2. A laptop with a long-lived key was reported missing during travel. Key had Editor on the dev project.
3. An ex-employee's key kept working for 11 months because no one rotated it. Discovered during a routine audit.

Static credentials don't expire and don't know who's using them. They're the worst of all worlds.

What Workload Identity Federation Actually Is#

WIF lets external identities (GitHub OIDC tokens, AWS IAM roles, OIDC from Kubernetes, SAML, etc.) impersonate GCP service accounts without any long-lived credential.

The flow:

External Identity (GitHub OIDC token)
            │
            ▼
Workload Identity Pool (validates issuer + claims)
            │
            ▼
Workload Identity Provider (maps claims → attributes)
            │
            ▼
Service Account Impersonation (short-lived token, ≤1h)
            │
            ▼
GCP API call

The "credential" your workload sees is an identity token from its native trust system (GitHub, AWS, K8s). GCP validates that token and issues a short-lived access token in exchange.

The 6-Week Migration#

Week 1: Inventory#

# Find every service account key in every project
for proj in $(gcloud projects list --format="value(projectId)"); do
  gcloud iam service-accounts list --project=$proj --format=json \
    | jq -r '.[].email' \
    | while read sa; do
        gcloud iam service-accounts keys list --iam-account=$sa --project=$proj \
          --filter="keyType=USER_MANAGED" --format=json
      done
done > all-keys.json

We found 34 user-managed keys across 5 projects. For each, we mapped: who downloaded it, where it lives now, what it's used for.

Two keys turned out to be completely unused — leftover from migrations years ago. Delete them first; instant security win.

Week 2: Stand Up the Pool#

resource "google_iam_workload_identity_pool" "github" {
  workload_identity_pool_id = "github-actions-pool"
  display_name              = "GitHub Actions"
  description               = "Identity pool for GitHub Actions OIDC"
}

resource "google_iam_workload_identity_pool_provider" "github" {
  workload_identity_pool_id          = google_iam_workload_identity_pool.github.workload_identity_pool_id
  workload_identity_pool_provider_id = "github"

  attribute_mapping = {
    "google.subject"            = "assertion.sub"
    "attribute.repository"      = "assertion.repository"
    "attribute.repository_owner" = "assertion.repository_owner"
    "attribute.workflow_ref"    = "assertion.workflow_ref"
  }

  attribute_condition = "assertion.repository_owner == 'kirilurbonas'"

  oidc {
    issuer_uri = "https://token.actions.githubusercontent.com"
  }
}

That attribute_condition is the most important line. Without it, any GitHub Actions workflow on the entire internet can authenticate to your pool. Restrict to your org.

Week 3: Bind Service Accounts to External Identities#

For each service account that CI used:

resource "google_service_account_iam_binding" "ci_deployer" {
  service_account_id = google_service_account.ci_deployer.name
  role               = "roles/iam.workloadIdentityUser"

  members = [
    "principalSet://iam.googleapis.com/projects/${var.project_number}/locations/global/workloadIdentityPools/${google_iam_workload_identity_pool.github.workload_identity_pool_id}/attribute.repository/kirilurbonas/devopsness",
  ]
}

This binding says: any GitHub Actions workflow in kirilurbonas/devopsness can impersonate ci_deployer@. Not other repos in our org. Specific.

Week 4: Migrate Workflows in Waves#

Old workflow:

- uses: google-github-actions/auth@v2
  with:
    credentials_json: ${{ secrets.GCP_SA_KEY }}

New workflow:

permissions:
  id-token: write   # required for GitHub OIDC
  contents: read

steps:
  - uses: google-github-actions/auth@v2
    with:
      workload_identity_provider: projects/123456789012/locations/global/workloadIdentityPools/github-actions-pool/providers/github
      service_account: ci_deployer@my-project.iam.gserviceaccount.com

No secrets. The OIDC token is generated automatically by GitHub for each job.

Week 5: Migrate On-Prem Workloads (the hard one)#

We had a Jenkins instance running outside any cloud. WIF requires an OIDC issuer, which Jenkins doesn't natively provide.

Solution: stand up a small OIDC provider next to Jenkins (we used spiffe/spire) that issues short-lived JWTs to Jenkins jobs. Configure GCP WIF to trust that issuer. Jenkins jobs now get GCP credentials via WIF without ever holding a static key.

The setup took a week. Worth it: this Jenkins instance was the holder of two of the most powerful service account keys we had.

Week 6: Decommission#

# For each migrated service account:
for sa in ci_deployer@... jenkins_deployer@... ; do
  gcloud iam service-accounts keys list --iam-account=$sa \
    --filter="keyType=USER_MANAGED" --format='value(name)' \
    | while read keyId; do
        gcloud iam service-accounts keys delete $keyId \
          --iam-account=$sa --quiet
      done
done

We waited two weeks after this before deleting the now-disabled service accounts entirely. Nothing broke.

The Org-Wide Policy We Now Enforce#

# Organization Policy
constraint: constraints/iam.disableServiceAccountKeyCreation
listPolicy:
  allValues: DENY

This blocks creation of new user-managed service account keys org-wide. The single exception: a tagged emergency-only project where keys are allowed but require a 2-person approval to issue.

Within 6 weeks we went from "trust the team to do the right thing" to "the platform doesn't permit the wrong thing."

What Surprised Us#

1. WIF Is Not Free in Latency#

Each WIF token exchange adds ~150ms to a job's startup. For short jobs (< 30s), that's noticeable. We mitigated by reusing the access token within a single job (gcloud config set instead of re-authenticating per command).

2. Some SDKs Don't Auto-Refresh#

Older client libraries fetched a token once and held it. After 1 hour, calls failed. We had to upgrade three Python services to google-cloud-* libraries that auto-refresh.

3. Audit Log Volume Tripled#

Every WIF token exchange is logged. For active CI fleets this is a lot of events. We added a sink to BigQuery, now 90% of our audit log queries hit BQ instead of Cloud Logging — significantly cheaper.

4. Third-Party SaaS Tools Are the Last Mile#

Two SaaS tools (a security scanner and a deploy automation tool) only support static keys. We:

• Filed feature requests with both.
• For one, we issue a short-lived key (gcloud iam service-accounts keys create --validity=72h) on a weekly schedule via Cloud Scheduler. Not perfect but bounded blast radius.
• For the other, replaced the tool. Cheaper than maintaining a perpetual key.

Best Practices We'd Recommend#

1. Apply constraints/iam.disableServiceAccountKeyCreation at the organization level before you migrate. Otherwise old patterns creep back in.
2. Use attribute_condition on every WIF provider. Restrict to your repos/accounts.
3. Bind service accounts to specific repos, not the whole pool. Principle of least privilege.
4. Log every WIF token exchange to a SIEM. The audit trail is the value.
5. Test the rollback before deleting old keys. We kept disabled keys for 2 weeks before deletion.
6. Document the WIF provider configuration in your runbook. The first time CI breaks at 2am, the on-call engineer will need to re-grant the binding.

What's Still Worse Than Static Keys#

• Initial complexity. WIF has more moving parts. The first hour of setup feels heavier than gcloud iam service-accounts keys create.
• Debugging mismapped attributes is annoying. Permission denied errors are generic; you have to enable detailed audit logs to see what claim didn't match.
• GCP-specific knowledge is required. The pool/provider/binding model has its own vocabulary.

These costs are paid once. The benefits compound forever.

Would We Migrate Again?#

Without question. The single biggest security improvement we shipped this year — and the second-biggest reduction in toil. No more "rotate this key" Jira tickets. No more wondering where a credential file ended up. No more 11-month-old access from former employees.

If you have any service account keys in your org, start with the inventory query above. The number will be larger than you think; the migration will be smaller than you fear.