GitHub's September 13 Outage: One Cleanup Job, 28 Services

Between 08:43 and 10:44 UTC on 13 September 2026, GitHub was degraded across roughly 28 services, and the trigger was not an attack, a deploy or a hardware fault. It was an internal data-cleanup job whose only brake was replica lag, a signal that stayed healthy while the primary drowned. The lesson for anyone running background work is to pace it on several health signals, and the lesson for anyone depending on GitHub is to stop assuming your CI can always reach it.

What GitHub says happened#

GitHub's incident report lists Issues, Pull Requests, Actions, Codespaces, Pages, Notifications, Code Scanning, Git LFS and new account signup among the affected services. The cleanup job began writing to a shared database cluster at 07:33 UTC, over an hour before the user-visible window opened. That cluster stores permission data, which means nearly every authenticated request reads from it.

The reported impact was uneven and, in places, brutal. Creating issues through the web interface failed for about 96% of attempts, and signup failures were above 90%. At peak, 8.8% of requests to create GitHub App installation access tokens failed, and Actions workflows were affected at roughly 4%. GitHub reports mitigation, load shedding plus pausing the job, by 10:26 UTC, with recovery at 10:44 UTC.

We are deliberately quoting only GitHub's own numbers. Third-party trackers disagree on start times and percentages, and none of the disagreement changes the engineering story.

The safeguard measured the wrong thing#

GitHub's post says the safeguard pacing the job "watched only one health signal", how far the replicas were lagging, and that signal stayed low the whole time. The job wrote to the primary, and the primary ran out of headroom. Replicas replay what the primary has already committed, so lag tells you about the replication pipeline, not about how much the primary is struggling to accept work.

Replica lag is a lagging, single-purpose indicator. It answers "will a read-after-write hit stale data?" and gets used as a proxy for "is the database okay?", which is a different question. GitHub's own freno throttler was built around replica lag, and it is a good tool for the problem it solves. The mistake is treating one good signal as a complete one.

Throttle on saturation, not just lag#

A background job should slow down when any of several signals goes bad: primary connection utilization, active-query latency, replica lag, and error rate on the foreground path. Any single one crossing its line pauses the job. This is the cheap version, and it fits in a shell loop around a batch.

# throttle-cleanup.sh: run one batch only if the primary has headroom
$ MAX_CONN_PCT=60
$ pct=$(psql "$PRIMARY_URL" -Atc \
    "SELECT round(100.0 * count(*) / current_setting('max_connections')::int)
     FROM pg_stat_activity")
$ if [ "$pct" -gt "$MAX_CONN_PCT" ]; then
    echo "connections at ${pct}%, sleeping"; sleep 30; exit 0
  fi
$ psql "$PRIMARY_URL" -c "DELETE FROM audit_rows
    WHERE id IN (SELECT id FROM audit_rows
                 WHERE created_at < now() - interval '400 days'
                 LIMIT 1000)"

Three details matter more than the threshold. Batches stay small so the job can stop within seconds. The check runs before every batch, not once at startup. And the job needs a kill switch a human can flip without a deploy, because GitHub's own recovery came from pausing it.

Retries turned a spike into a plateau#

GitHub says a retry loop around token creation kept re-sending writes that were already failing, which held the database saturated, and that database calls lacked fast timeouts. Its listed follow-ups include rate-limiting background jobs, watching primary load automatically, bounding retries, adding request-level timeouts and breaking up the cluster.

That is the amplification pattern in every large outage: the first failure is small, and clients turn it into a sustained one. If your services retry, they need a cap, jitter and a deadline shorter than the caller's. If yours are unbounded, you are one bad hour from writing the same paragraph about yourselves.

What your own CI should do while GitHub is degraded#

About 4% of Actions workflows were affected, so most of your pipelines probably ran fine that morning and a few failed for reasons that had nothing to do with your code. That distribution is the dangerous one, because it looks like flakiness. Set timeouts so a stuck job does not eat runner minutes, and retry only the steps that fetch from the network.

# .github/workflows/ci.yml
jobs:
  test:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    steps:
      - uses: actions/checkout@v4
      - name: Install dependencies
        run: |
          for i in 1 2 3; do npm ci && break || sleep $((i * 20)); done
      - run: npm test

Do not add blanket workflow-level reruns. Retrying a whole pipeline against a degraded token service is the same amplification GitHub described, multiplied by your team. Our GitHub Actions recipes cover the timeout and concurrency settings worth defaulting on.

Also decide ahead of time who can ship a hotfix without GitHub, and write it in the runbook.

What the postmortem should ask#

The useful question is not who wrote the cleanup job. It is why a job writing to the highest-fanout database was governed by one metric. Our notes on incident post-mortems that drive change argue for ending a review with owned, dated actions, and GitHub's follow-up list reads like that.

The decision, concretely#

• Pacing a background job on replica lag alone? Add primary connection utilization and foreground error rate, and pause on any one of them.
• Running batch deletes or backfills against a shared primary? Keep batches small, check before each one, and keep a kill switch that needs no deploy.
• Retrying failed writes in a client? Cap attempts, add jitter, and set a request timeout shorter than the caller's.
• Depending on GitHub for deploys? Set timeout-minutes, retry only network-fetching steps, and document a manual hotfix path.

The call we'd make#

Treat every background job as production traffic with a lower priority, and give it the same multi-signal brakes you would give a foreground service. GitHub's follow-ups point the same direction. If you can only do one thing this week, add a connection-utilization check to your heaviest batch job, then run a review using our blameless postmortem template on your own last near-miss.