GitHub's September 13 Outage: One Cleanup Job, Many Services

GitHub's September 13, 2026 outage was caused by an internal data-cleanup job whose only brake was replica lag. Lag stayed healthy while the primary database hit its connection limit, breaking issues, pull requests, token creation, signups and some Actions runs. The fix for your own systems: pace background work on primary load too, bound retries, and set fast timeouts.

What GitHub says happened

GitHub's availability report dates the incident to 09:16 UTC, lasting 1 hour 28 minutes, and describes roughly two hours of customer impact overall. The trigger was a background data-cleanup job that increased load on a shared database. It kept running until the primary hit its connection limit, and requests that depended on that database failed or stalled. The status page at the time listed impact across a long run of services. The report itself groups the damage as opening or updating issues and pull requests, API calls, access token creation, starting Actions workflows, and signups.

The reported impact was uneven and, in places, brutal. About 96% of requests to create issues through the web interface were affected, about 90% of new signups had intermittent failures, and about 4% of Actions workflow runs did. Responders reduced internal load and paused the job, and error rates returned to normal.

We are deliberately quoting only GitHub's own numbers. Third-party trackers disagree on start times and percentages, and none of the disagreement changes the engineering story.

The safeguard measured the wrong thing

According to the report, the job slowed its writes when replicas fell behind, but its safeguard "did not account for incoming load on the primary database server." Replica lag stayed low the whole time. The job wrote to the primary, and the primary ran out of headroom. Replicas replay what the primary has already committed, so lag tells you about the replication pipeline, not about how much the primary is struggling to accept work.

Replica lag is a lagging, single-purpose indicator. It answers "will a read-after-write hit stale data?" and gets used as a proxy for "is the database okay?", which is a different question. GitHub's own freno throttler was built around replica lag, and it is a good tool for the problem it solves. The mistake is treating one good signal as a complete one. Our notes on routing reads to Postgres replicas cover what lag does and doesn't tell you.

Throttle on saturation, not just lag

A background job should slow down when any of several signals goes bad: primary connection utilization, active-query latency, replica lag, and error rate on the foreground path. Any single one crossing its line pauses the job. This is the cheap version, and it fits in a shell loop around a batch. It reads pg_stat_activity, which has one row per server process.

# throttle-cleanup.sh: run one batch only if the primary has headroom
MAX_CONN_PCT=60
pct=$(psql "$PRIMARY_URL" -Atc \
    "SELECT round(100.0 * count(*) / current_setting('max_connections')::int)
     FROM pg_stat_activity")
if [ "$pct" -gt "$MAX_CONN_PCT" ]; then
    echo "connections at ${pct}%, sleeping"; sleep 30; exit 0
  fi
psql "$PRIMARY_URL" -c "DELETE FROM audit_rows
    WHERE id IN (SELECT id FROM audit_rows
                 WHERE created_at < now() - interval '400 days'
                 LIMIT 1000)"

Three details matter more than the threshold. Batches stay small so the job can stop within seconds. The check runs before every batch, not once at startup. And the job needs a kill switch a human can flip without a deploy, because GitHub's own recovery came from pausing it.

Retries turned a spike into a plateau

GitHub says a retry loop around token creation kept re-sending writes that were already failing, which held the database saturated, and that database calls had no quick timeout, so request handlers waited on the stalled database instead of failing fast. Its listed follow-ups: make background work respond to available database capacity, reduce unnecessary retries during failures, limit the impact of slow database requests, and keep temporary connectivity problems from delaying service startup and recovery.

That is the amplification pattern in every large outage: the first failure is small, and clients turn it into a sustained one. If your services retry, they need a cap, jitter and a deadline shorter than the caller's. If yours are unbounded, you are one bad hour from writing the same paragraph about yourselves.

What your own CI should do while GitHub is degraded

About 4% of Actions workflow runs were affected, so most of your pipelines probably ran fine that morning and a few failed for reasons that had nothing to do with your code. That distribution is the dangerous one, because it looks like flakiness. Set timeouts so a stuck job does not eat runner minutes (the default timeout-minutes is 360), and retry only the steps that fetch from the network.

jobs:
  test:
    runs-on: ubuntu-latest
    timeout-minutes: 20
    steps:
      - uses: actions/checkout@v7
      - name: Install dependencies
        run: |
          for i in 1 2 3; do npm ci && break || sleep $((i * 20)); done
      - run: npm test

Do not add blanket workflow-level reruns. Retrying a whole pipeline against a degraded token service is the same amplification GitHub described, multiplied by your team. Our GitHub Actions recipes cover the timeout and concurrency settings worth defaulting on.

Also decide ahead of time who can ship a hotfix without GitHub, and write it in the runbook.

What the postmortem should ask

The useful question is not who wrote the cleanup job. It is why a job writing to a high-fanout shared database was governed by one metric. Our notes on incident post-mortems that drive change argue for ending a review with owned, dated actions, and GitHub's follow-up list reads like that.

Questions people ask

What caused the GitHub outage on September 13, 2026?

A background data-cleanup job overloaded a shared database. Its safeguard slowed writes only when replicas lagged, and lag stayed low while the primary filled up to its connection limit. Requests depending on that database failed or stalled, a retry loop around token creation kept it saturated, and missing timeouts slowed recovery. GitHub paused the job and shed internal load to recover.

Why is replica lag not enough to throttle a batch job?

Replica lag measures how far replicas trail the primary, not how loaded the primary is. A write-heavy job can exhaust the primary's connections or CPU while replicas keep up perfectly, which is what GitHub described. Throttle on primary connection utilization, query latency and foreground error rate as well, and pause when any one of them crosses its threshold.

The decision, concretely

• Pacing a background job on replica lag alone? Add primary connection utilization and foreground error rate, and pause on any one of them.
• Running batch deletes or backfills against a shared primary? Keep batches small, check before each one, and keep a kill switch that needs no deploy.
• Retrying failed writes in a client? Cap attempts, add jitter, and set a request timeout shorter than the caller's.
• Depending on GitHub for deploys? Set timeout-minutes, retry only network-fetching steps, and document a manual hotfix path.

The call we'd make

Treat every background job as production traffic with a lower priority, and give it the same multi-signal brakes you would give a foreground service. GitHub's follow-ups point the same direction. If you can only do one thing this week, add a connection-utilization check to your heaviest batch job, then run a review using our blameless postmortem template on your own last near-miss.

Sources

• GitHub availability report: September 2026
• GitHub Actions workflow syntax reference
• PostgreSQL: the cumulative statistics system and pg_stat_activity