Azure OpenAI's Sweden Central Outage: A Health Check Postmortem
A slow database made a backend service look dead, and the health check that was supposed to protect it killed it faster. The lesson has nothing to do with AI.
Key takeaways
- Microsoft's incident review 7QL5-Z50 shows the September 29, 2026 Azure OpenAI outage in Sweden Central was a restart loop, not a model failure.
- A cache clear sent backend traffic to about 20 times normal, health checks tied to database connectivity restarted instances, and impact lasted from 10:03 to 15:58 UTC until the check was changed.
On this page
The September 29, 2026 Azure OpenAI outage in Sweden Central was a health check restart loop. A cache clear sent backend traffic to about 20 times normal, connections to the cache and database ran out, and health checks that depended on database connectivity kept restarting instances. Impact ran 10:03 to 15:58 UTC and ended when Microsoft reconfigured the check.
Verified against
Microsoft's post incident review for tracking ID 7QL5-Z50 on the Azure status history page, and the Azure Architecture Center Health Endpoint Monitoring pattern. Checked October 2026.
Azure OpenAI Service, Foundry Agent Service, Foundry Models, and Azure AI services all went intermittent in the region. Nothing about the failure involves AI. It is a restart storm, the same shape of outage that takes down ordinary web services every week.
What actually broke
Per Microsoft's review, a planned deployment of an internal service began at 09:38 UTC and cleared its caches. Two defects in how that service read its caches sent most lookups straight to a backend service, and by 10:00 that backend's traffic was about 20 times normal. The backend's cache hit full utilization, latency rose, connections stayed open longer, and node connection capacity ran out, so new connections to the cache and database failed. Customers saw intermittent failures, higher latency, and HTTP 500 and 503 errors for close to six hours.
The review says the incident lasted longer because the backend's health checks depended on database connectivity. Instances that couldn't connect were marked unhealthy and restarted, which shrank healthy capacity and drove more retries. Raising compute and resource limits at 15:00 did not resolve it on its own. Recovery began at 15:40, when engineers reconfigured the health check to stop the restart loop.
That last detail is the whole story. The fix was not "make the database faster." It was "stop restarting instances." Microsoft's completed repair item says the same thing: the health check no longer depends on database connectivity.
The restart storm, mechanically
A saturated dependency and a dead instance look identical to a health check that includes the dependency. The database and cache were overloaded, not gone, and every instance that couldn't get a connection failed its check. The check read that as "this instance is broken" and marked it unhealthy, when the instance itself was fine.
Restarting an unhealthy instance does not fix a slow downstream dependency, it just removes one more instance from the pool. The remaining instances carry more load each, which makes them slower too, which fails the same health check, which restarts more of them. Each restart cycle also adds reconnection and cache-warming load back onto the struggling database, so the dependency worsens at the exact moment the fleet needs it to recover. It self-reinforces until something outside the loop intervenes, which here meant engineers reconfiguring the health check.
The question every health check should answer, and usually doesn't
Most health checks answer one question: did this instance and everything behind it respond within N milliseconds. That conflates two different states. Microsoft's own Health Endpoint Monitoring pattern draws the line: use shallow checks that don't contact dependencies for liveness, "so that a dependency outage doesn't trigger unnecessary application restarts", and keep deep dependency checks for readiness. A dead instance should be restarted. A slow-but-alive one needs load shed from it, not a restart, since restarting does nothing for the slowness and briefly makes capacity worse.
A check that distinguishes the two looks less like a ping and more like a circuit breaker:
# health_check.py: distinguish "slow" from "dead" before acting on it
import time
class DependencyHealth:
def __init__(self, failure_threshold=5, reset_after_s=30):
self.consecutive_timeouts = 0
self.failure_threshold = failure_threshold
self.reset_after_s = reset_after_s
self.opened_at = None
def record(self, latency_ms, timed_out):
if timed_out:
self.consecutive_timeouts += 1
else:
self.consecutive_timeouts = 0
def should_shed_load(self):
# Trips on a pattern, not one slow call, and recovers on a timer
# rather than needing a human to flip it.
if self.consecutive_timeouts >= self.failure_threshold:
self.opened_at = self.opened_at or time.time()
return True
if self.opened_at and time.time() - self.opened_at < self.reset_after_s:
return True
self.opened_at = None
return False
The instance stays in the pool, stops hitting the slow dependency for a cooldown window, and serves what it can from cache or a degraded response. Nobody restarts it. Restarts stay reserved for a process that's actually crashed or refusing connections, a narrower definition of "unhealthy" than "took longer than usual to answer."
Rate-limit your own remediation
The second, independent mistake: nothing capped how many restarts could happen in a window. A health check with no rate limit on the action it triggers is a feedback loop waiting for a slow dependency to set it off:
This config is illustrative pseudo-config, not any specific orchestrator's schema:
# ops/restart-policy.yaml: cap how fast automated remediation can act
restart_policy:
max_restarts_per_window: 3
window_seconds: 300
on_limit_exceeded: page_oncall # not: restart anyway
min_healthy_fraction: 0.6 # never restart below this fleet-wide
min_healthy_fraction is the part teams skip. A restart policy that only looks at one instance at a time has no concept of "the fleet is already thin." Tie the restart decision to current fleet health, not just the instance in front of you, and the storm can't self-reinforce past a floor you chose on purpose.
What this means if you run production workloads against Azure OpenAI
The lesson is not "leave Azure." A single-region data-plane dependency inherits whatever restart-storm risk that region's internal services carry, invisibly, until it's failing. The practical response is regional redundancy for the data plane: a secondary deployment behind health-aware routing, so a Sweden Central-specific incident degrades throughput rather than availability. We cover picking that pattern in multi-region active-active vs active-passive, and the routing layer that makes a second region useful for LLM traffic in multi-provider LLM gateways and fallback routing.
This is a different incident from the September 3 Azure outage we covered earlier, where a regional infrastructure failure took multiple LLM providers down together because they shared a cloud substrate. That one was about provider correlation; this one is about a health check mismanaging a degraded dependency inside a single provider's region. Same category of fix either way: don't assume one region.
The decision, concretely
- Does your health check distinguish slow-but-alive from actually-dead? If it only checks response time against a fixed timeout, no. Add a failure-count threshold before acting.
- Is your restart policy rate-limited? Unlimited restarts in a short window turn a slow dependency into an outage. Cap restarts per window and require sign-off past that cap.
- Do you have a circuit breaker on your database or cache dependency? If a slow read just blocks the request thread, shed load during a cooldown instead of queuing every instance behind the same slow call.
- Does your liveness check touch the database? Then a database brownout restarts your fleet. Move dependency checks to readiness only.
- Are you running production inference against a single Azure region? A six-hour regional incident taking down your product means you need a second region in the routing path, not a second vendor.
Questions people ask
What caused the Azure OpenAI Sweden Central outage on September 29, 2026?
Microsoft's post incident review (tracking ID 7QL5-Z50) says a planned deployment cleared an internal service's caches, and two caching defects pushed backend traffic to about 20 times normal. Connection capacity to the cache and database ran out. Health checks that depended on database connectivity then restarted instances in a loop, which kept capacity low until the check was reconfigured at 15:40 UTC.
Should a liveness probe check the database?
No. A liveness probe should confirm the process can respond, without contacting dependencies, because restarting an instance never repairs a shared database. Put dependency checks on readiness, which removes the instance from traffic without killing it. Microsoft's Health Endpoint Monitoring guidance and the Sweden Central review both point the same way: dependency-aware liveness checks turn a slow database into a restart storm.
The call we'd make
Audit your own autohealing before you audit anyone else's. The Sweden Central incident is a clean, publicly documented case of a safety mechanism amplifying the failure it was built to catch, and that pattern shows up in internal systems more often than postmortems admit. If your health checks can restart an instance without checking fleet health first, fix that this week. Regional redundancy matters too, but it's the slower fix, and the restart-rate-limit is the one you can ship today.
Sources
Cite this article
Kiril Urbonas (2026). Azure OpenAI's Sweden Central Outage: A Health Check Postmortem. DevOpsNess. Updated 2026-11-15. https://www.devopsness.com/blog/azure-sweden-central-openai-outage-postmortem
[Azure OpenAI's Sweden Central Outage: A Health Check Postmortem](https://www.devopsness.com/blog/azure-sweden-central-openai-outage-postmortem) (Kiril Urbonas, DevOpsNess, updated 2026-11-15)
More from Cloud
Explore more articles in this category
GCP Workload Identity Federation: Replacing Service Account Keys
We deleted every static GCP service account key in our org over six weeks. Here's the migration plan, the gotchas, and the policies we now enforce.
External Secrets Operator: One Secrets Workflow Across Clouds
We ran secrets three different ways across AWS, GCP, and Vault. External Secrets Operator gave us one Kubernetes-native workflow. Here's the setup and the gotchas.
Cloud Cost Optimization: The Practical Guide for AWS, Azure, and GCP
Cloud bills grow quietly until someone asks why. This is the map for cutting spend without cutting reliability: where the money actually goes, the levers that work, and the tools worth paying for.
You might have missed
Evergreen posts worth revisiting.