Our failover config looked perfect in the console and did nothing during a real outage. Here's the health-check design that actually flipped regions when it mattered.
Multi-Region Failover with Route 53: Health Checks and Gotchas
The us-east-1 region degraded on a Tuesday, our app in that region started returning 503s, and Route 53 did absolutely nothing. Traffic kept flowing to the dead region for eleven minutes while we scrambled. The failover config had passed every review. The problem was that our health check was pinging / on the load balancer, and the load balancer was perfectly healthy. It was the app behind it that was broken. The health check was checking the wrong thing.
Route 53 failover routing is simple in shape: a primary record and a secondary record, each tied to a health check. When the primary's health check fails, Route 53 serves the secondary. The trap is that the health check has to test the thing that actually breaks, not the nearest green light.
A health check that hits your ALB's / tells you the ALB is up. It tells you nothing about whether the app can reach its database. We moved the check to a deep endpoint that exercises the real dependency path:
@app.get("/healthz/deep")
async def deep_health():
# actually touch the dependencies that define "working"
await db.execute("SELECT 1")
await cache.ping()
return {"status": "ok", "region": REGION}
If the database is unreachable, /healthz/deep returns 500, the health check fails, and failover fires. A shallow check would have stayed green through the whole outage, which is exactly what happened to us.
resource "aws_route53_health_check" "primary" {
fqdn = "us-east-1.api.devopsness.io"
port = 443
type = "HTTPS"
resource_path = "/healthz/deep"
failure_threshold = 3
request_interval = 10
}
resource "aws_route53_record" "primary" {
zone_id = var.zone_id
name = "api.devopsness.io"
type = "A"
set_identifier = "primary"
failover_routing_policy { type = "PRIMARY" }
health_check_id = aws_route53_health_check.primary.id
alias {
name = aws_lb.us_east.dns_name
zone_id = aws_lb.us_east.zone_id
evaluate_target_health = true
}
}
With request_interval = 10 and failure_threshold = 3, Route 53 declares the endpoint unhealthy after roughly 30 seconds of failures. Health checkers run from multiple AWS locations and need a majority to agree, so a single flaky checker won't trip it.
Here's the number people ignore until it hurts: the DNS record TTL. Route 53 can flip the record the moment the health check fails, but clients keep using the cached old answer until their TTL expires. If your record TTL is 300 seconds, some clients stay pinned to the dead region for five minutes after failover technically happened.
We set failover records to a 60-second TTL. Lower feels tempting but increases query volume and cost, and many resolvers won't honor anything under 30 seconds anyway. Sixty is the sweet spot: fast enough that recovery is measured in a minute, not five, and not so aggressive that you're hammering DNS.
# on non-alias records, set this explicitly
ttl = 60
Alias records to AWS resources don't take a manual TTL (AWS manages it, effectively 60 seconds), which is another reason to use alias records for the failover targets.
Setting evaluate_target_health = true on the alias makes Route 53 also consider the ALB's own target health. This is good, but it interacts with your explicit health check. If both are set, both have to be happy. We once had a case where the deep health check passed but the ALB reported zero healthy targets during a deploy, and failover fired mid-deploy and shifted traffic we didn't want to move. Decide which signal is authoritative. We kept the deep check as the primary signal and set evaluate_target_health = false on the alias for the app records to avoid the deploy-time flapping.
A failover config that never tests the secondary is a config that fails twice. We run the same /healthz/deep check against the secondary region on the same schedule, and we alert if the secondary is unhealthy even while the primary is fine. A dead standby is a landmine you step on at the worst possible moment. We also run a monthly game day that forces the primary health check to fail (block the checker IPs at the security group) and confirms traffic actually lands in the secondary region end to end.
Route 53 failover works, but only if the health check tests your real dependency chain, not a load balancer's pulse. Point it at a deep endpoint that touches the database and cache. Set a 60-second TTL so recovery is a minute, not five. Decide whether the explicit check or target health is authoritative and don't let them fight during deploys. And monitor the standby as closely as the primary. Do that and the next Tuesday outage is a 60-second blip in a graph instead of eleven minutes of scrambling.
