Multi-Region Failover with Route 53: Health Checks and Gotchas
Our failover config looked perfect in the console and did nothing during a real outage. Here's the health-check design that actually flipped regions when it mattered.
Key takeaways
- Route 53 failover only fires when the health check tests what actually breaks.
- Point it at a deep endpoint that touches the database and cache, not the load balancer root.
- Keep record TTLs near 60 seconds, choose one authoritative health signal so deploys do not trigger failover, and monitor the secondary region as closely as the primary.
On this page
Route 53 failover only fires if the health check tests what actually breaks. Point an HTTPS health check at a deep endpoint that touches your database and cache, not the load balancer's /. Keep record TTLs around 60 seconds, pick one authoritative health signal so deploys don't flap, and monitor the secondary region as closely as the primary.
Verified against
Route 53 health checks and failover routing, against the AWS Route 53 Developer Guide (health check values, how Route 53 determines endpoint health, alias records) and the Route 53 pricing page. Checked October 2026.
The us-east-1 region degraded on a Tuesday, our app in that region started returning 503s, and Route 53 did absolutely nothing. Traffic kept flowing to the dead region for eleven minutes while we scrambled. The failover config had passed every review. The problem was that our health check was pinging / on the load balancer, and the load balancer was perfectly healthy. It was the app behind it that was broken. The health check was checking the wrong thing.
Failover records need a health check that means something
Route 53 failover routing is simple in shape: a primary record and a secondary record, each tied to a health check. When the primary's health check fails, Route 53 serves the secondary. The trap is that the health check has to test the thing that actually breaks, not the nearest green light. (Whether you want a passive standby at all is a separate decision; we compare the options in active-active vs active-passive multi-region.)
A health check that hits your ALB's / tells you the ALB is up. It tells you nothing about whether the app can reach its database. We moved the check to a deep endpoint that exercises the real dependency path:
async def deep_health():
# actually touch the dependencies that define "working"
await db.execute("SELECT 1")
await cache.ping()
return {"status": "ok", "region": REGION}
If the database is unreachable, /healthz/deep returns 500, the health check fails, and failover fires. A shallow check would have stayed green through the whole outage, which is exactly what happened to us. Keep the deep check cheap, though: Route 53 checkers in many locations hit it, so a slow query here becomes load.
The Route 53 config
resource "aws_route53_health_check" "primary" {
fqdn = "us-east-1.api.devopsness.io"
port = 443
type = "HTTPS"
resource_path = "/healthz/deep"
failure_threshold = 3
request_interval = 10
}
resource "aws_route53_record" "primary" {
zone_id = var.zone_id
name = "api.devopsness.io"
type = "A"
set_identifier = "primary"
failover_routing_policy { type = "PRIMARY" }
health_check_id = aws_route53_health_check.primary.id
alias {
name = aws_lb.us_east.dns_name
zone_id = aws_lb.us_east.zone_id
evaluate_target_health = false # the deep check is authoritative; see below
}
}
With request_interval = 10 and failure_threshold = 3, an endpoint is marked unhealthy after roughly 30 seconds of consecutive failures. Two details from the AWS docs matter. First, the verdict is not a majority vote: Route 53 considers an endpoint healthy if more than 18% of its health checkers report it healthy, so it only goes unhealthy when nearly all of them agree. Second, the 10-second interval is the paid "fast" option and can't be changed after the health check is created, so decide before you apply.
The TTL trap that stretches your outage
Here's the number people ignore until it hurts: the DNS record TTL. Route 53 can flip the record the moment the health check fails, but clients keep using the cached old answer until their TTL expires. If your record TTL is 300 seconds, some clients stay pinned to the dead region for five minutes after failover technically happened.
We set non-alias failover records to a 60-second TTL. Lower feels tempting but increases query volume and cost, and some resolvers and client runtimes apply their own minimum or cache longer regardless. Sixty is our default: recovery measured in about a minute, not five, without hammering DNS.
# route53.tf (non-alias failover records only)
ttl = 60
Alias records to AWS resources don't take a manual TTL: per the Route 53 docs, Route 53 uses the default TTL for the target resource, which is one more reason to use alias records for load balancer failover targets. When failover looks slow in testing, check client-side caching with the steps in our Linux DNS troubleshooting guide before blaming Route 53.
Evaluate target health, but understand the double-check
Setting evaluate_target_health = true on an alias makes Route 53 also consider the ALB's own target health. This is useful, but it interacts with your explicit health check: if both are set, both have to be happy. We once had a case where the deep health check passed but the ALB reported zero healthy targets during a deploy, and failover fired mid-deploy and shifted traffic we didn't want to move. Decide which signal is authoritative. We kept the deep check as the primary signal and set evaluate_target_health = false on the alias for the app records, as in the config above, to avoid the deploy-time flapping.
Don't forget the secondary can be sick too
A failover config that never tests the secondary is a config that fails twice. We run the same /healthz/deep check against the secondary region on the same schedule, and we alert if the secondary is unhealthy even while the primary is fine. A dead standby is a landmine you step on at the worst possible moment. We also run a monthly game day that forces the primary health check to fail (block the checker IPs at the security group) and confirms traffic actually lands in the secondary region end to end. If the standby's database is a replica, rehearse that side too; Postgres replication lag and failover practice covers what breaks there.
Questions people ask
How long does Route 53 DNS failover take?
Detection plus caching. With a 10-second request interval and a failure threshold of 3, Route 53 marks the primary unhealthy after about 30 seconds of consecutive failures; with the standard 30-second interval it is about 90 seconds. Clients then keep the old answer until their cached TTL expires, so a 60-second TTL adds up to another minute. Expect one to three minutes end to end, longer for clients that cache aggressively.
Should I use evaluate target health with a Route 53 health check?
Pick one authoritative signal. With both set on an alias failover record, Route 53 needs both the explicit health check and the load balancer's target health to be healthy, so a deploy that briefly empties the target group can trigger failover even when the app is fine. We keep a deep application health check as the source of truth and disable target health evaluation on those records.
How much do Route 53 health checks cost?
As of October 2026, a basic health check on an AWS endpoint is $0.50 per month and $0.75 for non-AWS endpoints. Optional features such as HTTPS, string matching and the fast 10-second interval add $1.00 per feature per month on AWS endpoints ($2.00 on non-AWS). AWS lists up to 50 health checks on AWS endpoints in the same account as free, and checks of Elastic Load Balancing resources are not billed.
The call we'd make
Route 53 failover works, but only if the health check tests your real dependency chain, not a load balancer's pulse. Point it at a deep endpoint that touches the database and cache. Set a 60-second TTL so recovery is a minute, not five. Decide whether the explicit check or target health is authoritative and don't let them fight during deploys. And monitor the standby as closely as the primary. Do that and the next Tuesday outage is a short blip in a graph instead of eleven minutes of scrambling.
Sources
Cite this article
Kiril Urbonas (2026). Multi-Region Failover with Route 53: Health Checks and Gotchas. DevOpsNess. Updated 2026-10-31. https://www.devopsness.com/blog/multi-region-failover-route-53-health-checks
[Multi-Region Failover with Route 53: Health Checks and Gotchas](https://www.devopsness.com/blog/multi-region-failover-route-53-health-checks) (Kiril Urbonas, DevOpsNess, updated 2026-10-31)
More from Cloud
Explore more articles in this category
Prometheus vs Datadog: When Open Source Wins
The metrics stack you self-host is free software plus a real ops bill. Datadog hands you everything and mails you the invoice. Here's how we pick.
Best Serverless Databases in 2026 (Compared)
A practitioner comparison of the leading serverless databases by use case, cold-start behavior, branching, pricing model, and lock-in.
Cloud Disaster Recovery Runbook Design: How Small Teams Rehearse Multi-Region Failover
A practical disaster recovery runbook guide for small cloud teams that need realistic failover steps, clear ownership, and repeatable rehearsals instead of shelfware documents.
You might have missed
Evergreen posts worth revisiting.