Multi-Provider LLM Gateways: Routing, Fallback, and Cost Control
When a single LLM provider has an outage, every AI feature goes dark at once. A gateway with routing and fallback fixes that, and enforced budgets cut spend as a bonus.
Key takeaways
- An LLM gateway is one internal service that every app calls by task name, and it picks the provider and model.
- Route cheap tasks to small models, fail over on 429, 5xx, 529 and timeouts but never on your own 400s, cap retries, and enforce per-tenant spend limits before the call, not after.
On this page
Put every LLM call behind one internal gateway that apps call by task, not by model. The gateway maps each task to a primary and a fallback provider, fails over only on provider-side errors (429, 5xx, 529, timeouts), caps retries, and refuses calls that would break a spend limit. One fallback provider beats none.
Verified against
Claude API model lineup (Claude Opus 5.5, Sonnet 5.5, Haiku 5.5) and error codes, OpenAI API error codes and models list, and the LiteLLM router docs. Checked October 2026.
The incident that usually buys the budget: the primary provider has a partial outage, and every AI feature fails at once because each service hardcodes the same SDK and endpoint. One provider isn't enough for three reasons: outages, per-organization rate limits that return 429s until the window resets, and cost, because most calls don't need a flagship model.
One internal endpoint, many backends
Every service stops importing a provider SDK and calls the gateway with a normalized request. The gateway owns keys, retries, routing, and logging. Apps get dumber, which is the goal.
# what every service now calls
resp = gateway.complete(
task="support_reply", # logical task, not a model name
messages=messages,
max_tokens=800,
)
The app asks for a task, not a model, so we can reroute without touching a single service. If callers know about providers, every migration is spread across every caller.
Routing by task, not one model for everything
Paying flagship prices for a job a cheap model handles fine is pure waste. We route classification and short extraction to a small fast model and reserve the expensive model for open-ended generation. Model IDs below are the current ones as of October 2026; Anthropic lists Haiku 5.5 at from $0.10 per million input tokens against $4 for Opus 5.5, which is the whole argument in two numbers.
routes:
classify_ticket:
primary: { provider: anthropic, model: claude-haiku-5-5 }
fallback: { provider: openai, model: gpt-6-luna }
max_cost_per_1k_input: 0.20
support_reply:
primary: { provider: anthropic, model: claude-sonnet-5-5 }
fallback: { provider: openai, model: gpt-6.1-sol }
legal_summary:
primary: { provider: anthropic, model: claude-opus-5-5 }
fallback: { provider: openai, model: gpt-6-astra }
Classification is typically the bulk of call volume and rarely needs a top-tier model, so moving it is where the savings come from. Prove parity on a labeled eval set first, then flip it.
Log every call with its task, provider, model, tokens, latency, and computed cost; daily per-task aggregates show which routing decisions pay off. Sometimes one doesn't: a cheap model can regress quality enough that users retry, costing more than the better model would have. We pick providers statically per task and revisit quarterly; live price arbitrage adds complexity for marginal savings.
For the highest-volume cheap tasks, a self-hosted small open-weights model can undercut per-token pricing, but it goes down for upgrades and runs out of capacity in bursts, so it always sits in front of an API fallback.
Fallback that triggers on the right things
The tricky part of fallback is deciding when. Fail over on provider-side problems, never on your own bad request: a 400 for a malformed prompt fails the same way on every provider, so retrying elsewhere wastes money and time. Each error class gets a different policy:
- Rate limit (429): go straight to the next provider. Both Anthropic and OpenAI also return 429 when you exhaust credits or a spend cap, and those never clear on retry, so retrying the same provider just burns time.
- Transient (500, 503, 504, 529, timeout): retry the same provider once, then fail over. Anthropic uses 529
overloaded_error; OpenAI uses 503server_is_overloaded. - Permanent (other 4xx): don't retry. Surface the error. One trap: Anthropic returns a 400 when you hit a spend limit you set yourself, so alert on that code rather than treating every 400 as a prompt bug.
RATE_LIMITED = {429}
TRANSIENT = {500, 502, 503, 504, 529}
def complete_with_fallback(route, req, deadline):
for backend in route.backends[:3]: # primary + max 2 failovers
for attempt in range(2): # max 1 retry per provider
try:
return call(backend, req, timeout=min(30, deadline.remaining()))
except ProviderError as e:
if e.status in RATE_LIMITED:
break # next provider immediately
if e.status not in TRANSIENT:
raise # our bug, don't fail over
log.warning("failover", backend=backend, status=e.status)
raise AllBackendsFailed()
The caps matter as much as the classification: at most two failovers per request, one retry per provider, and a 60-second total budget. Too aggressive and you multiply cost against a degraded provider; too conservative and you fail requests you could have served. Watch for stacked retries, too: the official Anthropic SDK already retries transient failures twice by default, so set max_retries to 0 inside the gateway or your "one retry" becomes six calls.
We also run a circuit breaker per provider: after 5 failures in 30 seconds, skip it for a minute, so a struggling provider doesn't add 30-second timeouts to every request. A continuous health score handles partial degradation better, sending a slow provider fewer requests rather than zero. Off-the-shelf routers ship the same idea: LiteLLM cools a deployment down on a 429 or a failure rate above 50% in the current minute, for a default cooldown_time of 5 seconds.
Cost control you can enforce, not just observe
Dashboards tell you what you already spent. We want limits that stop spend before it happens. The gateway tracks per-task and per-tenant token usage in Redis and refuses calls that would blow a cap.
key = f"spend:{tenant}:{today}"
spent = float(redis.get(key) or 0)
if spent + estimate_cost(req) > tenant_daily_cap(tenant):
raise BudgetExceeded(tenant)
redis.incrbyfloat(key, actual_cost)
The classic catch is a batch job stuck in a retry loop: without a cap it spends all afternoon; with one, it stops at the cap and pages someone instead.
Parity is a production problem, not an eval problem
A prompt tuned on Claude can behave oddly on GPT and vice versa, so every model in the pool needs two checks: an eval set per task, re-run whenever the pool changes, and mirrored production traffic before a new model becomes a fallback. A model that passes the eval can still degrade on real traffic the eval didn't cover.
Streaming costs the most code: each provider's chunk shape is normalized to one internal format, and a stream that fails mid-response is closed cleanly, not retried, because partial output already reached the user, the same partial-result handling we describe in LLM streaming UX: backpressure, cancellation, partial results.
The abstraction tax, and what we watch
A gateway is not free. You give up provider-specific features, prompt caching semantics differ (see prompt caching for production LLM apps for why a request that fails over lands on a cold cache), some providers have structured-output modes others don't, and the normalized request either finds a lowest common denominator or carries per-provider escape hatches. We keep a provider_options passthrough for the cases that genuinely need it and accept that the fallback for those tasks may behave slightly differently.
It is also a new single point of failure, so run at least three stateless replicas. We monitor per-provider success rate per task, failover rate, per-task cost and p99 latency, and error-budget burn per provider.
Questions people ask
Should an LLM gateway fail over on a 429 error?
Yes, to a different provider, not the same one. A 429 means your organization hit a rate limit, or on Anthropic and OpenAI possibly an exhausted credit balance or spend cap, and neither clears with an immediate retry. Send the request to the next backend at once, and let a circuit breaker skip the limited provider for a short cooldown so later requests don't each pay for the failed first attempt.
Should I build an LLM gateway or use LiteLLM?
Start with an existing router unless you have unusual needs. LiteLLM already implements ordered fallbacks, per-error retry policies, cooldowns and routing strategies such as simple-shuffle, latency-based and cost-based. Build your own when you need tenant-level budgets tied to your billing, custom parity mirroring, or tight control over streaming normalization, and even then the patterns here are the ones to copy.
The decision, concretely
- More than two or three services calling LLMs? Build or adopt the gateway; hardcoded providers make every outage a code change.
- One script calling one model? Don't. You're adding a moving part for nothing.
- High-volume classification or extraction? Route it to a small model or self-hosted, with an API fallback, after proving parity.
- Adding a new provider? Mirror real traffic before making it a fallback, and only promote it to primary after that.
The call we'd make
Even one fallback provider is a large resilience improvement over none, so start there: a task-labelled gateway with a primary and a fallback per task, error-class-aware failover with hard caps, and enforced spend limits. The routing and cost controls end up mattering more day to day than the failover, but it's the failover that justifies the project after your first provider outage.
Sources
Cite this article
Kiril Urbonas (2026). Multi-Provider LLM Gateways: Routing, Fallback, and Cost Control. DevOpsNess. Updated 2026-11-15. https://www.devopsness.com/blog/multi-provider-llm-gateways-routing-fallback
[Multi-Provider LLM Gateways: Routing, Fallback, and Cost Control](https://www.devopsness.com/blog/multi-provider-llm-gateways-routing-fallback) (Kiril Urbonas, DevOpsNess, updated 2026-11-15)
More from AI
Explore more articles in this category
Best RAG Frameworks in 2026 Compared
A practitioner comparison of the RAG frameworks worth using in 2026, from LlamaIndex and LangChain to Haystack, DSPy, and raw code.
Build Your First RAG App in 100 Lines of Python
A working retrieval-augmented generation app you can run today. Markdown ingestion, embeddings, semantic search, and an LLM answer, start to finish in one afternoon.
Building RAG Applications: A Complete Guide to Retrieval Augmented Generation
The end-to-end pattern for production RAG applications, and the steps teams most often skip.
You might have missed
Evergreen posts worth revisiting.