Multi-Provider LLM Gateways: Routing, Fallback, and Cost Control

On a Tuesday afternoon our primary model provider had a partial outage, roughly 40 minutes of elevated 529s and timeouts. Every AI feature in the product (summarization, the support bot, the classification pipeline) failed at once because they all called the same SDK pointed at the same endpoint. We had no way to shift traffic because the provider was hardcoded in a dozen services.

That incident bought the budget for an LLM gateway: a single internal service every app calls, which then decides which provider and model to actually hit. One provider isn't enough for three reasons, in the order we feel them: outages (every major provider has had multi-hour incidents), per-organization rate limits that return 429s until the window resets, and cost, because most calls don't need a flagship model.

One internal endpoint, many backends

The first win is boring but important. Every service stops importing a provider SDK and instead calls our gateway with a normalized request. The gateway owns keys, retries, routing, and logging. Apps get dumber, which is the goal.

# what every service now calls
resp = gateway.complete(
    task="support_reply",       # logical task, not a model name
    messages=messages,
    max_tokens=800,
)

The app asks for a task, not a model. That indirection lets us reroute without touching a single service, and it's why we never let application code pick a provider: if callers know about providers, every migration is spread across every caller. The mapping from task to model lives in gateway config.

Routing by task, not one model for everything

Paying flagship prices for a job a cheap model handles fine is pure waste. We route classification and short extraction to a small fast model and reserve the expensive model for open-ended generation.

# gateway/routes.yaml
routes:
  classify_ticket:
    primary:  { provider: openai,    model: gpt-4o-mini }
    fallback: { provider: anthropic, model: claude-haiku }
    max_cost_per_1k_input: 0.20
  support_reply:
    primary:  { provider: anthropic, model: claude-sonnet }
    fallback: { provider: openai,    model: gpt-4o }
  legal_summary:
    primary:  { provider: anthropic, model: claude-opus }
    fallback: { provider: openai,    model: gpt-4o }

Moving classification off the flagship model was where the 30% savings came from. That one task was 55% of our call volume and none of it needed a top-tier model. We proved parity on a labeled eval set first, then flipped it.

Task labels come before routing rules. Every call is logged with its task, provider, model, input and output tokens, latency, and computed cost, and daily per-task aggregates show which routing decisions pay off. Sometimes one doesn't: a task moved to a cheap model can regress in quality enough that users retry, costing more than the better model would have the first time. We pick providers statically per task and revisit quarterly; per-request price arbitrage on live pricing added complexity for marginal savings.

Self-hosting the cheap tail

For the highest-volume cheap tasks we also run a self-hosted Llama-3-8B-Instruct on a single L4 GPU, about $200 a month all-in. For work that fits its capability, that's far cheaper than per-token API pricing.

Self-hosted has a different operational shape: it goes down for upgrades and runs out of capacity during bursts. So it always sits in front of an API fallback. In practice about 40% of our classification calls land on the self-hosted model and 60% fall through to the API mini model, and the blended cost per call is roughly 30% of what pure API pricing would be.

Fallback that triggers on the right things

The tricky part of fallback is deciding when. Fail over on provider-side problems, never on your own bad request: a 400 for a malformed prompt fails the same way on every provider, so retrying elsewhere wastes money and time. Each error class gets a different policy:

• Rate limit (429): go straight to the next provider. The limit is real; retrying the same provider just burns time.
• Transient (5xx, 529, timeout): retry the same provider once, then fail over.
• Permanent (other 4xx): don't retry. Surface the error.

RATE_LIMITED = {429}
TRANSIENT = {500, 502, 503, 529}

def complete_with_fallback(route, req, deadline):
    for backend in route.backends[:3]:          # primary + max 2 failovers
        for attempt in range(2):                # max 1 retry per provider
            try:
                return call(backend, req, timeout=min(30, deadline.remaining()))
            except ProviderError as e:
                if e.status in RATE_LIMITED:
                    break                       # next provider immediately
                if e.status not in TRANSIENT:
                    raise                       # our bug, don't fail over
                log.warning("failover", backend=backend, status=e.status)
    raise AllBackendsFailed()

The caps matter as much as the classification: at most two failovers per request, one retry per provider, and a 60-second total budget. Too aggressive and you multiply cost against a degraded provider; too conservative and you fail requests you could have served. After the caps are exhausted, the caller gets the last error and decides whether to degrade, retry later, or tell the user.

We also run a short circuit breaker per provider. After 5 failures in 30 seconds, the gateway skips that provider for a minute and goes straight to fallback, so a struggling provider doesn't add 30-second timeouts to every request. In the earlier design we used a continuous health score instead (successes raise it, 429s, 5xx and slow responses lower it, and a small fraction of probe traffic lets it recover), which handles partial degradation better: a provider that's slow but not failing gets fewer requests rather than zero. Either way, during the next provider blip users saw a slightly slower response instead of an error. That's the entire point.

Cost control you can enforce, not just observe

Dashboards tell you what you already spent. We wanted limits that stop spend before it happens. The gateway tracks per-task and per-tenant token usage in Redis and refuses calls that would blow a cap.

key = f"spend:{tenant}:{today}"
spent = float(redis.get(key) or 0)
if spent + estimate_cost(req) > tenant_daily_cap(tenant):
    raise BudgetExceeded(tenant)
redis.incrbyfloat(key, actual_cost)

This caught a runaway a week later: a batch job stuck in a retry loop that would have spent $1,800 in an afternoon. The cap stopped it at $200 and paged us instead.

Parity is a production problem, not an eval problem

Providers don't produce identical output. A prompt tuned on Claude can behave oddly on GPT and vice versa, so every model in the routing pool needs two checks. First, an eval set per task, re-run whenever the pool changes. Second, mirrored production traffic: before a new model becomes a fallback, we send a small percentage of real requests to both it and the incumbent, score both, and log the comparison. Twice, a provider that passed the eval set degraded on a real traffic pattern the eval didn't cover, and the mirror caught it before promotion.

Streaming is where the abstraction costs the most code. Each provider's stream has its own chunk shape, so the gateway normalizes every stream to one internal format. When a stream fails mid-response we don't retry, because the partial output has already reached the user; we close cleanly and let the caller decide, the same partial-result handling we describe in LLM streaming UX: backpressure, cancellation, partial results.

The abstraction tax, and what we watch

A gateway is not free. You give up provider-specific features, prompt caching semantics differ, some providers have structured-output modes others don't, and the normalized request either finds a lowest common denominator or carries per-provider escape hatches. We kept a provider_options passthrough for the cases that genuinely need it and accepted that the fallback for those tasks may behave slightly differently.

There's also one extra network hop (about 8ms in our setup) and a new single point of failure. We run the gateway as three stateless replicas behind a load balancer so it isn't what takes everything down next time.

What we monitor: per-provider success rate per task, failover rate (spikes mean upstream trouble), per-task cost trend, per-task p95 and p99 latency, and error-budget burn per provider, treating each provider like a dependency with its own SLO.

The decision, concretely

• More than two or three services calling LLMs? Build the gateway; hardcoded providers make every outage a code change.
• One script calling one model? Don't. You're adding a moving part for nothing.
• High-volume classification or extraction? Route it to a mini model or self-hosted, with an API fallback, after proving parity.
• Adding a new provider? Mirror real traffic before making it a fallback, and only promote it to primary after that.

The call we'd make

Even one fallback provider is a large resilience improvement over none, so start there: a task-labelled gateway with a primary and a fallback per task, error-class-aware failover with hard caps, and enforced spend limits. The routing and cost controls end up mattering more day to day than the failover, but it's the failover that justifies the project after your first provider outage.