Multi-Provider LLM Gateways: Routing, Fallback, and Cost Control

Multi-Provider LLM Gateways: Routing, Fallback, and Cost Control#

On a Tuesday afternoon our primary model provider had a partial outage, roughly 40 minutes of elevated 529s and timeouts. Every AI feature in the product, summarization, the support bot, the classification pipeline, failed at once because they all called the same SDK pointed at the same endpoint. We had no way to shift traffic because the provider was hardcoded in a dozen services.

That incident bought the budget for an LLM gateway: a single internal service every app calls, which then decides which provider and model to actually hit.

One internal endpoint, many backends#

The first win is boring but important. Every service stops importing a provider SDK and instead calls our gateway with a normalized request. The gateway owns keys, retries, routing, and logging. Apps get dumber, which is the goal.

# what every service now calls
resp = gateway.complete(
    task="support_reply",       # logical task, not a model name
    messages=messages,
    max_tokens=800,
)

Note the app asks for a task, not a model. That indirection is what lets us reroute without touching a single service. The mapping from task to model lives in gateway config.

Routing by task, not one model for everything#

Different tasks have different needs, and paying flagship prices for a job a cheap model handles fine is pure waste. We route classification and short extraction to a small fast model, and reserve the expensive model for open-ended generation.

routes:
  classify_ticket:
    primary:  { provider: openai,    model: gpt-4o-mini }
    fallback: { provider: anthropic, model: claude-haiku }
    max_cost_per_1k_input: 0.20
  support_reply:
    primary:  { provider: anthropic, model: claude-sonnet }
    fallback: { provider: openai,    model: gpt-4o }
  legal_summary:
    primary:  { provider: anthropic, model: claude-opus }
    fallback: { provider: openai,    model: gpt-4o }

Moving classification off the flagship model was where the 30% savings came from. That one task was 55% of our call volume and none of it needed a top-tier model. We proved parity on a labeled eval set first, then flipped it.

Fallback that actually triggers on the right things#

The tricky part of fallback is deciding when. You want to fail over on provider-side problems (5xx, 429 rate limits, timeouts) but not on your own bad request (a 400 for a malformed prompt fails the same way on every provider, so retrying elsewhere just wastes money and time).

RETRYABLE = {429, 500, 502, 503, 529}

def complete_with_fallback(route, req):
    for backend in [route.primary, route.fallback]:
        try:
            return call(backend, req, timeout=30)
        except ProviderError as e:
            if e.status not in RETRYABLE:
                raise                      # our bug, don't fail over
            log.warning("failover", backend=backend, status=e.status)
            continue
    raise AllBackendsFailed()

We also added a short circuit breaker per provider. After 5 failures in 30 seconds, the gateway stops trying that provider for a minute and goes straight to fallback, so a struggling provider doesn't add 30-second timeouts to every request. During the next provider blip, users saw a slightly slower response instead of an error. That's the entire point.

Cost control you can enforce, not just observe#

Dashboards tell you what you already spent. We wanted limits that stop spend before it happens. The gateway tracks per-task and per-tenant token usage in Redis and refuses calls that would blow a cap.

key = f"spend:{tenant}:{today}"
spent = float(redis.get(key) or 0)
if spent + estimate_cost(req) > tenant_daily_cap(tenant):
    raise BudgetExceeded(tenant)
redis.incrbyfloat(key, actual_cost)

This caught a runaway one week later, a batch job stuck in a retry loop that would have spent $1,800 in an afternoon. The cap stopped it at $200 and paged us instead.

The abstraction tax#

A gateway is not free. You give up provider-specific features, prompt caching semantics differ between providers, some have structured-output modes others don't, and your normalized request has to find a lowest common denominator or carry per-provider escape hatches. We kept a provider_options passthrough for the cases that genuinely need it, and accepted that our fallback for those tasks might behave slightly differently.

There's also a latency cost, one extra network hop, about 8ms in our setup, and a new single point of failure. We run the gateway as three stateless replicas behind a load balancer so it isn't the thing that takes everything down next time.

The call we'd make#

Build the gateway if you have more than two or three services calling LLMs, or if a provider outage would take down something users notice. If you've got one script calling one model, don't, you're adding a moving part for no reason. The routing and cost controls end up mattering more day to day than the failover, but it's the failover that justifies the project after your first provider outage.