Once you call more than one LLM provider, a gateway saves you from reinventing routing, fallback, caching, and spend limits in every service.
AI Gateway Comparison — Portkey, LiteLLM, Cloudflare, and More
The first time you integrate an LLM, you call the provider SDK directly and ship it. The trouble starts around the third provider. Now you have OpenAI in one service, Anthropic in another, a self-hosted model behind vLLM for the cheap batch jobs, and each one has its own SDK, auth scheme, retry logic, and billing dashboard. An AI gateway is the piece that collapses all of that into one endpoint.
At its core, a gateway presents a single OpenAI-compatible endpoint in front of many providers. Your code talks to one URL with one request shape, and the gateway translates to whatever backend you route to. That translation layer is the foundation, but it is not why teams adopt one. The reasons are the features stacked on top:
- Routing and load balancing: Split traffic across providers or across multiple keys on the same provider to spread rate limits and cost.
- Fallback and retries: When GPT-4 times out or returns a 529, retry on a backup deployment or a different provider without your app noticing.
- Caching: Return a stored response for identical (or semantically similar) prompts and skip the API call entirely.
- Rate limiting: Cap requests per key, per user, or per team so one runaway loop does not exhaust your quota.
- Spend and budget controls: Hard limits per virtual key, so a misbehaving job stops at $50 instead of $5,000.
- Key management: Issue virtual keys to teams and rotate real provider keys behind them without redeploying anything.
- Observability: One place to see latency, token counts, error rates, and cost per model, per user, per request.
Teams add a gateway when the alternative is copy-pasting the same resilience and accounting code into every service that touches an LLM. Centralizing it means one team owns provider onboarding, one dashboard answers "why did the bill jump," and product engineers get a stable endpoint that does not change when you swap models underneath. This is the same problem space covered in multi-provider LLM routing and failover, viewed from the infrastructure side.
LiteLLM is the open-source default. It started as a Python library that normalized 100-plus providers to the OpenAI format, and grew a proxy server that runs as a standalone gateway. Provider coverage is its headline strength: OpenAI, Anthropic, Bedrock, Vertex, Azure, Cohere, Mistral, local Ollama, and a long tail of everything else, all through the same call.
You self-host it, which means you own the deployment, the Postgres it wants for keys and logs, and the uptime. In exchange you get full control and no per-request markup. A minimal proxy config looks like this:
# config.yaml
model_list:
- model_name: gpt-4o
litellm_params:
model: openai/gpt-4o
api_key: os.environ/OPENAI_API_KEY
- model_name: claude-sonnet
litellm_params:
model: anthropic/claude-3-7-sonnet-latest
api_key: os.environ/ANTHROPIC_API_KEY
router_settings:
fallbacks:
- gpt-4o: ["claude-sonnet"] # if gpt-4o fails, retry on claude
litellm_settings:
cache: true
max_budget: 200 # USD/month across the proxy
Start it with litellm --config config.yaml, then point any OpenAI SDK at it:
from openai import OpenAI
client = OpenAI(base_url="http://localhost:4000", api_key="sk-my-virtual-key")
resp = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "Summarize this incident report."}],
)
print(resp.choices[0].message.content)
If gpt-4o errors, the request lands on claude-sonnet automatically. LiteLLM is the right pick when you want broad provider support, no vendor lock, and a team comfortable running one more service.
Portkey is the managed counterpart. You get the same OpenAI-compatible gateway idea, but hosted, with observability, guardrails, and prompt management built into the product rather than bolted on. Configs are expressed as routing objects (weighted load balancing, conditional fallback, retry counts) that you version in their UI or ship as JSON. The guardrails layer can check inputs and outputs for PII, JSON validity, or policy violations before a response reaches your user.
The trade is the usual managed one: less to operate, a per-request cost, and your traffic transiting their infrastructure (they also offer an open-source gateway you can self-host if that is a blocker). Portkey fits teams that want the resilience and analytics without staffing an internal platform team to run it.
Cloudflare comes at this from the edge. AI Gateway is provider-agnostic and focuses on caching, rate limiting, and analytics rather than deep routing logic. You prefix your existing provider URL with a Cloudflare endpoint, keep using each provider's native SDK, and get caching and a request log for near-zero integration effort. Because it runs on Cloudflare's edge, cached hits are fast and global.
It is the lightest touch of the three. You do not get LiteLLM's translation layer or Portkey's guardrails, but if what you want is a cache, spend visibility, and a kill switch in front of providers you already call, it is hard to beat on setup time, especially if you are already a Cloudflare customer.
Kong AI Gateway: If you already run Kong for your regular API traffic, its AI plugins add LLM routing, prompt templating, token-based rate limiting, and semantic caching to the same gateway. The pitch is one control plane for all traffic, API and AI alike. It makes most sense where Kong is already the standard.
Cloud-native: AWS Bedrock, Azure AI Foundry, and Google Vertex each act as a gateway within their own walls. You get routing, quotas, and billing rolled into the platform, with the obvious limit that they mostly front their own hosted models. Good if you are all-in on one cloud, weaker if you need genuine multi-vendor flexibility.
Self-hosting (LiteLLM, Kong, Portkey OSS) keeps data on your infrastructure, avoids per-request fees, and gives you total control, at the cost of running and scaling the thing. Managed (Portkey, Cloudflare) trades a fee and a network hop for someone else's uptime.
On latency: a gateway adds a hop, but a well-run one adds single-digit to low-tens of milliseconds, which is noise next to a multi-second LLM generation. The exception is caching, which makes the gateway a net negative on latency because a cache hit skips the model entirely. Do not let latency fear drive this decision; the model call dominates.
- You call many providers and want no lock-in: LiteLLM, self-hosted.
- You want managed resilience plus guardrails and analytics: Portkey.
- You mostly want caching and spend visibility with minimal setup: Cloudflare AI Gateway.
- You already run Kong: Kong AI Gateway.
- You live inside one cloud: the native gateway (Bedrock, Vertex, Azure).
For most teams past the two-provider mark, start with LiteLLM self-hosted. It is free, covers every provider you are likely to touch, and the OpenAI-compatible endpoint means you can rip it out later with almost no code change. Put Cloudflare AI Gateway in front of it if you want an edge cache and a second layer of spend analytics for near-zero effort. Reach for Portkey when guardrails, prompt governance, and not operating the gateway yourself are worth the per-request cost, which is usually the point where an AI feature becomes a core product rather than an experiment.
Whatever you choose, put the gateway in early. Retrofitting one after five services each grew their own retry logic is far more painful than adopting it while you still have two. For the wider picture on providers and tooling, see our guide to the best LLM APIs and AI infrastructure.
