AI Gateway Comparison: Portkey, LiteLLM, Cloudflare, and More

Our default LLM gateway is self-hosted LiteLLM: free, OpenAI-compatible, and wide enough to cover every provider you will touch. Portkey wins when you want guardrails and analytics without running anything, and Cloudflare AI Gateway wins when a free cache and rate limiter in front of existing provider calls is all you need.

What an AI gateway actually does

A gateway presents one endpoint in front of many providers. Your code talks to one URL with one request shape, and the gateway translates to whatever backend you route to. The translation is the foundation; the features on top are why teams adopt one:

• Routing and fallback: split traffic across providers or keys, and retry on a backup model when the primary times out or returns a 529.
• Caching: return a stored response for identical prompts and skip the call.
• Rate limits and budgets: cap requests or spend per key, user, or team, so a runaway job stops at $50 instead of $5,000.
• Key management: issue virtual keys and rotate real provider keys behind them without redeploying.
• Observability: one place for latency, tokens, errors, and cost per model and per user.

The alternative is copy-pasting the same retry and accounting code into every service that touches an LLM. This is the same problem covered in multi-provider LLM routing and failover, viewed from the infrastructure side.

The options side by side

LiteLLM

LiteLLM started as a Python library that normalized 100-plus providers to the OpenAI format and grew a proxy that runs as a standalone gateway: OpenAI, Anthropic, Bedrock, Vertex, Azure, Mistral, local Ollama, and a long tail behind the same call. You own the deployment, the Postgres it wants for keys and logs, and the uptime. In exchange there is no per-request markup. A minimal proxy config:

# config.yaml
model_list:
  - model_name: gpt-main
    litellm_params:
      model: openai/gpt-6-astra
      api_key: os.environ/OPENAI_API_KEY
  - model_name: claude-sonnet
    litellm_params:
      model: anthropic/claude-sonnet-5-5
      api_key: os.environ/ANTHROPIC_API_KEY

router_settings:
  num_retries: 3
  fallbacks: [{"gpt-main": ["claude-sonnet"]}]   # if gpt-main fails, retry on claude

litellm_settings:
  cache: true
  max_budget: 200                 # USD across the proxy

Start it with litellm --config config.yaml, then point any OpenAI SDK at it:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:4000", api_key="sk-my-virtual-key")

resp = client.chat.completions.create(
    model="gpt-main",
    messages=[{"role": "user", "content": "Summarize this incident report."}],
)
print(resp.choices[0].message.content)

After three failed retries on gpt-main, the request lands on claude-sonnet automatically.

Portkey

Portkey is the managed counterpart: the same OpenAI-compatible gateway idea, hosted, with observability, guardrails, and prompt management built into the product. Routing configs (weighted load balancing, conditional fallback, retry counts) are JSON objects you version in their UI. Guardrails check inputs and outputs for PII, JSON validity, or policy violations before a response reaches your user. The trade is the usual managed one: less to operate, a per-request cost, and traffic transiting their infrastructure, though the open-source gateway can be self-hosted if that is a blocker.

Cloudflare AI Gateway

Cloudflare comes at this from the edge. You route your existing provider calls through a Cloudflare gateway endpoint, keep each provider's native SDK, and get caching, rate limiting, request retries, model fallback, and a request log. It is the lightest touch here: no LiteLLM-style translation layer, no Portkey-style guardrails on the free core, but hard to beat on setup time if you are already a Cloudflare customer.

Kong, Vercel, and the clouds

Kong AI Gateway adds semantic caching, token-based rate limiting, and provider load balancing to the gateway you may already run for API traffic. It makes sense where Kong is already the standard. Vercel AI Gateway is a managed option that works from any infrastructure, not only Vercel, with budgets per team, project, or key. Bedrock, Azure AI Foundry, and Vertex each act as a gateway inside their own walls; good if you are all-in on one cloud, weaker for genuine multi-vendor flexibility.

On latency: a gateway adds a network hop, which is small next to a multi-second generation, and a cache hit skips the model entirely. Don't let latency fear drive the choice.

Questions people ask

What is the best LLM gateway?

For most teams it is LiteLLM, self-hosted: MIT-licensed, OpenAI-compatible, and able to front 100+ providers with retries, fallbacks, caching, and per-key budgets at no per-request cost. The catch is that you operate it, including its Postgres. Teams that won't run another service should pick a managed gateway such as Portkey or Vercel AI Gateway instead.

What is a good Portkey alternative?

If you want to self-host, LiteLLM covers the same routing, fallback, and budget features with no license fee. If you want managed and cheap, Vercel AI Gateway charges zero markup on token prices, and OpenRouter passes provider prices through with a 5.5% fee on card credit purchases. Portkey's own open-source gateway is also an option when the hosted plan is the objection.

Is there a Cloudflare AI Gateway alternative with deeper routing?

Yes. Cloudflare's free core covers caching, rate limiting, retries, and model fallback, but not a translation layer or guardrails. LiteLLM adds one OpenAI-format API across providers plus virtual keys and team budgets; Portkey adds guardrails and prompt management. Many teams run LiteLLM behind Cloudflare to get both the edge cache and the routing.

The decision, concretely

• Calling many providers and refusing lock-in? LiteLLM, self-hosted.
• Want managed resilience plus guardrails? Portkey.
• Mostly want caching and spend visibility for near-zero effort? Cloudflare AI Gateway.
• Already run Kong, or live inside one cloud? Use what you have: Kong AI Gateway, or Bedrock, Vertex, or Azure.

The call we'd make

Past the two-provider mark, start with LiteLLM self-hosted. The OpenAI-compatible endpoint means you can rip it out later with almost no code change. Add Cloudflare AI Gateway in front if you want an edge cache for free, and move to Portkey when guardrails and not operating the gateway are worth the monthly fee. Whatever you choose, add the gateway early; retrofitting one after five services grew their own retry logic is far more painful. For the provider side, see our guide to the best LLM APIs and AI infrastructure.

Sources

• LiteLLM documentation
• Portkey pricing
• Cloudflare AI Gateway pricing
• Vercel AI Gateway documentation