Skip to main content
AI Gateway Comparison: Portkey, LiteLLM, Cloudflare, and More

AI Gateway Comparison: Portkey, LiteLLM, Cloudflare, and More

KU
· Founder & Chief Editor
• 8 min read•Updated •158 views

Once you call more than one LLM provider, a gateway saves you from reinventing routing, fallback, caching, and spend limits in every service.

Key takeaways

  • For most teams past two LLM providers, the best AI gateway is self-hosted LiteLLM, which is MIT-licensed, OpenAI-compatible, and fronts 100+ providers with fallbacks and per-key budgets.
  • Choose Portkey for managed guardrails and analytics, from $49 a month as of October 2026, and Cloudflare AI Gateway when free caching and rate limiting are enough.

Our default LLM gateway is self-hosted LiteLLM: free, OpenAI-compatible, and wide enough to cover every provider you will touch. Portkey wins when you want guardrails and analytics without running anything, and Cloudflare AI Gateway wins when a free cache and rate limiter in front of existing provider calls is all you need.

Verified against

LiteLLM v1.106.6, Portkey, Cloudflare AI Gateway, Vercel AI Gateway and OpenRouter, against each vendor's official docs and pricing pages. Checked October 2026.

What an AI gateway actually does

A gateway presents one endpoint in front of many providers. Your code talks to one URL with one request shape, and the gateway translates to whatever backend you route to. The features on top are why teams adopt one:

  • Routing and fallback: split traffic across providers or keys, and retry on a backup model when the primary times out or errors.
  • Caching: return a stored response for identical prompts and skip the call.
  • Rate limits and budgets: cap requests or spend per key, user, or team, so a runaway job stops at $50 instead of $5,000.
  • Key management: issue virtual keys and rotate real provider keys behind them without redeploying.
  • Observability: one place for latency, tokens, errors, and cost per model and per user.

The alternative is copy-pasting the same retry and accounting code into every service that touches an LLM. This is the same problem covered in multi-provider LLM routing and failover, viewed from the infrastructure side.

The options side by side

GatewayHostingFallbacksPricing (as of October 2026)Best for
LiteLLMSelf-hosted, MIT licenseYesFree; you run it and its Postgres100+ providers, no lock-in
PortkeyManaged, or open-source self-hostYesFree to 10k recorded logs/month; $49/month for 100k, then $9 per 100k requestsGuardrails and analytics without ops
Cloudflare AI GatewayManaged, Cloudflare edgeYesAnalytics, caching, rate limiting free; 5% fee on Unified Billing creditsCache and spend visibility, minimal setup
Vercel AI GatewayManaged, callable from any infraProvider and model fallbacksZero markup on token prices, including BYOKBudgets and failover without a proxy to run
OpenRouterManaged aggregatorAutomaticNo inference markup; 5.5% platform fee on credit purchases (Standard)One key and one bill for many models
Kong AI GatewayKonnect control plane, your data planeLoad balancing across providersEnterprise (Konnect)Shops already standardized on Kong

LiteLLM, as a working setup

LiteLLM started as a Python library that normalized 100-plus providers to the OpenAI format and grew a proxy that runs as a standalone gateway. You own the deployment, the Postgres it needs for virtual keys and spend tracking, and the uptime. In exchange there is no per-request markup. A config with two models and a fallback:

YAML
# litellm_config.yaml
model_list:
  - model_name: gpt-main
    litellm_params:
      model: openai/gpt-6-astra
      api_key: os.environ/OPENAI_API_KEY
  - model_name: claude-sonnet
    litellm_params:
      model: anthropic/claude-sonnet-5-5
      api_key: os.environ/ANTHROPIC_API_KEY

router_settings:
  num_retries: 3
  fallbacks: [{"gpt-main": ["claude-sonnet"]}]   # if gpt-main fails, retry on claude

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL

Run it with the official image, then mint a virtual key with a monthly budget:

code
$ docker run -v $(pwd)/litellm_config.yaml:/app/config.yaml \
    -e OPENAI_API_KEY -e ANTHROPIC_API_KEY -e LITELLM_MASTER_KEY -e DATABASE_URL \
    -p 4000:4000 docker.litellm.ai/berriai/litellm:latest --config /app/config.yaml
$ curl http://localhost:4000/key/generate \
    -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H 'Content-Type: application/json' \
    -d '{"max_budget": 10, "budget_duration": "30d"}'

The response contains a key starting with sk-. Point any OpenAI SDK at the proxy with it:

Python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:4000", api_key="sk-my-virtual-key")
resp = client.chat.completions.create(
    model="gpt-main",
    messages=[{"role": "user", "content": "Summarize this incident report."}],
)
print(resp.choices[0].message.content)

After three failed retries on gpt-main, the request lands on claude-sonnet automatically. The master_key must start with sk-; generate one with openssl rand -hex 32.

Portkey

Portkey is the managed counterpart: the same OpenAI-compatible gateway idea, hosted, with observability, guardrails, and prompt management built in. Routing configs (weighted load balancing, conditional fallback, retry counts) are JSON objects you version in their UI. Guardrails check inputs and outputs for PII, JSON validity, or policy violations before a response reaches your user. Note the free Developer plan keeps logs for 3 days and simply stops recording past 10,000 a month; requests still flow. The trade is the usual managed one: less to operate, a usage cost, and traffic transiting their infrastructure, though the open-source gateway can be self-hosted if that is a blocker.

Cloudflare AI Gateway

Cloudflare comes at this from the edge. You route existing provider calls through a gateway endpoint, keep each provider's native SDK, and get caching, rate limiting, retries, model fallback, and a request log. Dashboard analytics, caching and rate limiting are free on all plans. Accounts that created their first gateway on or after September 24, 2026 store logs under Workers Logs pricing, and Unified Billing adds a 5% fee on credit purchases. It is the lightest touch here, and hard to beat on setup time if you already run on Cloudflare.

Kong, Vercel, and the clouds

Kong AI Gateway adds semantic caching, token-based rate limiting, and provider load balancing to the gateway you may already run for API traffic. It makes sense where Kong is already the standard. Vercel AI Gateway works from any infrastructure, not only Vercel, with budgets per team, project, API key, or member, and ordered provider and model fallbacks. Bedrock, Microsoft Foundry, and Vertex each act as a gateway inside their own walls; good if you are all-in on one cloud, weaker for real multi-vendor flexibility.

On latency: a gateway adds a network hop, which is small next to a multi-second generation, and a cache hit skips the model entirely. Don't let latency fear drive the choice.

When the gateway fails: the error strings

  • No connected db. (LiteLLM): a request used a virtual key but the proxy has no Postgres. Set DATABASE_URL, or use the Compose quickstart that bundles one.
  • Budget set but nothing stops spend (LiteLLM): without a database, the docs warn litellm_settings.max_budget is not a spend cap. Run with Postgres or cap spend at the provider.
  • ExceededTokenBudget: Current spend for token: ...; Max Budget for Token: ... or exceeded budget for model=... with HTTP 429: a key or user hit its budget. Raise max_budget or wait for budget_duration to reset.
  • max_budget is over max limit set in config (HTTP 400): you asked for a key budget above upperbound_key_generate_params.
  • Stored provider keys suddenly unreadable (LiteLLM): LITELLM_SALT_KEY changed. Keep it stable from day one.
  • Vercel budget overshoot: budgets are soft caps and only cover system-credential spend; BYOK spend is metered separately.

Where this breaks

A gateway is the wrong layer when you call one provider with one key; the provider SDK's own retries are enough. Self-hosted LiteLLM moves fast (several releases a week as of October 2026), so it is also a patching commitment: read what to patch and how to fence in a LiteLLM proxy before you expose one. And no gateway fixes provider-specific features that don't map to the OpenAI shape; expect some calls to bypass it.

Questions people ask

What is the best LLM gateway?

For most teams it is LiteLLM, self-hosted: MIT-licensed, OpenAI-compatible, and able to front 100+ providers with retries, fallbacks, and per-key budgets at no per-request cost. The catch is that you operate it, including its Postgres, its salt key, and its fast release cadence. Teams that won't run another service should pick a managed gateway such as Portkey or Vercel AI Gateway instead.

What is a good Portkey alternative?

If you want to self-host, LiteLLM covers the same routing, fallback, and budget features with no license fee. If you want managed and cheap, Vercel AI Gateway adds zero markup to token prices, and OpenRouter bills inference at provider list price with a 5.5% platform fee on Standard-plan credit purchases. Portkey's own open-source gateway is also an option when the hosted plan is the objection.

Is there a Cloudflare AI Gateway alternative with deeper routing?

Yes. Cloudflare's free core covers caching, rate limiting, retries, and model fallback, but not a translation layer across request formats. LiteLLM adds one OpenAI-format API across providers plus virtual keys and team budgets; Portkey adds guardrails and prompt management. Many teams run LiteLLM behind Cloudflare to get both the edge cache and the routing.

The decision, concretely

  • Calling many providers and refusing lock-in? LiteLLM, self-hosted.
  • Want managed resilience plus guardrails? Portkey.
  • Mostly want caching and spend visibility for near-zero effort? Cloudflare AI Gateway.
  • Already run Kong, or live inside one cloud? Use what you have: Kong AI Gateway, or Bedrock, Vertex, or Foundry.

The call we'd make

Past the two-provider mark, start with LiteLLM self-hosted. The OpenAI-compatible endpoint means you can rip it out later with almost no code change. Add Cloudflare AI Gateway in front if you want an edge cache for free, and move to Portkey when guardrails and not operating the gateway are worth the monthly fee. Whatever you choose, add the gateway early; retrofitting one after five services grew their own retry logic is far more painful. Pair it with prompt caching at the provider for the biggest cost cut.

Sources

Explore topics:LLMAI
React
Share this post
Cite this article
Plain text

Kiril Urbonas (2026). AI Gateway Comparison: Portkey, LiteLLM, Cloudflare, and More. DevOpsNess. Updated 2026-10-31. https://www.devopsness.com/blog/ai-gateway-comparison

Markdown

[AI Gateway Comparison: Portkey, LiteLLM, Cloudflare, and More](https://www.devopsness.com/blog/ai-gateway-comparison) (Kiril Urbonas, DevOpsNess, updated 2026-10-31)

KU

Kiril Urbonas

Founder & Chief Editor

586 articles
View all articles by Kiril Urbonas

You might have missed

Evergreen posts worth revisiting.