Best LLM Gateways in 2026 (Compared by Use Case)

An LLM gateway is one endpoint that sits between your applications and the model providers they call. Instead of every service talking to OpenAI, Anthropic and Google directly, they all talk to the gateway, and the gateway forwards each request to the right place. Because every call passes through one point, that point is where you can add failover, apply policy, capture usage, and attribute cost.

The category has grown quickly enough that "best LLM gateway" is now the wrong question. The tools below do not compete on one axis. They split into four jobs, and the right choice depends on which job you are hiring for:

Routing and resilience. Fallbacks between providers, load balancing, retries, caching, rate limits.
Aggregation. One key and one bill for hundreds of models, so you never open a provider account.
Platform-native. A gateway bundled with infrastructure you already run.
Cost attribution and governance. Per-request cost by team, feature and customer, reconciled to what providers bill, with policy on which models may be used.

Most teams need more than one of these, and gateways chain, so the honest answer is often two tools rather than one. Here is each of the main options, what it is for, and what to watch.

1. LiteLLM

Job: routing and resilience, self-hosted.

LiteLLM is the open-source default. It exposes one OpenAI-compatible endpoint in front of over 140 providers by its own count, with model fallbacks, load balancing across deployments, retries and semantic caching. It issues virtual keys, tracks spend per key, user, team and organisation, and enforces hard budgets that stop requests once a cap is hit. Single sign-on, role-based access, audit logs and tag-based spend reports sit behind an Enterprise licence.

Watch for: it is infrastructure you operate. A gateway container, a Postgres database for keys and budgets, and Redis once you run more than one instance. Budget state lives in that database, which puts the database on the critical path of every model call. Prompt logging is configurable, so decide deliberately whether content is stored.

Best for: platform teams who want control of the request path and are happy to run it.

2. Portkey

Job: routing and resilience, with observability, hosted or self-hosted.

Portkey packages routing (fallbacks, load balancing, conditional routing), guardrails, a prompt library and request logging into one gateway, available as a hosted service or a self-hosted deployment. It covers a very large model catalogue and integrates with the common agent frameworks.

Watch for: requests are logged by default, so like the observability tools it holds prompt and response content unless you configure otherwise. Cost reporting is organised around engineering views of the traffic. Self-hosting is available, but the managed tier is where most of the product experience lives, and it is priced accordingly.

Best for: engineering teams who want routing and observability from one vendor without assembling it from parts.

3. OpenRouter

Job: aggregation.

OpenRouter is a marketplace rather than a proxy in front of your own accounts. One API key gives access to hundreds of models from dozens of providers, billed through OpenRouter, with automatic fallback between the endpoints serving a given model. The cost of each request is returned in the response, which makes it unusually easy to know what a call cost the moment it finishes.

Watch for: you are buying through an intermediary, so the provider relationship, data-processing terms and pricing are OpenRouter's rather than the model creator's. The same model can be served by several endpoints at different prices, and which one you get depends on routing. Attribution to your own teams and features is not what it is for.

Best for: breadth of model access with minimum account admin, especially for experimentation and for products that switch models often.

4. Vercel AI Gateway

Job: platform-native, for Vercel AI SDK users.

Vercel's gateway gives AI SDK applications one endpoint and one bill across the major providers, with the cost of each request reported in the response at the provider's list price. You can bring your own provider keys or pay through Vercel.

Watch for: it is most natural if your application is already built on the Vercel AI SDK. Cost is reported per request, but rolling that up into cost by customer or cost centre across the rest of your estate is left to you. If you bring your own keys, check how cost is reported for those calls, because a zero cost figure is not the same as a missing one.

Best for: teams standardised on the Vercel AI SDK who want a gateway they do not have to think about.

5. Cloudflare AI Gateway

Job: platform-native, on Cloudflare.

Cloudflare's gateway adds caching, rate limiting, retries, logging and analytics in front of the major providers, and it runs on the same edge network as the rest of Cloudflare's products. If your application already fronts through Cloudflare, it is a small step to route model traffic through it too.

Watch for: the analytics are traffic analytics. You get request counts, token counts, cache hit rates and errors per gateway, which is useful operationally and thin for finance. Attribution to business dimensions is not the design goal.

Best for: teams already on Cloudflare who want caching and rate limiting at the edge with minimal setup.

6. Kong AI Gateway

Job: platform-native, for organisations already running Kong.

Kong extended its API gateway with AI plugins: a unified interface to multiple providers, prompt guardrails, rate limiting by token, semantic caching, and the same policy and observability tooling Kong customers already use for their other APIs.

Watch for: it assumes you have, or want, Kong. For an organisation with a platform team and an existing API management estate that is a strength. For a product team who just needs a base URL it is a large surface area.

Best for: enterprises with an established API gateway practice who want LLM traffic governed the same way.

7. Helicone AI Gateway

Job: observability with a proxy, now in maintenance mode.

Helicone made the one-line base URL proxy popular and built prompt-level observability on top of it: request logs with full content, sessions, custom properties for tagging, and a self-hosted Rust gateway. In March 2026 it was acquired by Mintlify and moved into maintenance mode, with security patches, bug fixes and new model support continuing but no new features.

Watch for: the roadmap. Existing users are not in immediate trouble, but Mintlify has said it will help customers migrate elsewhere, which is a clear steer. It stores prompts and responses by design. We have written a fuller comparison with LiteLLM and a migration guide.

Best for: nobody starting fresh. Existing users should plan a move.

8. AI SpendOps

Job: cost attribution and governance, managed.

AI SpendOps is the gateway on this list built around the cost question rather than the routing one. It runs as a managed service on Cloudflare's edge, so there is nothing to deploy. You point your existing OpenAI, Anthropic or other SDK at it with a base URL change and keep your own provider keys. Every request has its prompt, completion, cached, cache-write and reasoning tokens captured separately and priced against the provider's own rate card, with the provider-specific cache accounting handled so the totals reconcile to the invoice. Requests carry the dimensions you choose (team, feature, environment, customer), and untagged requests can be rejected at the gateway so the allocation is complete. Model and provider allow-lists are enforced at call time. Budgets are tracked and forecast with burn alerts. Prompt and response content is never stored.

Watch for: it is not a routing gateway. Requests pass through unchanged, so there are no fallbacks, no load balancing and no semantic caching. If you need those, run a routing gateway as well. It covers 16+ providers directly, and several of those are aggregators, so the reachable model set is large, but it is not the 140-plus of LiteLLM. It is a paid service, from £29 a month with the first three months free.

Best for: finance and FinOps teams who need the numbers, and engineering teams who would rather not be the ones producing them.

Side by side

Gateway	Job	Hosting	Routing logic	Stores prompts	Cost attribution
LiteLLM	Routing	Self-hosted	Extensive	Configurable	Per key, team and tag; hard budgets
Portkey	Routing plus observability	Hosted or self-hosted	Extensive	Yes by default	Engineering views of spend
OpenRouter	Aggregation	Hosted marketplace	Between endpoints	Per policy	Cost per request in the response
Vercel AI Gateway	Platform-native	Hosted	Basic	Per policy	Cost per request in the response
Cloudflare AI Gateway	Platform-native	Hosted	Caching and retries	Optional logging	Traffic analytics per gateway
Kong AI Gateway	Platform-native	Self-hosted or Konnect	Plugins	Per policy	API-management style metrics
Helicone	Observability, maintenance mode	Hosted or self-hosted	Limited	Yes by design	Custom properties
AI SpendOps	Cost attribution and governance	Managed edge	Pass-through only	Never	Team, feature, environment, customer; reconciled to the invoice

How to choose

Ask four questions, in order.

Who has to read the output? If the answer is engineers, most of the list will do. If the answer includes finance, the gateway needs to produce cost by business dimension with the token accounting done correctly per provider, and that narrows the field to one or two.

What happens when a provider is down? If the answer must be "we fail over automatically", you need a routing gateway. If the answer is "we page someone", you may not.

Can the gateway store your prompts? For consumer or regulated data, this decides the procurement timeline. A gateway that never sees content has nothing to review.

Who runs it? Self-hosting is control and cost. A managed gateway is neither, and that is the point.

If the answers pull in two directions, that is expected. A routing gateway behind a cost gateway is a normal architecture: the finance layer sees every request and produces the numbers; the routing layer decides where each request goes. What does not work is choosing a routing tool and expecting the finance report to fall out of it.

Sources

LiteLLM documentation: provider count, routing features, virtual keys, budgets and Enterprise scope.
Portkey documentation: gateway routing, guardrails, logging and deployment options.
OpenRouter documentation: model routing, provider endpoints and per-request cost in responses.
Vercel AI Gateway documentation: supported providers, bring-your-own-key and cost reporting.
Cloudflare AI Gateway documentation: caching, rate limiting, logging and analytics.
Kong AI Gateway: AI plugins for the Kong gateway.
Mintlify, "Mintlify acquires Helicone": the March 2026 acquisition and maintenance-mode commitment.

The Best LLM Gateways in 2026, Sorted by the Job You Need Done