What Is an AI Gateway? When You Need One and How to Choose

Published 2026-11-11

An AI gateway is a service that sits between your application and one or more model providers. Your code sends every request to the gateway in one format, and the gateway forwards it to the right provider, then returns the response. "LLM gateway", "LLM proxy" and "AI API gateway" are mostly different names for the same idea. This article explains what a gateway does, when it is worth having, when it is not, and how to choose. For a side-by-side of specific products, see the LLM gateway comparison.

Disclosure: we build OriginCoder, a hosted AI gateway, so we have an interest in this topic. We have tried to describe the trade-offs fairly, including the cases where you do not need a gateway at all. Everything here is current as of November 2026, and products change, so check each project's own documentation.

What a gateway actually does

Most gateways combine some of these jobs:

• One API for many models. You write against one request format, often the OpenAI Chat Completions format, and switch between Claude, GPT, Gemini or open-weight models by changing a model name.
• Key management. Your application holds one gateway key instead of a key per provider. You can issue separate keys per team or project and revoke them without touching the provider account.
• Usage and cost tracking. Requests are logged with token counts and cost, so you can see who spent what.
• Limits and budgets. Rate limits, spend caps and per-key model allowlists.
• Routing and fallbacks. Retry a failed request on another provider or model, or spread load across several accounts.
• Caching, logging and guardrails. Some gateways cache identical requests, keep request logs, or filter content. This is where products differ most.
• Billing. Hosted gateways that sell tokens themselves also handle top-ups and invoices, so you do not need an account with each provider.

The problems it solves

1. Provider lock-in in your code. Without a gateway, each provider's SDK and request shape leaks into your application. A gateway keeps that behind one interface.
2. Key sprawl. Provider keys copied into many services and laptops are hard to rotate. One gateway key per consumer is easier to track and revoke.
3. No shared view of spend. Several teams, several providers, several dashboards. A gateway gives one record.
4. Outages and rate limits. A single provider can fail or throttle you. Fallback routing reduces the impact.
5. Tools that expect a custom address. Coding agents such as Claude Code or Codex let you set a Base URL. A gateway is what that address points to. See How to use Claude Code with a custom API endpoint.

When you probably need one

• You call more than one provider, or expect to switch models regularly.
• Several people or services share model access and you need per-key budgets or logs.
• You want one invoice, or you cannot easily open accounts with every provider.
• You run coding agents for a team and want usage records per person.
• You need failover between providers for production traffic.

When you probably do not

• You use one model from one provider, and the provider's own dashboard already tells you what you need.
• You are prototyping alone. One provider key in an environment variable is simpler.
• You have strict requirements about where prompts may travel and no gateway you can run or audit. An extra hop adds a party that sees your data.
• Latency is critical and the gateway is far from you or your provider. Every hop adds some delay, usually small, but measure it.
• You only need a retry loop. A few lines in your own code may be enough.

Types of gateway

The boundaries blur: several products offer more than one of these modes. The OpenRouter alternative guide looks at this choice from the aggregator side.

A checklist for choosing

1. Where must your data go? If prompts must stay on your infrastructure, only a self-hosted gateway qualifies. Otherwise ask any hosted operator what they store and for how long.
2. Which request formats do you need? OpenAI-style only, or also Anthropic Messages and Gemini formats, for tools that speak those natively.
3. Whose accounts? Bring your own provider keys, or buy usage from the operator? The second is simpler, the first gives you the provider's own terms and prices.
4. Which models? Check the model list and per-model prices, not just the headline. For OriginCoder see the model pricing page.
5. What controls? Per-key budgets, model allowlists, usage records, audit needs.
6. How is it billed? Pay as you go, subscription, platform fee, or margin on tokens. Work out a typical month.
7. What happens when it fails? Ask about status history, support channel and how a gateway outage affects you. If it is on your critical path, plan a fallback.
8. How hard is it to leave? If you only use the standard request format, switching gateways means changing a Base URL and a key. Avoid features that tie your code to one vendor unless you need them.

Then run a small real workload and compare cost, latency and error rate on your own traffic. That test is worth more than any feature table, including ours.

Costs and risks to keep in mind

• A gateway is another party in the request path, so it can see your prompts and responses. Read its terms and privacy policy.
• Compatibility is not automatic. New models or new provider features can arrive at a gateway later than at the provider.
• Self-hosting trades money for work: deployment, a database, upgrades, monitoring and security are yours.
• OriginCoder is an independent platform and is not affiliated with the model providers whose models it offers.

If you want to try the hosted route with a coding agent, start with the Claude Code setup guide or the Codex setup guide.