What is an AI Gateway? The View from Trillions of Production Tokens
We run an AI gateway that routes trillions of tokens for over a thousand companies, so instead of another definition post, this is what the model layer actually looks like when you sit in the middle of it. Every figure below comes from live production traffic in 2026, paid traffic only, BYOK excluded, largest accounts excluded where noted. No synthetic benchmarks anywhere in this post.
The short version: the model layer now moves too fast, breaks too often and is priced too inconsistently to be hardcoded into your application. That is the entire reason AI gateways exist, and the data makes the case better than any vendor diagram.
The definition, in one paragraph
An AI gateway (also called an LLM gateway) is a control plane that sits between your applications or agents and the model providers. Your code calls one OpenAI compatible API. The gateway resolves that call to a concrete model on a concrete provider in a concrete region, applies your policies (allowed models, data residency, spend limits), manages prompt caching, retries and fails over when providers break, and logs cost and latency per key, per team and per model. Where a classic API gateway moves opaque requests, an AI gateway understands tokens, prompts, caches and model identity, because that is where all the money and all the failure modes live.
That is the theory. Here is why it became mandatory.
Reason 1: models expire faster than your sprint cycle
In the week of 3 August 2026, 88% of all tokens on our gateway ran on models we first served less than 6 months ago. Half of all tokens ran on models younger than 90 days.
This is not one whale distorting the average. Of 672 organizations above 5M weekly tokens, the median organization runs 62% of its own traffic on a model younger than 90 days, and 53% of organizations are above half.
Think about what that means for a hardcoded integration. The model name string you ship today is, statistically, legacy infrastructure by Christmas. Teams that wire directly to one provider re do that integration work every quarter. Teams behind a gateway change a routing policy.
The churn is not only new models replacing old ones. It is new providers undercutting each other on the same weights. When moonshot released kimi-k3, it took exactly 11 days for the originating lab to lose the token majority on its own model, as four other hosts lit up inference on the same weights within one 72 hour window and traffic followed price.
Reason 2: the same model has wildly different prices depending on who serves it
Open weights models are served by many competing hosts. Identical weights, identical outputs, very different bills. Here is glm-5.2 across six hosts on our gateway over 30 days, expressed as the effective price customers actually paid as a percentage of list price (effective price folds in each host's real world cache performance):
A 5.7x spread on the same model. Not because of secret discounts, mostly because cache hit rates differ massively between hosts. The cheapest host cached 94% of input tokens, the most expensive cached none.
Zoom out and the aggregate picture is even stranger. Open weights models grew from under 5% of our tokens in January 2026 to half of all tokens in the week of 3 August. But they are only about 13% of spend, because a closed frontier token costs about 6.4x an open one net of caching. The result: 86% of all spend still flows to three vendors (Anthropic 51%, OpenAI 21%, Google 15%) while half the volume has moved elsewhere.
Price discovery across hosts, in real time, per request, is not something an application should be doing. It is a gateway's job.
Reason 3: caching is now most of the bill, and nobody models it
This is the least understood part of LLM economics, and it is where a gateway earns its keep.
First, the anatomy of production traffic. Across 30 days of gateway traffic, cache re reads are 78% of all tokens processed. Fresh input is 19%. Visible output is under 2%. Models write fewer than 3 of every 100 tokens they touch.
Agents are the reason. Agent loops resend the whole conversation on every step, so the read to write ratio on our gateway went from 15x input tokens per output token in January to 36x in August, and average context per request nearly tripled from 9.6k to 28k tokens. If your mental model of LLM pricing is "input price times prompt plus output price times completion", you are modelling a 2024 workload.
The consequence is that effective price falls with agent depth. Here is what requests actually cost as a percentage of list price, by step number within a session:
Step 1 of a session costs 92% of list. By step 51 and beyond, 23% of list, with cache hit rates above 90%. Long running agents pay a quarter of the sticker price. Cold starts pay full freight.
Now the part nobody tells you: caching behavior is wildly inconsistent across providers, and it is undocumented or wrongly documented in several places. From our production traces:
- Anthropic models have a hard 5 minute TTL. We measure hit rates near 70% at a 4 to 5 minute idle gap that collapse to 26% at 5 to 6 minutes and 9% past 6. If your agent pauses to think for 6 minutes, you re bill the entire context at full price.
- OpenAI models decay gradually, LRU style, from about 65% at 5 minutes to near zero past 30 minutes.
- deepseek's cache lives for hours. We still measure ~52% hit rates after a 30 to 60 minute gap.
- Gemini's implicit cache underperforms badly in our traffic, with hit rates around 50% even on immediate followups. Explicit cache management is required to get what you pay for.
- At least one major provider has no prompt caching at all and bills every token at full price, every request.
- Same weights, different hosts, different caching: deepseek-v4-flash cached at 98% on some hosts and near 0% on others in the same week.
And one more that surprised us: switching providers mid session costs real money even when the model stays the same. In session followups that stayed on the same provider hit cache 91% of the time and paid 21% of list. Followups that switched hit 62% and paid 48% of list. Sticky, cache aware session routing is worth 2.2x on its own.
This is why blended cost per token on our gateway fell 57% between December and August while closed model list prices actually went up. Roughly half of the fall is mix shift toward cheaper models. Almost all of the other half is caching. If you take one sentence away from this post: the biggest lever on your AI bill is not model choice, it is cache discipline, and it is invisible unless something in your stack is measuring it.
Reason 4: providers fail constantly, and static failover does not work
Over a recent 30 day window, 3.2% of all requests on the gateway failed with a retryable error (429 rate limits or 5xx). That is one in thirty one requests. At agent loop volumes, every user hits this daily.
Here is the damning number: of the rapid retries that followed those failures within 60 seconds, 99.99% went back to the exact provider that had just failed. Client SDKs retry in place. Static fallback chains are configured once and never react to live conditions. In practice, nobody actually fails over.
They should. In 38% of failures, a healthy alternative for the same model family existed in the same minute, serving at 95%+ success. The rescue rate varies sharply by family:
deepseek failures are 83% rescuable because many hosts serve the same weights. Anthropic capacity tends to saturate across all its hosts at once, so routing alone rescues only 13% there, which is an argument for reserved capacity rather than clever routing. Timeouts are also not created equal: a failed deepseek request burned 154 seconds on average before dying, versus 2.7 seconds for Claude. Failure latency, not just failure rate, is what your users feel.
A gateway that tracks provider health per model per minute and reroutes automatically turns a 3.2% failure rate into roughly 2% instantly, using nothing but traffic that already exists.
Reason 5: governance stopped being a slide and became routing
The other half of the AI gateway story is control. Two production observations.
First, data residency is now a first class routing dimension. On our gateway (which is EU hosted, and popular with European customers, so mind the sample), a majority of frontier model traffic runs pinned to EU regions through policies like claude-opus-5@eu. When Opus 5 launched, 57% of its requests ran EU pinned in the first two weeks. Residency is now something customers configure on day one, not a migration project.
Second, enterprises do not govern models with documentation, they govern with allowlists. An approved models policy at the gateway is the only enforcement point that actually works when developers can otherwise paste any API key into any SDK. The same choke point is where spend limits, per team keys, audit logs and zero data retention routing live.
If you are evaluating this space, this list is the actual test, derived from everything above. Ask any AI gateway (ours included):
- Can it route by policy (model allowlists, region pinning, spend caps), not just by URL rewriting?
- Does it measure and manage caching per provider, including TTL aware session stickiness? This is the biggest cost lever in the stack.
- Does failover react to live provider health, or is it a static chain that retries into the same outage?
- Does it give you per key, per team, per model cost attribution the day you install it?
- Does it handle provider specific quirks (thought signatures, cache_control blocks, region suffixes) so your application code stays provider agnostic?
AI gateway vs API gateway, settled by the data
After all of the above, the difference is easy to state. An API gateway would have treated every request in this post as an opaque HTTP call. It cannot know that 78% of the tokens flowing through it are cache re reads, that the same model costs 5.7x more on one backend than another, that a 6 minute pause just silently multiplied the cost of the next request by 5, or that the retry it is about to forward is going straight back into a rate limited provider while a healthy host serves the same weights next door.
An AI gateway exists because tokens are not packets. The traffic is only legible, and only optimizable, at the layer that understands models, prompts, caches and prices. In 2026, with model half lives under six months, half the volume on open weights served by a dozen competing hosts, and caching quietly determining most of the bill, running LLM traffic without that layer means flying blind through the most volatile infrastructure market in software.
All statistics in this post come from production traffic on the Requesty AI gateway during 2026, most from a 30 day window ending late August 2026. Paid traffic only, BYOK traffic excluded, internal accounts excluded, and the largest accounts excluded where noted to avoid single customer skew. Percentages and ratios only, deliberately. Nothing is synthetic and nothing is a benchmark, this is what real workloads did.
I'm Thibault, CEO of Requesty. If you want to argue with any of these numbers, my DMs are open.