S
ix months after shipping your first AI feature, you have LLM calls in ten different places. The customer support flow calls OpenAI directly. The document summariser has its own API key. The internal search uses a different model. Nobody knows the total monthly cost until the AWS bill arrives, and by then it's too late to care why.This is what happens when AI capabilities get added feature-by-feature without any shared infrastructure layer. It's not a code quality problem. It's a structural one. The LLM gateway pattern is the fix and the right time to put it in is before the mess is already expensive.
What the LLM gateway pattern is
An LLM gateway is a single internal service that every part of your product calls to interact with language models. No team makes direct API calls to OpenAI, Anthropic, or any other provider. Everything goes through the gateway. The gateway handles the provider call, logs the request and response, applies rate limits, tracks costs by feature and team, and returns the result.
It sounds like added complexity. It's actually the opposite. Without a gateway, every team manages retries, error handling, token counting, and API key rotation independently. The same logic gets written four times. The same mistakes get made four times. With a gateway, you solve each problem once.
The pattern is directly comparable to how mature engineering teams handle other shared concerns. Authentication middleware sits in one place. Database connection pooling sits in one place. Logging configuration sits in one place. The LLM gateway is the same idea applied to AI model calls and if you've read anything about [→ avoiding premature service proliferation], this is precisely the kind of cross-cutting concern that belongs centralised rather than distributed.
What the gateway should actually do
A minimal LLM gateway has four responsibilities. Everything else is optional until you have a concrete reason to add it.
Provider abstraction. The gateway accepts a standard request format and translates it to whatever provider API it needs to call. Teams calling the gateway don't care whether the response came from GPT-4o or Claude Sonnet or Gemini. When you want to swap providers, change models, or run A/B tests between models, you do it in one place. No feature team rewrites a single line.
Observability. Every call gets logged which feature triggered it, how many input and output tokens were used, what the latency was, whether it succeeded or failed. Aggregated across a day, this is your AI cost dashboard broken down by feature. Without it, you cannot tell which feature is burning your budget or which prompt is slower than it should be. This data is also what tells you when a prompt change degraded performance not a user complaint.
Rate limiting and cost controls. Per-feature token budgets. Soft limits that trigger an alert. Hard limits that stop a runaway process from spending $4,000 in an afternoon. These can be simple to start a Redis counter with a daily cap per feature flag. They don't need to be clever. They need to exist before the incident that would have required them.
Retry and fallback logic. LLM providers have downtime. They rate limit at the API level. A gateway that handles retries with exponential backoff, and optionally fails over to a secondary provider, means your product degrades gracefully rather than failing completely. Each feature team doesn't reimplement this the gateway does it once.
How to build one without over-engineering it
Start with an HTTP service, not a platform.
Your first version needs three endpoints: one to send a completion request, one to get your cost dashboard, one to check rate limit status. That's it. Build it in whatever language your team is most productive in. Put it behind your internal load balancer. Give every team an API key scoped to their feature. Log everything to your existing observability stack. This can be production-ready in a week.
The temptation is to build the full platform immediately model registry, prompt versioning, evaluation pipelines, semantic caching, fine-tuned routing. Resist it. Build what you know you need. Operate it in production. Add capabilities when you have evidence they're worth the investment.
Semantic caching is worth mentioning because it has surprisingly fast payback. If many of your users ask similar questions, caching by embedding similarity rather than exact string match can cut LLM call volume by 20–40% on user-facing features. This is a second-phase addition. Know it exists, don't build it week one.
The gateway pattern scales well because the service itself is stateless. Horizontal scaling means running more instances behind the load balancer. State token counts, rate limit windows lives in Redis. The gateway instances read and write Redis. This is the same architecture your existing API uses, which means your ops team already knows how to run it.
The part teams consistently get wrong
They treat the gateway as optional once costs are visible.
The right order is: build the gateway before the second AI feature ships. Not after you've noticed the bill. By the time costs are visible enough to cause concern, you have multiple services making direct provider calls, multiple API keys in multiple places, and a refactor that touches everything. The migration cost is real. Some teams never complete it, and instead build the gateway as a new-feature-only requirement which leaves them running two systems indefinitely.
The second mistake: not enforcing the gateway.
An LLM gateway only works if it's the only path. One rogue direct integration breaks cost attribution and removes observability for that feature. Enforce it at the API key level. Production LLM provider credentials should exist in exactly one place: the gateway's environment. Feature teams get internal gateway credentials. They don't get provider credentials. This isn't bureaucracy it's the same reason your production database password isn't in five different config files.
The third mistake is conflating the gateway with a prompt management system. Prompt templates belong in your codebase, versioned like code, deployed like code. A gateway that tries to also be a prompt CMS ends up doing neither well. Keep concerns separate. The gateway handles transport, cost, and reliability. Prompt content is a product decision managed by product teams.
Real-world example
A fintech startup building automated financial document analysis. Three AI features in production: document classification, data extraction, and a Q&A interface for their compliance team. Each feature built independently, each calling OpenAI directly with its own key. Total monthly spend: visible only in the OpenAI dashboard, not broken out by feature.
When the document extraction feature had a prompt regression a developer changed how tables were described and it caused the model to become verbose token usage doubled overnight. Nobody noticed for four days, because there was no alerting. The $3,000 overrun that month was the forcing function.
They built a gateway service in Go over two weeks. All three features migrated to use it. Within a month they had: per-feature cost dashboards, a token budget per call that prevented runaway prompts, automatic failover to Claude when OpenAI returned 529 errors during their Lebaran traffic peak, and a semantic cache that cut their Q&A feature's LLM call volume by 31%.
Total saved in the first quarter after implementation: roughly $8,000 mostly from caching and the prompt regression catching a second incident early. The gateway became infrastructure their team was actively grateful to have.
FAQ
Q: Is an LLM gateway the same as a tool like LangChain or LlamaIndex? A: No. LangChain and LlamaIndex are application frameworks they help you orchestrate chains of LLM calls, retrieval, and tool use. An LLM gateway is infrastructure that sits beneath those frameworks. You can use LangChain inside your product and still route all its model calls through your internal gateway for observability and cost control.
Q: Can I use an off-the-shelf open source gateway instead of building one? A: Yes. LiteLLM and Portkey are two open source options that provide provider abstraction, basic logging, and routing. They're reasonable starting points. The tradeoff is that shared infrastructure critical to your AI cost control is code you don't own and didn't write. Evaluate whether the maintenance surface is worth the build time you save.
Q: How do I attribute costs to features when multiple AI calls happen in a single user request?
A: Pass a feature identifier in every request to the gateway a header or request field like feature: "document-extraction". The gateway logs it against every token count. This makes cost dashboards accurate at the feature level without any complex tracing setup.
Q: Does adding a gateway introduce latency? A: A same-region internal HTTP call to a gateway adds 1–3ms. Against LLM call latency of 500ms–3,000ms, this is noise. It doesn't affect user-facing performance.
Q: When is the gateway not worth building? A: If you have a single AI feature with low call volume and no plans to add more, the overhead may not be justified yet. As soon as you're building a second AI feature, the gateway pays back within weeks.
The right time to build this was before your last AI feature shipped. The second-best time is now, before the next one. If your AI spend is already opaque or your teams are managing provider credentials directly, that's the signal to move. SpectreDev has built this layer for teams where the gateway became one of the highest-leverage infrastructure decisions of the year.