T

This is not a hypothetical. I've seen versions of this three times in the last year alone. LLM api cost optimization isn't something most teams think about until the invoice makes it impossible to ignore. The product works fine. The features shipped. The bills grew in ways nobody planned for.

The mistakes aren't exotic. They're the same five or six problems, made by teams who were moving fast and didn't build cost controls in from the start. Here's what they are and how to fix them before your CFO calls a meeting.

Every token costs money. Your code doesn't know that.

The most common mistake: teams build their LLM integration the same way they built their REST API. Request comes in, call the model, return the response. Clean. Simple. Catastrophically expensive at scale.

You pay for every token in and every token out. That means your system prompt, the user's message, the full conversation history, and the model's response. All of it. Every call.

A system prompt that starts at 200 tokens becomes 600 tokens six months later when someone adds "also make sure to..." three times across three different sprints. Nobody notices because it's just a string in a config file. But at 10,000 calls a day, that's $150/month of pure prompt overhead. Multiply that across a few features and the bill starts making sense in the worst possible way.

The uncomfortable fix: treat tokens like database writes. Audit system prompts quarterly. Strip anything that isn't load-bearing. Set a token budget per call type and enforce it in code.

One pattern worth testing: prompt caching. If your system prompt is identical across 90% of calls, you don't need to pay full price to send it every time. Anthropic and OpenAI both support server-side caching for repeated prompt prefixes. The savings compound fast at volume.

You're using GPT-4 to sort emails

This one produces the biggest immediate savings, and it's the mistake I see most often.

Most LLM use cases don't need the most capable model. They need a model that's good enough and 10–20x cheaper per token. Teams default to the frontier model because that's what they tested with during development. Nobody goes back and asks: does this feature actually need this level of capability?

Break down your LLM calls by task type.

Classification positive/negative/neutral sentiment, spam detection, category assignment a smaller model handles this at 95%+ accuracy. You don't need GPT-4 for this. gpt-4o-mini or claude-haiku at a fraction of the cost.

Extraction pull this field from that document, structure unstructured data again, smaller models. The task is deterministic enough that capability differences matter less than most teams assume.

Generation write a product description, summarise a report, draft a reply this is where frontier models earn their place. Quality differences are real and user-facing.

One Indonesian B2B SaaS team I worked with was running all three task types through the same GPT-4 endpoint. Routing classification to a smaller model cut their OpenAI bill by 40% in month one. Nothing in the product changed. No user noticed.

The cache nobody built

This is the part that hurts the most to explain, because it's so fixable.

LLM calls are expensive. They're also, in many products, surprisingly repetitive. Users ask similar questions. They trigger the same feature over and over with variations on the same input.

Semantic caching storing LLM responses and matching similar future requests against them instead of calling the API again can cut call volume significantly. If you've thought about this for database queries, the same logic applies. [→ Read: Caching Explained: Redis, Memcached, and Why Your Database Is Screaming]. Tools like GPTCache or a custom Redis-based semantic cache with vector similarity matching return a stored response when the new input is close enough to a previous one.

For FAQ-style features, support chatbots, and document summarisation, cache hit rates above 30% are common in production. That's 30% fewer API calls. Not nothing.

The objection I always hear: "our responses need to be fresh." Sometimes that's true. Often it's an assumption nobody has tested. Run the analysis on your actual query logs before ruling it out.

Exact caching is even simpler and even more overlooked. If the same user asks the same question twice, or if your product generates reports on demand and the underlying data hasn't changed, return the cached response. No vector similarity required. A hash of the input and a TTL. Start here before you build anything sophisticated.

The prompt engineering debt nobody pays down

Here's the counter-intuitive one.

Teams spend weeks prompt-engineering their way to a good output. The prompt gets long examples, edge case handling, format instructions, persona framing. It works. They ship it. Nobody touches it again.

Six months later the system prompt is 1,400 tokens because three engineers each added their section "just to handle this edge case." The examples at the bottom are testing artifacts from development. The format instructions repeat themselves twice because nobody read the whole thing before adding to it.

Nobody reviews the system prompt the way they review code, because it doesn't feel like code. But every token in that prompt is money on every call.

The gotcha is subtler than length. Verbose prompts tend to produce verbose responses. A system prompt that says "please make sure to always provide comprehensive, detailed answers that cover all relevant aspects of the topic" will reliably inflate your output token count. A prompt that says "be concise" costs less to send and produces cheaper output.

Prompt debt compounds. Treat it like code. Review it. Version it. Delete things that aren't doing work.

Real-world example: from $38k to $7k in 90 days

A logistics startup in Jakarta about 18 months post-launch had built AI features across three parts of their product: a driver support chatbot, an automated shipment exception handler, and a document extraction pipeline for customs forms.

Monthly LLM spend hit $38,000. They had roughly 8,000 daily active business users. The unit economics were already uncomfortable.

We ran a four-week cost audit. Three changes drove most of the savings.

Model routing. The exception handler and document extraction pipeline were running through GPT-4. We moved them to GPT-3.5-turbo for the initial pass, escalating to GPT-4 only when a confidence threshold wasn't met. 80% of cases never needed the escalation. Token cost dropped 65% for those two features.

Semantic caching on the chatbot. Driver queries clustered around roughly 40 distinct question types. A Redis-based semantic cache returned cached responses for 44% of incoming queries within the first two weeks.

System prompt audit. The chatbot had a 1,100-token system prompt. After removing duplicate instructions, stale examples, and three paragraphs added "just in case," it sat at 380 tokens. Same output quality. No user complaints.

End result: $7,200/month. No features removed. No quality regression users reported. The work took four weeks and two engineers part-time.

FAQ

Q: What's the fastest way to cut LLM costs without changing the product?

A: Run a model routing audit first. Identify which calls genuinely need a frontier model and which are doing something a smaller model handles just as well. That single change typically produces the biggest immediate reduction. Add exact caching for repeated queries second it's an afternoon of engineering for meaningful savings.

Q: Is semantic caching worth building for an early-stage startup?

A: Depends on your query patterns. If your product has any FAQ-style or repetitive queries, even a basic implementation pays for itself quickly. Start with exact caching before building the semantic layer you'll likely get 60–70% of the benefit with 10% of the engineering effort.

Q: How do you decide which model to use for which task?

A: Define the task type first classification, extraction, or generation. Test your specific task on a smaller model with 50–100 real examples from production. If accuracy is within a few percentage points of the larger model, route that task to the cheaper option. Most teams find that the majority of their call volume sits in classification and extraction.

Q: Does streaming reduce token costs?

A: No. Streaming changes how fast the user sees the response, not the token count. You pay the same either way. The indirect benefit is that faster perceived response time reduces user abandonment and retries, which eliminates a class of duplicate calls.

Q: How should I think about LLM costs as we scale?

A: Track cost per active user, not total monthly spend. Total spend grows as you grow that's expected. What matters is whether cost per user is trending down over time as you optimise. A startup spending $5/user/month on LLM calls has a very different problem from one spending $0.20/user/month at 10x the scale.

The bill doesn't fix itself. The fixes are unglamorous: audit prompts, route models correctly, cache aggressively. None of this requires architectural surgery. It requires treating LLM calls the way you'd treat database queries as a resource with a real cost that needs to be managed. If you're not sure where your LLM spend is going, that's usually the first problem worth solving. It's the kind of infrastructure review we cover in the [→ AI-Native Startup Architecture guide].