T

Your engineers built an AI feature in two weeks. It summarises customer support tickets, or generates personalised onboarding flows, or answers questions about your product in natural language. The prototype impressed the board. It closed a deal. Now you need to ship it to real users and the questions start arriving.

How do you handle it when the model gives a confidently wrong answer? What happens when OpenAI has an outage? Why did last month's AWS bill jump $40,000? Can you guarantee response times under two seconds? What does your data residency story look like for your enterprise client?

These are not engineering questions. They're architecture decisions, and they get made consciously or by accident in the first few months of building. The defaults your team picks when moving fast tend to stick. A retrieval pipeline that works fine for 100 users starts creating real problems at 10,000. A single LLM provider dependency is fine until it's not.

This post is for the CTO or technical founder who wants to understand the decisions before they're made. Not to write the code your engineers will do that but to ask the right questions, understand the tradeoffs, and avoid the architectures that look fine in a demo and fall apart in production.

The three layers every AI-native product needs

Most teams building AI features think about one thing: the model. Which LLM to use, which prompt to write, how to make the output better. That's the visible layer. The two layers beneath it data and infrastructure are where production systems actually break.

A useful mental model: think of your AI architecture as three stacked concerns.

The model layer is what most people mean when they say "AI." It's the LLM you call, the prompt you send, the response you get back. This layer gets all the attention and is actually the easiest to change. Swapping from GPT-4 to Claude 3.5 to Gemini is a few lines of code if your architecture isn't tightly coupled to a single provider.

The data layer is where your product's actual value lives. The model is a commodity every competitor can access the same GPT-4 API. What they can't access is your proprietary data: your customer history, your product catalogue, your domain-specific knowledge base. How you store, retrieve, and inject this data into model calls is the difference between a generic chatbot and a product that's genuinely useful. This is where most of the engineering complexity sits.

The infrastructure layer is everything underneath: caching, queuing, rate limiting, observability, cost controls, failover. Boring. Invisible when it works. Catastrophically visible when it doesn't. This layer is what separates a prototype that ran well in a demo from a product that handles 10,000 users at peak.

When your engineers come to you with architecture proposals, the question to ask is: which layer does this decision affect? A debate about which LLM to use is a model-layer debate. A debate about how to store and retrieve your knowledge base is a data-layer debate. A debate about how to handle OpenAI rate limits is an infrastructure-layer debate. They're different problems and they deserve different conversations.

The data layer decision: RAG, fine-tuning, or neither

Before your team writes any infrastructure code, they need to answer one question: how does your proprietary data get into the model's context?

There are three answers. Choosing the wrong one is the most expensive mistake AI-native startups make.

Prompt engineering is the simplest. You put your data directly into the prompt. "Here is our product documentation: [paste 10,000 tokens]. Now answer this question." This works surprisingly well for small knowledge bases. It's cheap, fast, and requires no additional infrastructure. If your data fits in a prompt window and doesn't change frequently, this is the right answer. Most startups don't need anything more complex.

Retrieval-Augmented Generation (RAG) is the right answer when your knowledge base is too large to fit in a prompt, or when it changes frequently and you can't rebuild fine-tuned models every week. RAG retrieves only the relevant chunks of your data at query time and passes them to the model. It keeps the model's context window focused and your retrieval results fresh. The tradeoff is infrastructure: you need a vector database, an embedding pipeline, a retrieval layer, and the engineering to keep all of it accurate. For the right use case it's worth it. For a knowledge base with 200 documents, it's overkill.

Fine-tuning is the answer almost no early-stage startup actually needs, but the one engineering teams most often propose. Fine-tuning trains a new version of the model on your specific data, making the model itself more likely to respond in your style, with your terminology, for your domain. It's expensive (compute costs during training), slow (days to weeks per training run), and hard to update when your domain knowledge changes, you retrain. The narrow case where it makes sense: you need the model to consistently follow very specific formatting or reasoning patterns that prompt engineering can't reliably produce.

The question to ask your engineering team: "Why RAG instead of prompt engineering? Why fine-tuning instead of RAG?" If they can't articulate the specific limitation of the simpler approach, you're probably adding complexity you don't need.

We'll go deeper on this decision in .

The model layer decision: vendor lock-in and the abstraction you need

Your team is probably defaulting to OpenAI. That's fine the models are good and the API is well-documented. The problem isn't which model you choose. The problem is how tightly your code is coupled to that choice.

Hard-coded OpenAI calls scattered across your codebase means that when OpenAI has an outage (and they do), your product is down. It means that when Anthropic releases a model that's 40% cheaper for your use case, switching requires touching dozens of files. It means that when a potential enterprise client asks "can you use Azure OpenAI for data residency reasons?" the answer is a painful rewrite.

The architectural answer is a model abstraction layer a single internal interface your application calls, which handles the translation to whichever underlying model API you're using. Your application code says "generate this text." The abstraction layer decides which model to call, handles retries, manages rate limits, and falls back to a secondary provider if the primary is down.

This is often called an LLM gateway. It doesn't have to be complex. At the simplest level, it's a small internal service or module with a consistent interface. At the more sophisticated end, it manages cost routing (cheap model for simple tasks, expensive model for complex ones), provider failover, caching of repeated queries, and centralised logging of every model call.

The question to ask: "If we needed to switch from OpenAI to Anthropic tomorrow, how many files would we touch?" If the answer is more than one, the abstraction layer doesn't exist yet.

One specific thing to understand: not every task needs the frontier model. GPT-4 at $0.03 per 1K output tokens is not the right choice for classifying a customer support ticket into one of five categories. A smaller, faster, cheaper model or even a traditional classifier is better for that job. Cost routing means deciding at the application level which tasks warrant the expensive model and which don't. Your engineers should have a clear policy for this, not a default of "use GPT-4 for everything."

The infrastructure layer: the four things that will break in production

Most AI prototypes have none of the following. All production AI systems need at least three of them.

Caching. LLM API calls are expensive and slow. If the same query comes in twice, you should not pay to generate the response twice. Semantic caching goes further if two queries are asking essentially the same question, return the cached response even if the wording differs. This is not a premature optimisation. At any meaningful volume, caching is the difference between a sustainable cost structure and an AWS bill that doubles every month.

How to think about this: ask your engineers what percentage of your LLM calls are for queries your system has seen before. If they don't know, you don't have observability. If the answer is above 20%, you need caching yesterday.

Rate limiting and queuing. LLM APIs have rate limits. Your application will hit them. When it does, the default behavior of most prototypes is to return an error to the user. The production behavior should be to queue the request, process it when capacity is available, and notify the user. Your architecture needs a queue in front of your LLM calls something as simple as a Redis-backed job queue works for most volumes.

The second reason for queuing: cost control. Without a queue, a traffic spike sends 500 concurrent LLM requests. With a queue, you process them at a rate your budget can sustain. You control the throughput. The users wait slightly longer; you don't receive a $50,000 surprise invoice.

Observability. You cannot improve what you cannot measure. At minimum, every LLM call in production should log: the model used, the token count (input and output separately), the latency, whether the response was cached, and a quality signal (thumbs up/down from the user, or an automated evaluation). This data tells you where your costs are going, where your latency is coming from, and whether your AI features are actually working.

The question to ask: "If our AI feature started giving bad answers to 10% of queries today, how long would it take us to find out?" If the answer is anything other than "we'd see it in our dashboards immediately," your observability isn't good enough.

Fallback and graceful degradation. What happens when your LLM provider has an outage? The honest answer for most startups: the AI feature returns an error and the user sees a broken product. The production answer: the AI feature falls back to a secondary provider, or degrades gracefully to a non-AI version of the same functionality. A search feature that uses an LLM to rank results should fall back to traditional keyword search when the LLM is unavailable not return an error page.

This requires thinking about your AI features as enhancements to existing functionality, not replacements for it. If there's no non-AI fallback, you've made your product's uptime dependent on a third-party API's uptime. That's a risk you need to consciously accept, not accidentally inherit.

What a production AI architecture actually looks like

To make this concrete, here's what a well-architected AI feature looks like for a B2B SaaS with an AI-powered customer support assistant. Not theoretical a composite of several real implementations.

The user sends a message. It hits a rate limiter first. If they've sent more than 10 messages in 60 seconds, they get queued. Otherwise, the request goes to the application layer.

The application checks a semantic cache. If a similar query has been answered in the last hour, return the cached response. This handles 25-35% of queries in practice with no LLM cost.

Cache miss goes to the retrieval layer. The user's message is embedded (converted to a vector) and used to search the knowledge base for the 5-7 most relevant chunks. These chunks, plus the user's message history (last 5 turns), plus a system prompt are assembled into a context window.

The assembled context goes to the LLM gateway. The gateway decides: is this a complex query requiring the frontier model, or a simple one that a smaller model handles adequately? It routes accordingly, sets a timeout of 8 seconds, and retries once on failure before falling back to the secondary provider.

The response comes back. Before it reaches the user, it goes through a simple output filter does it contain anything it shouldn't? Is it refusing to answer when it should answer? This is a fast, cheap classifier, not a complex evaluation.

The response is logged: model used, tokens, latency, cache hit/miss, user ID, session ID. It goes into the semantic cache. It goes to the user.

The whole flow takes 800ms on a cache hit, 2-3 seconds on a cache miss with the frontier model, 600ms on a cache miss with the smaller model. The average cost per query is around $0.002 a tenth of what it would be without caching and cost routing.

This is not a complex architecture. It's six components, each with a single responsibility. The complexity is in the decisions about what each component does decisions that should be made before the code is written, not discovered when the AWS bill arrives.

The decisions to make before your engineers start building

A set of questions worth working through with your team explicitly, rather than letting the answers emerge by default.

Which LLM provider is your primary, and who is your fallback? The answer to the second question matters as much as the first.

What is your acceptable cost per AI query? If you don't have a number, your engineers have no constraint. Without a constraint, costs will be higher than they need to be.

What is your data residency requirement? If you have enterprise clients in regulated industries, you may need Azure OpenAI or a self-hosted model. Knowing this now prevents a painful migration later.

What does graceful degradation look like for each AI feature? For every AI-powered feature in your product, what does the user experience when the LLM is unavailable? This needs an answer, not a placeholder.

How will you know if the AI is giving bad answers? Automated evaluation, user feedback, human review, or some combination. If the answer is "we'll know from support tickets," that's too slow.

What is your caching strategy? Exact match, semantic match, or none. Each has different infrastructure implications.

Getting explicit answers to these six questions before architecture begins is worth more than any technical document your engineers will produce. The technical decisions follow naturally from the constraints. The constraints are yours to set.

The AI cost mistake that catches everyone

One thing worth calling out specifically because it's expensive and predictable: token costs grow non-linearly with product complexity.

A prototype sends one LLM call per user action. As features mature, teams add system prompts (more tokens), conversation history (more tokens), retrieved context (more tokens), output formatting instructions (more tokens), chain-of-thought reasoning steps (more tokens). Each addition feels small. Collectively, a feature that cost $0.002 per query in prototype can cost $0.02 in production a 10x increase driven entirely by prompt growth.

Watch for this specifically in the first three months after launch. If your cost-per-query is growing month-over-month despite your query volume being flat, prompt bloat is the likely cause. The fix is a prompt audit cutting system prompt length without degrading output quality is a legitimate engineering task, not a shortcut.

We cover the specific failure modes in .

FAQ

Q: How early should we think about AI architecture? We're still in prototype. A: The abstraction layer question should be answered before you write your second LLM call. Everything else can wait until you have real users. The abstraction layer is cheap to add early and expensive to retrofit it's the one decision that compounds.

Q: Do we need a vector database to build AI features? A: Only if you need RAG. Many AI features don't. If your knowledge base fits in a prompt window, put it in the prompt. Add vector infrastructure when you've confirmed that prompt engineering won't scale not before.

Q: How do we handle users getting different answers to the same question? A: LLM outputs are probabilistic. You'll need to decide whether consistency matters for your product and address it at the architecture level if it does through caching, lower temperature settings, or structured output formats. "The AI is sometimes inconsistent" is a product decision, not just an engineering one.

Q: When should we consider self-hosting a model instead of using the API? A: When any of three conditions are true: your data residency requirements can't be met by a managed API; your query volume makes the per-call API cost more expensive than the compute cost of self-hosting; or you need a fine-tuned model that the API providers don't offer. For most Seed-to-Series-A companies, none of these conditions apply yet.

Q: How do we evaluate whether our AI features are actually working well? A: Start with explicit user feedback thumbs up/down on AI responses, or asking "was this helpful?" Track this per feature, not in aggregate. Add automated evaluation for specific failure modes you care about (refusals, factual errors, format violations). Human review of a random sample each week is underrated and cheap.

The teams that build AI features well aren't the ones who picked the best model. They're the ones who thought clearly about the three layers model, data, infrastructure and made deliberate choices at each level before the code was written.

The decisions are yours. The abstractions are your engineers'. Getting that division right is what separates an AI product that scales from one that looks great until it doesn't.

If you want a second perspective on the architecture decisions before your team commits, [→ Read: How to choose a software development company in Indonesia] covers what to look for in a technical partner for exactly this stage.

Internal links used:

• G-2, direct downstream post
• G-3, cost deep dive
• architecture reviews Pillar #11, soft CTA

External links used:

• None kept internal

Word count: 3,520