T

The team builds a prototype. The prototype uses direct prompts. It works well enough to impress people. Then someone usually the most senior engineer in the room says "we should add RAG" or "we should fine-tune the model." The rest of the team nods. Architecture reviews happen. Infrastructure gets designed. Six weeks of engineering later, the product is more complex, more expensive to operate, and performs about the same as it did with well-written prompts.

This is not a hypothetical. It happens because RAG and fine-tuning are genuinely impressive techniques, and engineers are drawn to impressive techniques. The problem is that they solve specific problems and if you don't have those specific problems, you've added complexity without buying anything.

This post gives you a decision framework for the conversation you should have before your team commits to an approach. Not how to implement any of these your engineers know that. How to decide which one your product actually needs.

Start here: what problem are you actually solving?

The three approaches answer three different questions.

Prompt engineering answers: how do I get the model to behave the way I want, given what it already knows?

RAG answers: how do I give the model access to information it doesn't have my specific data, my knowledge base, my documents?

Fine-tuning answers: how do I change the model's default behaviour, style, or reasoning patterns at a fundamental level?

Most products need the first. Some need the first and second. Very few early-stage startups need the third.

The mistake is treating these as a progression as if you start with prompts, graduate to RAG, and eventually fine-tune like a serious company. They're not a ladder. They're three different tools for three different jobs. Using a fine-tuned model when prompt engineering would have worked is like hiring a machinist to hang a picture frame.

Before any architecture decision, ask: which of these three questions is my product actually trying to answer?

Prompt engineering: the option that gets dismissed too quickly

Prompt engineering has an image problem. It sounds like something you do when you don't know how to do real AI engineering. Senior engineers often want to move past it quickly.

This is wrong. A well-crafted system prompt with a clear persona, specific instructions, and a few good examples can match or outperform RAG for a surprisingly large class of problems. And it costs almost nothing to iterate on change the prompt, test the output, repeat.

The practical ceiling for prompt engineering is roughly this: if your knowledge base fits inside the model's context window, and doesn't change so frequently that keeping the prompt current is a maintenance burden, prompt engineering is the right answer. For a product FAQ, a customer-facing assistant trained on your pricing and feature set, or a coding assistant that follows your internal style guide these are all prompt engineering problems.

Modern context windows are large. GPT-4 handles 128,000 tokens. Claude handles up to 200,000. That's roughly 150,000 to 300,000 words of context per call. Most startup knowledge bases fit comfortably inside this. If your documentation is under 100,000 words and you're reaching for RAG, ask whether the complexity is warranted.

The red flag that prompt engineering is hitting its ceiling: your system prompt is growing past 2,000 tokens and still missing edge cases; your knowledge base is updated daily and maintaining the prompt is becoming a part-time job; response quality degrades noticeably on questions that require combining information from multiple parts of your knowledge base.

When you hit those red flags, RAG is the right next step.

RAG: the right upgrade, but only when you've earned it

Retrieval-Augmented Generation (pola pengambilan data berbasis vektor sebuah metode di mana sistem mencari potongan informasi yang relevan sebelum memanggil model) solves one problem well: getting relevant information from a large, changing knowledge base into the model's context window at query time.

The mechanics matter for the decision. Your documents get split into chunks, converted to vectors (numerical representations of meaning), and stored in a vector database. When a user asks a question, the question is also converted to a vector. The system finds the chunks most semantically similar to the question and passes them to the model alongside the question. The model answers using those chunks as context.

This is powerful when it works. When it doesn't work, the failure modes are subtle and expensive.

Retrieval quality is everything. The model can only answer from what it retrieves. If your retrieval layer returns the wrong chunks because the question is ambiguous, because your chunking strategy cut a key sentence in half, because the embedding model doesn't understand your domain's jargon the model answers confidently from irrelevant context. The output looks fine. It's wrong. This is worse than "I don't know" because the user trusts it.

RAG requires ongoing maintenance. Your vector database needs to stay in sync with your source documents. When a document changes, the old chunks need to be removed and new ones added. When documents are deleted, their chunks need to be purged. Teams consistently underestimate this. The pipeline that looked simple in a prototype becomes a piece of infrastructure that breaks in non-obvious ways when documents change upstream.

Cost scales with query volume. Each RAG query involves at minimum two API calls: one to embed the question, one to call the LLM. Often more if you're re-ranking retrieved results. At low query volumes this is negligible. At 100,000 queries per day, it compounds.

The honest checklist for whether you need RAG:

• Is your knowledge base larger than fits in a single prompt? (If no: don't use RAG yet)
• Does it update more than weekly? (If no: managed prompt updates may be easier)
• Are users asking questions that require combining information from multiple documents? (If no: per-document context may be enough)
• Do you have the engineering capacity to maintain a retrieval pipeline? (If no: the simpler approach buys you time to build that capacity)

If you answered yes to all four, RAG is probably right. If you answered yes to two or fewer, prove that prompt engineering has failed before adding the complexity.

The internal post that goes deeper on the vector database options is once it's live.

Fine-tuning: the answer to a question you probably aren't asking yet

Fine-tuning takes a base model and trains it further on your specific data, adjusting the model's weights to make it more likely to respond in particular ways. It's the most technically sophisticated of the three approaches and the one with the narrowest legitimate use case at early stages.

What fine-tuning actually changes: the model's default style, tone, and reasoning patterns. What it doesn't change: the model's knowledge cutoff or its access to your proprietary data. A fine-tuned model trained on your support documentation doesn't "know" your documentation it knows the patterns and style of responses that documentation exemplifies.

This distinction is critical and widely misunderstood. Teams fine-tune when they want the model to "know" their data. That's a retrieval problem, not a fine-tuning problem. RAG is the right tool. Fine-tuning on top of a model that doesn't have retrieval doesn't give it knowledge of your data it gives it a style trained on your data, which is a different thing.

The genuine fine-tuning use cases are narrow:

You need the model to consistently follow a very specific output format that you can't reliably achieve through prompting. You're building a code generation tool that must always produce code in your company's internal style with your specific patterns and that style is too complex to specify in a prompt. You have a domain with specialist terminology where the base model's defaults are consistently wrong, and you have thousands of high-quality examples of the correct behaviour.

Notice that last item: thousands of examples. Fine-tuning requires substantial labelled data. It requires compute during training. It requires re-training when your domain knowledge evolves. It requires evaluation infrastructure to verify that the new version is better than the previous one. For most Seed-to-Series-A teams, this is infrastructure they shouldn't be building yet.

The question to ask when an engineer proposes fine-tuning: "What specifically is the base model doing wrong that we can't fix with a better system prompt?" If there's a clear, specific answer and it's not a retrieval problem in disguise fine-tuning deserves the conversation. If the answer is vague ("it doesn't quite sound like us" or "it sometimes misses the point"), that's a prompt engineering problem.

The hybrid approach nobody talks about honestly

Real production AI systems rarely use just one of these approaches. They use them in layers, and the layering is where the interesting decisions live.

A common production pattern: a well-crafted system prompt (prompt engineering) defines the model's persona, constraints, and output format. RAG retrieves the relevant context for each query. A small, fine-tuned classification model (not the main LLM a separate, cheap classifier) routes queries to the right retrieval strategy before the main model call.

The classifier is the underrated piece. Training a small model to classify queries into categories "product question," "billing question," "technical support" is cheap, fast to train, and dramatically improves retrieval quality because you retrieve from the right subset of your knowledge base. This is fine-tuning applied correctly: not to the main LLM, but to a small, cheap, purpose-built model with a narrow job.

The principle is: use the most powerful technique at the smallest scale where it's needed. Fine-tune the 100M parameter classifier, not the 70B main model. Use RAG for the dynamic knowledge base, prompt engineering for the stable instructions.

The decision framework, condensed

Work through this with your team before any architecture is committed.

First: can you solve this with prompt engineering alone? Write the best system prompt you can. Test it against 50 real user queries. If it handles 80%+ adequately, ship it. Add complexity when the gap between 80% and "good enough" is causing real user pain.

Second: if prompt engineering fails, is the failure because of knowledge scale (too much data to fit in a prompt), knowledge freshness (data changes too fast to maintain manually), or retrieval complexity (users need answers that combine multiple sources)? If yes to any: RAG.

Third: if RAG is working but you have a persistent, specific behavioural problem consistent format violations, domain-specific reasoning errors that prompting can't fix and you have the labelled data to train against it: fine-tuning, but only for the model or model component that needs it.

Everything else: keep it simple and keep iterating. The teams shipping the best AI products right now are not the ones with the most sophisticated architectures. They're the ones who understand which layer their problem actually lives in.

For the broader context on where these decisions sit within your overall AI architecture, the [→ Read: How to Architect an AI-Native Startup] pillar covers the full picture.

FAQ

Q: Our engineers say we need RAG because our knowledge base is too large. How do we verify that's true? A: Check the actual size of your knowledge base in tokens most LLM providers have tokenizer tools. If it fits in the context window of your primary model, test prompt engineering first. Context windows have grown significantly; what required RAG 18 months ago often doesn't today.

Q: Fine-tuning sounds expensive. What does it actually cost? A: Training costs depend on model size and dataset size. A fine-tuning run on a mid-size open-source model with a few thousand examples costs $50–500 in compute. The expensive part is the ongoing cost: you re-train when your domain evolves, and each iteration needs evaluation. The total engineering cost over 12 months is typically far higher than the compute cost alone.

Q: Can we use RAG and fine-tuning together? A: Yes, and sometimes it's the right answer a fine-tuned model that's better at your domain, combined with RAG for dynamic knowledge retrieval. The risk is compounding complexity. Both techniques need maintenance. If you're considering both, make sure you've exhausted the simpler approaches first and have clear evidence that each layer is earning its place.

Q: How do we know if our RAG implementation is actually working? A: The key metric is retrieval precision what percentage of retrieved chunks are actually relevant to the query. Build an evaluation set of 50-100 representative queries with known correct answers and manually review which chunks your retrieval layer returns. Poor retrieval quality is the most common cause of RAG systems that look fine in demos and disappoint in production.

Q: Is prompt engineering a permanent solution or just a stopgap? A: For many products, it's permanent. If your knowledge base is stable and fits in a context window, prompt engineering with good maintenance is the right long-term architecture. "Graduate" to RAG only when you have a specific problem that RAG solves. There's no shame in running a prompt-engineered AI feature at Series B.

The teams that get this right aren't the ones chasing the most sophisticated technique. They're the ones who stayed with the simpler approach until the simpler approach provably failed and then chose the upgrade that solved the specific failure, not the upgrade that sounded most impressive in a technical review.

Your engineers will build whatever you greenlight. Making sure that's the right thing is the job.