T

"Working well" for an LLM feature is not a feeling. It's not the absence of complaints. It's a set of measurements you either have or you don't and most startups don't have them until something goes badly wrong. This is how to get ahead of that.

Why "it looks good" is not enough

Language models are confident by design. They produce fluent, grammatically correct, authoritative-sounding text whether the answer is correct or not. This makes subjective evaluation worthless at scale. A founder reviewing ten conversations sees nothing wrong. The system is generating wrong answers in 15% of cases and your confidence in the 90% that look fine is masking the signal.

The second problem: degradation is slow and invisible. A developer adjusts a prompt. The change improves one use case and silently degrades three others. Responses get slightly less accurate over two weeks. No single conversation flags it. No user writes in to complain they just stop using the feature, or worse, they trust it when they shouldn't.

This is the operational reality of LLM features in production. You need instrumentation before the degradation happens, not after the damage is done. What that instrumentation looks like depends on what your feature does but the underlying framework is the same regardless.

The three things you're actually measuring

Every LLM evaluation question reduces to three properties.

Correctness. Did the model produce an accurate answer? For factual use cases a support AI answering product questions, a document analyser extracting structured data correctness is measurable. You have a ground truth. Compare the output to it. For generative use cases with no single right answer, correctness becomes harder. You're evaluating quality dimensions instead: completeness, relevance, absence of hallucination.

Consistency. Does the feature produce similar outputs for similar inputs, across users, across time, across model versions? An LLM feature that answers the same question differently on Tuesday than it did on Monday is not a feature you can trust. Consistency is tracked by logging input-output pairs over time and monitoring for drift semantic drift, format drift, length drift.

Failure rate. How often does the model produce something that should never leave the system? Hallucinated facts. Responses that ignore the provided context entirely. Outputs that violate your format requirements. Answers that are technically responsive but completely unhelpful. These are failures, and they should be counted, not anecdotally described.

The framing that matters for founders: correctness is what users notice in the short term, consistency is what they notice over weeks, and failure rate is what creates trust or destroys it. All three need monitoring from the day a feature ships.

Practical methods that don't require a research team

Human spot-checking with a structured rubric. The simplest starting point. Once a week, someone on the team reviews fifty random conversations using a scoring rubric a fixed set of yes/no questions: Did the response use the provided context? Did it answer the actual question asked? Did it avoid fabricating information? Consistent rubrics make week-over-week comparison meaningful. Unstructured "does this look good" reviews don't.

LLM-as-judge. Use a second model call to evaluate the output of the first. Your production model generates a response. An evaluator prompt passes that response back to a model with a specific critique task: "Given this user question and this context, rate whether the following response is accurate and based solely on the provided information. Return a score and a reason." This scales to thousands of evaluations per day without a team of human reviewers. It's not perfect models have biases as evaluators but it catches systematic failure modes that human review misses at volume. This is now standard practice at companies [→ building AI features at scale].

Reference-based evaluation. For use cases where you have known good answers a support AI covering your documentation, a data extractor that should produce structured output build a test set. A hundred input-output pairs where you know what the correct answer is. Run your production prompt against this test set weekly. Track accuracy over time. A prompt change that improves accuracy on the test set is probably a genuine improvement. One that drops it is a regression, caught before it reaches users.

User feedback signals. Explicit thumbs up/down buttons, if you've built them. Copy button clicks as a proxy for useful responses. Time-to-next-message as a proxy for confusion. These are noisy signals but directional. A sustained drop in thumbs-up rate following a prompt change is a signal worth investigating even if you can't prove causation.

What to actually measure in your dashboard

Your AI quality dashboard needs at minimum:

Response latency by feature, broken down by p50 and p95. Latency spikes often precede quality degradation a model behaving unexpectedly tends to also be slow.

Token usage per response, tracked over time. A prompt that used to generate 200-token responses now generating 600-token responses is not a neutral change. Length drift almost always reflects a quality change in the underlying behaviour.

LLM-as-judge scores, aggregated weekly by feature. A rolling average makes trends visible. A threshold alert "if weekly average drops below 0.75, page the on-call" gives you a circuit breaker.

Failure classification counts. How many responses triggered format validation failures? How many were flagged by content filters? How many returned no usable answer? These aren't soft quality signals they're hard failures, and their rate should be tracked like error rates anywhere else in your system.

The question founders should ask their engineering team

Not "is the AI working?" that question produces useless answers.

Ask these instead: "What is our current LLM-as-judge score, and how has it trended over the last thirty days?" "What's our definition of a failure for this feature, and how many failures happened last week?" "If a developer changes the prompt on Tuesday, how would we know by Wednesday whether quality improved or degraded?"

If your team can't answer those questions with actual numbers, you don't have production-quality AI you have a demo that's been deployed. That gap is worth closing before users close it for you.

The engineering work to get here is not enormous. A structured evaluation suite, a weekly automated run, a Slack alert when scores drop. One solid sprint. What makes it feel hard is that nobody builds it until something breaks. The companies whose [→ AI features stay reliable in production] are the ones that treated evaluation as infrastructure from the start, not as something to add after users complained.

FAQ

Q: How is LLM evaluation different from normal software testing? A: Traditional software tests check deterministic behaviour input X always produces output Y. LLM outputs are probabilistic. The same prompt produces different text on each call. Evaluation instead asks: does the distribution of outputs meet quality criteria? This requires statistical thinking and larger sample sizes than unit tests.

Q: What is an evaluation set and how do I build one? A: An evaluation set is a collection of input-output pairs where you know what a correct or high-quality response looks like. Start small: fifty examples, manually reviewed by someone who understands the feature. Run your production prompt against them weekly. When accuracy drops, investigate before shipping the next prompt change.

Q: Do I need a separate evaluation framework or can I build this myself? A: You can build the basics yourself. A script that runs your prompt against your test set, calls a judge model, and writes scores to a database is a few hundred lines. Frameworks like LangSmith and Braintrust add tooling around this prompt versioning, trace logging, comparison views and are worth evaluating if your team is running multiple AI features with active prompt iteration.

Q: How should I think about hallucination rate as a metric? A: Hallucination is when the model presents fabricated information as fact. For RAG-based systems, the practical metric is grounding rate what percentage of factual claims in the response can be traced to the provided context? LLM-as-judge prompts can score this systematically. Aim to understand your baseline grounding rate before you promise users that your AI is accurate.

Q: My team says manual evaluation doesn't scale. Are they right? A: For systematic quality measurement, yes. Manual review is essential for calibrating your automated evaluations and catching qualitative failure modes that metrics miss. Use both: automated evaluation for scale and trend detection, human review for weekly calibration and for investigating anomalies the automated scores flag.

Every founder should know two numbers about their AI feature: what's the current quality score, and what was it last month. If those numbers don't exist yet, the conversation to have with your engineering team is about evaluation infrastructure, not about adding more AI features. Get the measurement layer right first everything built on top of it will be better for it.