Evaluating AI Systems in Production

Most teams do not ship an AI feature because the model works. They ship because a demo worked, on a handful of inputs, on a good day. The gap between that demo and a system that holds up under real traffic is almost always an evaluation gap. The teams that move fastest are not the ones with the largest models, they are the ones who can answer a simple question on demand: is this version better or worse than the last one, and how do we know?

This guide lays out how we approach evaluation for the AI and data systems we build at Welzin. It is deliberately practical. The goal is a measurement habit that lets a team change prompts, swap models, and refactor pipelines without flying blind.

Why evaluation is the real bottleneck

Generative systems fail differently from traditional software. A function returns the wrong number and a test goes red. A language model returns a fluent, confident, plausible answer that is quietly wrong, and nothing breaks. Without a deliberate way to catch that, the failure surfaces where you least want it: in front of a user, or inside a downstream decision.

The cost compounds. Every change to a prompt, a retrieval step, a chunking strategy, or a model version can shift behavior in ways that are invisible until someone notices. If the only feedback loop is user complaints, iteration slows to the speed of trust, and trust is expensive to rebuild. A working evaluation harness turns that loop from weeks into minutes.

Three layers worth measuring

It helps to separate evaluation into three layers, because each answers a different question and each fails in a different way.

• Component. Does each step do its job? For a retrieval-augmented system, this is retrieval quality (did we fetch the passages that contain the answer), measured independently of the model that consumes them.
• System. Does the end-to-end output meet the task spec? This is the answer the user actually sees, scored for correctness, faithfulness to sources, format, and safety.
• Business. Does the system move the metric you were hired to move? Resolution rate, time saved, conversion, cost per task. This is the only layer that justifies the project, and the one most teams forget to instrument.

A system can be excellent at the component layer and still fail the business layer. Perfect retrieval feeding a model that ignores the retrieved context produces confident nonsense. Measuring only the end-to-end output, meanwhile, tells you something is wrong but not where. You want all three.

The eval set is the product

An evaluation is only as honest as the dataset behind it. The most common mistake is to build an eval set from inputs that are easy to collect rather than inputs that reflect reality. A few principles keep it honest:

• Mine real traffic. Once you have any production or pilot usage, your best eval cases are the questions people actually asked, especially the ones that went badly.
• Cover the tail, not just the mean. Include adversarial inputs, empty or malformed inputs, out-of-scope questions, and the rare cases that carry the most risk. The average case is rarely where systems break.
• Stratify by what matters. Group cases by intent, difficulty, or customer segment so a single aggregate score cannot hide a regression in one slice.
• Keep it versioned and growing. Every production incident becomes a new test case. The eval set is a living asset, not a one-time deliverable.

A 50-case eval set that mirrors your hardest real traffic is worth more than 5,000 synthetic cases that all look like the demo.

Choosing metrics that mean something

There is no universal score for a generative system. The right metric depends on the task and on whether you have a reference answer to compare against.

Approach	Good for	Watch out for
Exact or constrained match	Classification, extraction, structured output	Too strict for free-form text; brittle to formatting
Reference-based similarity	Tasks with a known gold answer	Rewards surface overlap, not correctness
Reference-free checks	Faithfulness, grounding, toxicity, format	Needs a reliable judge and clear criteria
Human review	High-stakes, subjective, or novel tasks	Slow and costly; needs a clear rubric to stay consistent

Two distinctions do most of the work. First, decide whether the task has a ground-truth answer or whether you are judging a quality (is this grounded in the sources, is this safe, is this in the right format). Second, decide whether a cheap automatic check is trustworthy enough, or whether the case is important enough to put a human in the loop.

LLM-as-judge, used with discipline

Using a strong model to grade another model's output scales human judgment to thousands of cases for a fraction of the cost. It is genuinely useful, and it is also easy to fool yourself with. A few rules keep it trustworthy:

• Give the judge a rubric, not a vibe. Score against explicit criteria with clear pass and fail definitions. Binary or low-cardinality scales are more stable than asking for a number from 1 to 10.
• Validate the judge against humans. Before you trust the judge at scale, confirm it agrees with human raters on a labeled sample. If it does not, fix the rubric before you fix the system.
• Control for known biases. Judges can favor longer answers, their own style, or whichever option is shown first. Randomize order and watch for these effects.
• Do not let the model grade its own homework unchecked. Keep a human-audited slice as a permanent calibration check on the automated judge.

Offline gates, online truth

Offline evaluation runs against your curated set on every change. Treat it like a test suite: wire it into CI, set thresholds per slice, and block a merge that regresses a critical metric. This is what makes fast iteration safe, because a bad prompt change is caught before it ever reaches a user.

Offline evaluation cannot tell you everything, because real users are more creative than any eval set. Online evaluation closes the gap:

• Canary releases. Roll a change to a small slice of traffic first and compare it against the incumbent before a full rollout.
• Controlled experiments. When you can attribute a business metric, an A/B test is the cleanest evidence that a change helped.
• Production sampling. Continuously sample live outputs for human or automated review, so quality is monitored, not assumed.
• Drift monitoring. Inputs change, models change underneath you, and the world changes. Track input distributions and quality over time so a slow decline does not become an outage.

A rollout checklist

Before an AI feature carries real load, we want to be able to check every box below.

• A versioned eval set built from real, hard, and adversarial inputs, stratified by slice.
• Component, system, and business metrics defined, with thresholds per slice.
• An automated judge validated against human ratings, with a permanent human-audited sample.
• Offline evaluation wired into CI as a merge gate, not a manual step.
• A canary or A/B path for shipping changes to a slice of traffic first.
• Production sampling and drift monitoring with an owner and an alert.
• A loop that turns every incident into a new permanent test case.

The takeaway

Evaluation is not a phase at the end of a project, it is the instrument panel that makes every other decision faster and safer. The model you choose matters far less than your ability to tell, on any given day, whether the system is getting better or worse and why. Build that capability first, and the rest of the work compounds on top of it.

This is the discipline we bring to the AI and data systems we build, and stay accountable to the number a team actually cares about. If you are weighing an AI build and want a clear-eyed view of how you would measure it, we are happy to talk it through.