Continuous LLM evaluation: regression datasets that survive model swaps

Every team that puts an LLM into production learns the same lesson, sooner or later: the model that passed last week's manual test is not the one answering today. The provider quietly updates the model, someone tweaks a prompt "just a little," and quality drops with no warning. Without a regression dataset and automated gates, every change is a bet.

This article covers how to build that safety net: where to source cases, which metrics to measure, how to calibrate an LLM-as-judge, and how to wire all of it into your CI pipeline.

Why manual testing doesn't scale

Manual review works for validating a new feature. It doesn't work as a continuous guardrail because:

• it's too slow to run on every pull request;
• it's inconsistent across reviewers and over time;
• it misses subtle regressions (an answer that's 5% worse, not clearly wrong).

The replacement is a versioned dataset of real cases, evaluated automatically, with humans stepping in only to calibrate the evaluator — not to review every run.

Building the dataset from real cases

Synthetic datasets bias toward what the team imagined would happen. Real cases capture what actually happens.

1. 1.Collection: pull interactions from production (with consent and anonymization where required), prioritizing cases with negative user feedback, human escalation, or low model confidence.
2. 2.Curation: remove duplicates and trivial cases; keep a representative distribution of intents, not just the "hard" ones.
3. 3.Labeling: each case gets a reference (gold) answer or acceptance criteria, reviewed by a domain expert — not by the engineer who wrote the prompt.
4. 4.Versioning: treat the dataset as code. Every added case is a commit, with a rationale (bug found, new edge case, policy change).

A healthy regression dataset keeps growing: every production incident becomes a new case, so the same failure never slips through twice.

Metrics that matter

A single "quality" metric hides more than it reveals. Split it into dimensions:

Format, latency, and cost are cheap to check and should run as deterministic tests, without involving quality judgment. Reserve the LLM-as-judge for factual accuracy and source adherence, which require interpretation.

LLM-as-judge calibrated by human review

Using an LLM to evaluate another LLM is practical, but only trustworthy if calibrated:

1. 1.Run the judge against a sample of the dataset (50–100 cases) and collect its score.
2. 2.Ask human experts to evaluate the same cases, blind to the judge's score.
3. 3.Compute agreement (Cohen's kappa or simple correlation). Below roughly 0.7 agreement, the judge's prompt needs work — usually more specific criteria or calibration examples (few-shot) inside the evaluation prompt itself.
4. 4.Re-calibrate whenever you swap the judge model, not just the evaluated system.

A common mistake is using the same model as both system and judge without checking for self-preference bias: models tend to score responses with their own "style" higher. Prefer a judge from a different model family than the one being evaluated, or at least document the bias and adjust the threshold.

CI gates

With metrics and a calibrated judge in place, the pipeline is straightforward:

PR opens a prompt/model change
   │
   ▼
Run full regression dataset
   │
   ├─ Deterministic tests (format, schema) → failure blocks merge
   ├─ Latency/cost → failure if above agreed threshold
   └─ LLM-as-judge (accuracy, adherence) → failure if score drops below baseline
   │
   ▼
Merge allowed only if all gates pass

Define thresholds as a relative difference from baseline, not a fixed absolute value — this avoids the gate becoming stale as the product legitimately evolves. For example: "accuracy score cannot drop more than 2 percentage points relative to the previous commit on the main branch."

Detecting drift when swapping model or prompt

Automated regression covers PRs, but drift also happens outside explicit code changes: when the provider updates a model "behind the scenes," or when behavior shifts over time due to vendor-side tuning.

• Run the regression dataset on a fixed cadence (daily or weekly) against the production model, not just in PR CI.
• Alert when the score drops beyond the threshold even without a new deploy — that's the signal the provider changed something.
• Pin the model version (when the API allows it) for critical releases, and treat model upgrades as a deliberate change that goes through the same regression gate.
• Keep an output snapshot per model version, for manual diffing when the judge flags a drop but the cause isn't obvious.

What to do on Monday

• Export 30 to 50 real production interactions, prioritizing complaints and escalations, and build the first version of the regression dataset.
• Split metrics into deterministic (format, latency, cost) and judge-required (accuracy, adherence), and write automated tests for the former today.
• Run an initial LLM-as-judge calibration against 20 cases reviewed by a human on the team, and record the agreement as baseline.
• Add a simple CI gate: block merge if the accuracy score drops more than a value agreed with the team.

Further reading

Executive track:

• Prompt engineering didn't die. It moved up a layer. — the business view of this same topic.