September 15, 2025
If you've written regression tests for software, you have an intuition for how they work: a test encodes expected behavior, and if a code change breaks that behavior, the test fails. The assertion is exact. A function that returns 42 either returns 42 or it doesn't. The regression is binary.
LLM regression testing doesn't work this way, and trying to apply the same mental model leads to test suites that either miss real regressions or generate constant noise. Understanding the differences — there are several — is the prerequisite to building evaluation that actually works.
The most fundamental difference: the same input doesn't reliably produce the same output. Temperature settings, sampling parameters, and model internals all introduce variability. A test case that passes on Monday might technically fail on Tuesday not because anything changed, but because the model drew differently from its probability distribution.
This means you can't test for exact string matches on open-ended outputs. A response that says "the answer is 42" and a response that says "42 is the answer" are identical in meaning but fail a string equality check. Testing open-ended outputs for exact matches produces false positives every time the phrasing varies, training engineers to ignore test failures as noise.
AI regression tests need to assert on properties of the output — semantic meaning, presence of key claims, absence of prohibited content — not exact content. This requires different assertion mechanisms than traditional test frameworks provide.
In software, a regression is typically a function that used to return the right answer and now returns the wrong one. The failure is clear and the cause is a specific code change. In LLM systems, quality degradation is usually gradual and statistical. A prompt change might drop hallucination scores from 0.95 to 0.88 on average. That's a meaningful regression — but no individual output clearly "broke."
This has implications for how you define thresholds and interpret results. A single test run that shows a 0.05 drop in a metric might be noise from probabilistic outputs, or it might be a real regression. Running evaluations multiple times and averaging results reduces false positives from variance. Tracking metric trends over time, rather than just pass/fail on individual runs, catches gradual degradation that single-point checks miss.
For a multiplication function, the correct answer to 6 * 7 is exactly 42. For an LLM summarizing a document, there's no single correct summary — there's a range of acceptable outputs that accurately capture the key points without introducing errors or omitting critical information.
Writing regression tests for this requires specifying acceptance criteria rather than expected values. "The summary should mention the product name, note the deadline, and not introduce information not present in the source document." Each criterion is independently evaluable. A regression occurs when a previously-passing criterion starts failing, not when the output changes.
The implication: your test cases need more thought, not less. The criteria you write encode your quality standards. Vague criteria produce vague tests that catch nothing. Specific, independently evaluable criteria produce tests that catch real regressions.
Traditional software regressions come from code changes you made. LLM regressions can come from changes you didn't make: a model provider updates their model weights, changes their fine-tuning approach, or adjusts their safety filters. Your prompt and codebase are identical. The model behavior changed.
This means regression testing for AI systems needs to run not just on PRs but on a schedule. A weekly evaluation run against a fixed golden dataset catches model-side regressions that would otherwise surface through user complaints.
It also means versioning your evaluation results matters more than in traditional software. When you need to determine whether a regression came from a code change or a model change, you need evaluation results timestamped relative to both your deployment history and model version changes.
Traditional test suites treat false positives as something to fix. A flaky test is an annoyance. In AI evaluation, false positives are a systemic risk: if tests fail randomly due to output variance, engineers stop treating failures as meaningful. They approve PRs with failing tests. The test suite becomes decoration.
Managing false positive rate in AI evaluation requires deliberate design: run evaluations multiple times and require consistent failures before blocking, use deterministic metrics where possible and reserve probabilistic metrics for informational reporting, set thresholds with enough margin that normal output variance doesn't cross them.
The goal is a test suite that cries wolf infrequently enough that engineers take failures seriously when they happen. Precision matters as much as recall.
Not everything is different. The core principles hold: write tests before you see failures, add test cases for every production incident, keep the test suite fast enough to run on every change, don't skip the suite under deadline pressure. These disciplines apply just as much to LLM evaluation as to unit tests.
The teams that maintain high AI quality over time are the ones that treat evaluation as engineering infrastructure — same rigor, different mechanics. They adapt the methods but not the mindset.
Confident AI