Confident AI

Software engineering solved the "how do we know this is safe to ship?" problem with continuous integration: automated tests that run on every change and block deployments that fail. Machine learning has been slower to adopt the same discipline, partly because LLM evaluation is fundamentally different from unit testing and partly because the tooling to do it well has only recently matured.

This guide covers how to integrate LLM evaluation into your CI/CD pipeline as a hard quality gate — one that actually blocks bad deployments rather than producing reports that get ignored.

Why ML CI/CD Is Different From Software CI/CD

Traditional software tests are deterministic: given the same input, the same code always produces the same output, and the test either passes or fails. LLM evaluation is probabilistic: the same prompt can produce different outputs on different runs, and quality is a continuous dimension rather than a binary pass/fail.

This creates two practical challenges for CI/CD integration. First, you need evaluation metrics that are meaningful as gate criteria — something with enough signal to distinguish a good model update from a bad one, and stable enough that random output variation does not cause false failures. Second, you need to define what "failing" means in a probabilistic context — is a 2% drop in accuracy a regression or noise? How confident do you need to be before blocking a deploy?

These are not unsolvable problems, but they require more thought than simply running pytest and counting test failures. Getting them right is the difference between evaluation gates that the team trusts and evaluation gates that get bypassed after the third false positive.

Pipeline Architecture: Three Gate Types

Rather than a single evaluation gate, effective ML CI/CD uses layered gates at different stages with different trade-offs between thoroughness and speed.

Fast Gate (Pre-Commit or Pre-Push): 30-60 seconds. A small subset of your test cases — 10 to 20 high-signal examples that cover your most important scenarios and your most dangerous failure modes. The goal is a quick sanity check that catches obvious regressions without adding significant friction to the development loop. This gate should have a very low false-positive rate because it runs on every change; a slow or noisy gate here will be disabled or bypassed.

Standard Gate (Pull Request CI): 5-15 minutes. Your full evaluation suite against your complete test dataset. This gate runs on every PR and blocks merge if scores fall below defined thresholds. Results are reported in the PR as a detailed comment, giving reviewers visibility into which specific metrics changed and by how much. The standard gate is the primary quality enforcement mechanism for your team.

Deep Gate (Pre-Deployment): 30-60+ minutes. Comprehensive evaluation including adversarial scenarios, edge case coverage, and cross-model comparison if you are switching providers or configurations. This gate runs before production deployment and may include evaluations too expensive or too slow to run on every PR. The deep gate is where you validate that a model update is ready for real users, not just that it does not break your standard test suite.

Defining Thresholds That Mean Something

Threshold setting is where most ML CI/CD implementations either become too restrictive (blocking good changes due to noise) or too permissive (allowing real regressions through). A few principles for calibrating thresholds:

Baseline from your current production model, not from benchmarks. Your threshold for accuracy on your evaluation dataset should be set relative to what your current deployed model achieves, not relative to what the model scores on academic benchmarks. A threshold of "90% accuracy" means nothing without knowing whether your current model scores 88% or 94%.

Differentiate threshold types. Some thresholds should be absolute minimums (if hallucination rate exceeds X%, block regardless of other scores). Others should be regression thresholds (if accuracy drops more than Y% relative to the current baseline, block). Mixing these two types leads to confusing behavior.

Account for evaluation variance. Run your current production model through the evaluation suite multiple times to measure how much scores vary run-to-run due to temperature and sampling randomness. Your regression threshold should be set above this noise floor — otherwise you will get spurious failures on perfectly good model updates.

Tier thresholds by severity. Not every metric failure should block deployment. A hallucination rate increase should block; a 1% drop in BLEU score should generate a warning and require sign-off. Define which metrics are hard gates and which are informational.

GitHub Actions Integration Pattern

A practical GitHub Actions integration looks like this:

The workflow file defines an evaluation job that triggers on pull requests to main. It checks out the code, installs the evaluation SDK, runs the evaluation suite against the PR branch's model configuration, and fails the job if any hard-gate metrics fall below threshold. Evaluation results — including per-metric scores, delta from baseline, and any failed test cases — are posted as a PR comment.

Key implementation details that matter in practice:

• Use a fixed evaluation seed for the standard gate to reduce run-to-run variance. Variable seeds make baseline comparison noisier.
• Cache your evaluation dataset as a build artifact or in your evaluation platform's dataset registry. Re-downloading test cases on every run wastes time and creates drift risk if the source dataset changes.
• Store baselines as part of your deployment pipeline, updated automatically when a new version goes to production. Your regression thresholds should always be relative to the most recently deployed version, not an outdated baseline.
• Surface failures clearly. A CI step that fails with "evaluation failed" and no detail is nearly useless. The failure message should identify which test cases failed, what the model's output was, and what the expected output was.

Handling Gate Failures

Your process for handling evaluation gate failures determines whether the gate actually enforces quality or becomes a formality that engineers work around.

When a gate fails, the failing PR needs a clear path to resolution. That means: visible test failure details in the PR (not buried in CI logs), a classification of whether the failure is a real regression or a borderline case that needs human review, and a clear policy on when manual override is permitted (and who can authorize it).

Manual override capability is important — not because gates should routinely be bypassed, but because a gate that can never be overridden will eventually be disabled entirely when it blocks a time-critical fix. Build a well-governed override mechanism with audit logging rather than pretending the override case will never arise.

Track override frequency as a leading indicator. If your team is overriding the evaluation gate more than once a month, either your thresholds are miscalibrated (too tight) or your test dataset is outdated (catching failures that no longer represent real quality dimensions). Both require systematic fixes, not more overrides.

Evaluation Cost Management

LLM evaluation using a judge model has non-trivial cost. At scale, running your full evaluation suite on every PR can add up. Strategies to manage cost without reducing coverage:

• Stratified sampling for large test suites — run a random sample for initial screening, then run the full suite only if the sample passes. This reduces cost on most PRs while maintaining full coverage for the final gate before merge.
• Use cheaper evaluators for screening — lighter-weight models or heuristic metrics for the fast gate, with the more expensive judge model reserved for the standard and deep gates.
• Scope evaluation to changed components — if only the retrieval configuration changed, you may not need to re-evaluate generation quality for all test cases. Smarter scoping reduces cost without reducing meaningful coverage.

← Back to Blog