# LLM Evals in CI: Catching Prompt Regressions Before They Ship

URL: https://www.devopsness.com/blog/llm-evals-in-ci-catching-prompt-regressions
Author: Kiril Urbonas
Published: 2026-07-01
Updated: 2026-11-17
Categories: AI
Tags: LLM, Evals, CI/CD, Testing, AI

Summary: A prompt tweak that helped one case quietly broke twenty others. Here's the CI eval harness we built so that never ships silently again.

---
# LLM Evals in CI: Catching Prompt Regressions Before They Ship

To catch prompt regressions, run a fixed dataset of real user queries against every prompt change in CI, score each output with string assertions plus an LLM judge for fuzzy cases, and fail the job when the pass rate drops below a threshold set from measured run-to-run variance. Without that gate, a two-line fix breaks flows nobody re-read.

> [!VERIFIED]
> Claude Haiku 5.5 (`claude-haiku-5-5`) and Claude Sonnet 5.5 (`claude-sonnet-5-5`) model IDs and pricing, GitHub Actions workflow syntax and actions/checkout v7, against the platform.claude.com models overview and evaluation guide and docs.github.com. Checked October 2026.

The failure we design against looks like this. Someone fixes a support bot that gave wrong refund answers. The fix is a two-line prompt change and it ships on Friday. By Monday the bot is inventing order numbers in a completely different flow, because the new instruction bled into cases it was never meant to touch. Nobody notices, because the only test was eyeballing three examples.

That is why no prompt change merges without evals in CI. It's also the first item on the release checklist in our [10 production lessons from Building LLM Powered Applications](/blog/building-llm-powered-applications-lessons).

## The dataset is the hard part, not the harness

Everyone wants to talk about the framework. The real work is building a dataset of cases that represents what users actually send. Seed it from production logs: pull a hundred or so real queries, deduplicate them, and label each with the expected behavior. Not the expected exact output, the expected behavior, because two different phrasings can both be correct.

Each case is a small YAML record:

```yaml
# evals/support.yaml
- id: refund_within_window
  input: "I bought this 5 days ago and want my money back"
  assertions:
    - type: contains_any
      values: ["30-day", "30 day", "eligible", "refund"]
    - type: not_contains
      values: ["order number", "tracking"]
    - type: llm_judge
      criteria: "States the refund is allowed without inventing order details"
```

Three assertion types cover most of what we need. Cheap string checks catch obvious breakage. The `not_contains` guard is what catches the Monday incident: it fails if the model invents an order number. And an LLM judge handles the fuzzy cases where correctness can't be a substring match.

## Running it

The harness is boring Python, which is the point. Boring means it runs the same way every time.

```python
# run_evals.py
import yaml
from anthropic import Anthropic

client = Anthropic()

def judge(output: str, criteria: str) -> bool:
    resp = client.messages.create(
        model="claude-haiku-5-5",
        max_tokens=5,
        messages=[{
            "role": "user",
            "content": f"Criteria: {criteria}\n\nResponse:\n{output}\n\n"
                       "Does the response meet the criteria? Answer PASS or FAIL only.",
        }],
    )
    return resp.content[0].text.strip().upper().startswith("PASS")

def run_case(case, system_prompt):
    resp = client.messages.create(
        model="claude-sonnet-5-5",
        max_tokens=512,
        system=system_prompt,
        messages=[{"role": "user", "content": case["input"]}],
    )
    out = resp.content[0].text
    for a in case["assertions"]:
        if a["type"] == "contains_any" and not any(v.lower() in out.lower() for v in a["values"]):
            return False
        if a["type"] == "not_contains" and any(v.lower() in out.lower() for v in a["values"]):
            return False
        if a["type"] == "llm_judge" and not judge(out, a["criteria"]):
            return False
    return True
```

Use a small, cheap model as the judge and the production model for the generation. Anthropic's own evaluation guide recommends grading with a different model from the one that produced the output, and constraining the grader to a one-word answer keeps parsing trivial. As of October 2026, Claude Haiku 5.5 starts at $0.10 per million input tokens against $2 for Sonnet 5.5, so judge calls are a rounding error next to the generation calls. Price your own run from token counts rather than trusting anyone's per-run figure, ours included.

## Making it block a merge

The eval writes a score and CI fails below a threshold. Don't require 100%: LLM outputs have real variance, a hard 100% gate flaps, and a flapping gate gets muted, which defeats the purpose.

```bash
$ python run_evals.py --system prompts/support.txt --out results.json
$ python -c "
import json, sys
r = json.load(open('results.json'))
rate = r['passed'] / r['total']
print(f'Pass rate: {rate:.1%} ({r[\"passed\"]}/{r[\"total\"]})')
sys.exit(0 if rate >= 0.92 else 1)
"
```

Wire it to run only when prompts or eval cases change, so you are not paying for evals on every CSS fix:

```yaml
# .github/workflows/evals.yml
name: prompt-evals
on:
  pull_request:
    paths: ["prompts/**", "evals/**"]
jobs:
  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - run: pip install anthropic pyyaml
      - run: python run_evals.py --system prompts/support.txt --out results.json
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
      - run: python check_threshold.py results.json 0.92
```

Set the threshold from measurement, not vibes. Run the current prompt ten times against the dataset and look at the spread. If it lands between, say, 93% and 96%, a 92% gate catches a real regression without failing on noise. When someone raises the prompt quality, ratchet the threshold up. A 0.92 copied from a blog post, this one included, is a guess about your data.

## The judge lies sometimes

LLM judges are not oracles. A judge that only checks "does this state the refund is allowed" will happily pass a response that confidently cites a policy that doesn't exist, because it never checked factual grounding. Keep a small set of trap cases with known-wrong outputs and assert the judge fails them. If the judge starts passing traps, it has drifted, and you tighten the criteria. Eval the evaluator. The same discipline applies to retrieval: our [offline harness for RAG retrieval evaluation](/blog/rag-retrieval-evaluation-offline-harness) uses labeled cases to keep the retriever honest the same way.

One more failure mode: model upgrades. A pinned model ID makes runs comparable over time. Changing the model is a change like any other, so it goes through the same eval gate as a prompt edit, and the threshold may need re-measuring afterwards.

## Questions people ask

### How many test cases does an LLM eval suite need?

Start with about 30 real queries pulled from production logs, chosen to cover your main intents and the cases that have broken before. That is enough to catch gross regressions on a pull request. Grow it every time an incident slips through: each production failure becomes a new case. A hundred to a few hundred well-labeled cases beats thousands of synthetic ones.

### Should every eval use an LLM as a judge?

No. Use deterministic string, regex, or schema checks wherever correctness is checkable that way, because they are free, fast, and never drift. Reserve the LLM judge for genuinely fuzzy criteria like tone or grounding. When you do use one, grade with a different model from the one generating, force a one-word verdict, and audit it with trap cases. Pair it with [schema-constrained output validation](/blog/llm-output-validation-schema-constrained-generation) for structured responses.

### What pass rate should block a merge?

Pick the threshold just below the bottom of your measured variance. Run the unchanged prompt several times, record the lowest and highest pass rates, and set the gate slightly under the lowest. A 100% requirement flaps on normal model variance and gets disabled. Re-measure after model upgrades and ratchet the gate upward when prompt quality improves.

## The call we'd make

Start with 30 real cases from your logs and cheap string assertions before you touch LLM judges. That alone catches the Monday incident, and it costs almost nothing to run. Add judges only for the genuinely fuzzy cases, keep a trap set to audit them, and set your pass threshold from measured variance rather than a round number. The framework doesn't matter. The dataset does, and it only gets good if you keep feeding it the queries that broke in production.

## Sources

- [Claude Docs: Models overview (IDs and pricing)](https://platform.claude.com/docs/en/models/overview)
- [Claude Docs: Define success criteria and build evaluations](https://platform.claude.com/docs/en/test-and-evaluate/develop-tests)
- [GitHub Docs: Workflow syntax for GitHub Actions](https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-syntax)

