A prompt tweak that helped one case quietly broke twenty others. Here's the CI eval harness we built so that never ships silently again.
LLM Evals in CI: Catching Prompt Regressions Before They Ship
Someone on the team fixed a bug where our support bot gave wrong refund policy answers. The fix was a two-line prompt change. It shipped Friday. By Monday, the bot had started hallucinating order numbers in a completely different flow, because the new instruction bled into cases it was never meant to touch. Nobody caught it because we had no way to catch it. We were testing prompts by eyeballing three examples.
That Monday is why we now run evals in CI, and no prompt change merges without them.
Everyone wants to talk about the framework. The real work is building a dataset of cases that represents what users actually send. We seeded ours from production logs: 140 real queries, deduplicated, labeled with the expected behavior. Not the expected exact output, the expected behavior, because two different phrasings can both be correct.
Each case is a small YAML record:
- id: refund_within_window
input: "I bought this 5 days ago and want my money back"
assertions:
- type: contains_any
values: ["30-day", "30 day", "eligible", "refund"]
- type: not_contains
values: ["order number", "tracking"]
- type: llm_judge
criteria: "States the refund is allowed without inventing order details"
Three assertion types cover most of what we need. Cheap string checks catch obvious breakage. The not_contains guard is what would have caught the Monday incident: it fails if the model invents an order number. And an LLM judge handles the fuzzy cases where correctness can't be a substring match.
The harness is boring Python, which is the point. Boring means it runs the same way every time.
import yaml
from anthropic import Anthropic
client = Anthropic()
def judge(output: str, criteria: str) -> bool:
resp = client.messages.create(
model="claude-haiku-4-5",
max_tokens=5,
messages=[{
"role": "user",
"content": f"Criteria: {criteria}\n\nResponse:\n{output}\n\n"
"Does the response meet the criteria? Answer PASS or FAIL.",
}],
)
return resp.content[0].text.strip().upper().startswith("PASS")
def run_case(case, system_prompt):
resp = client.messages.create(
model="claude-sonnet-4-5",
max_tokens=512,
system=system_prompt,
messages=[{"role": "user", "content": case["input"]}],
)
out = resp.content[0].text
for a in case["assertions"]:
if a["type"] == "contains_any" and not any(v.lower() in out.lower() for v in a["values"]):
return False
if a["type"] == "not_contains" and any(v.lower() in out.lower() for v in a["values"]):
return False
if a["type"] == "llm_judge" and not judge(out, a["criteria"]):
return False
return True
We use a small, cheap model as the judge (Haiku) and the production model for the actual generation. Mixing them keeps judge cost near zero, roughly 40 cents for a full 140-case run at current pricing, and the run finishes in under two minutes with a bit of concurrency.
The eval writes a score and CI fails below a threshold. We don't require 100%, because LLM outputs have real variance and a hard 100% gate would flap and get muted, which defeats the purpose.
python run_evals.py --system prompts/support.txt --out results.json
python -c "
import json, sys
r = json.load(open('results.json'))
rate = r['passed'] / r['total']
print(f'Pass rate: {rate:.1%} ({r[\"passed\"]}/{r[\"total\"]})')
sys.exit(0 if rate >= 0.92 else 1)
"
The 92% threshold came from measurement, not vibes. We ran the current prompt ten times and it landed between 93% and 96%, so 92% catches a real regression without failing on noise. When someone raises the prompt quality, we ratchet the threshold up.
An honest warning. LLM judges are not oracles. Ours passed a response that confidently cited a policy that didn't exist, because the judge only checked for a refund statement, not for factual grounding. We now keep a small set of "trap" cases with known-wrong outputs and assert the judge fails them. If the judge starts passing traps, the judge itself has drifted and we tighten the criteria. Eval the evaluator.
Start with 30 real cases from your logs and cheap string assertions before you touch LLM judges. That alone would have caught our Monday incident, and it costs nothing to run. Add judges only for the genuinely fuzzy cases, keep a trap set to audit them, and set your pass threshold from measured variance rather than a round number. The framework doesn't matter. The dataset does, and it only gets good if you keep feeding it the queries that broke in production.
