Shadow Testing and Canary Releases for LLM Changes

The scariest deploy we've shipped wasn't a schema migration. It was a two-word prompt edit. Someone swapped "be concise" for "be brief and direct," merged it on a Friday, and by Monday the support-deflection rate had dropped four points. No error, no alert, no stack trace. The model just started answering slightly worse for everyone at once.

That's the thing about LLM changes. A model version bump, a prompt reword, a retriever swap, a chunking tweak, a temperature change — each one rewrites the behavior of every request simultaneously. There's no partial failure to catch your eye. Quality erosion looks exactly like normal traffic until someone reads the transcripts. So we stopped treating these as config edits and started treating them like risky code: gated, observed, and reversible.

Here's the pipeline we run now.

Gate offline before anything touches traffic

The first line of defense is a golden-set eval that runs in CI and blocks the merge. We keep a versioned set of a few hundred real queries with reference answers and expected retrieved docs. Every prompt or model change runs against it, and the job fails if any tracked metric regresses past a threshold.

# eval_gate.py — runs in CI, exit non-zero to block merge
import json, sys
from evals import run_suite, load_golden

BASELINE = json.load(open("evals/baseline_scores.json"))
THRESHOLDS = {"faithfulness": -0.02, "answer_quality": -0.03, "cost_usd": 0.15}

def main(candidate_cfg):
    scores = run_suite(load_golden("evals/golden_v7.jsonl"), candidate_cfg)
    failed = []
    for metric, floor in THRESHOLDS.items():
        delta = scores[metric] - BASELINE[metric]
        # cost floor is a max increase; quality floors are max drops
        regressed = delta > floor if metric == "cost_usd" else delta < floor
        if regressed:
            failed.append(f"{metric}: {BASELINE[metric]:.3f} -> {scores[metric]:.3f}")
    if failed:
        print("EVAL GATE FAILED\n" + "\n".join(failed)); sys.exit(1)
    print("eval gate passed"); json.dump(scores, open("evals/candidate_scores.json","w"))

if __name__ == "__main__":
    main(sys.argv[1])

The gate catches obvious regressions cheaply. What it can't catch is the long tail of real traffic your golden set never imagined. That's what shadow mode is for.

Shadow mode: run the new version, serve the old one

Once a change clears the gate, we deploy it in shadow. Both versions run on every live request. Production still serves the current version to the user. The candidate runs in the background, its output logged and never returned. Then we compare.

async def handle(request):
    prod = await llm_answer(request, cfg=PROD_CFG)
    # fire-and-forget the shadow so it never blocks the user path
    asyncio.create_task(shadow_compare(request, prod))
    return prod  # user only ever sees prod

async def shadow_compare(request, prod):
    try:
        shadow = await llm_answer(request, cfg=CANDIDATE_CFG)
        await metrics.emit({
            "req_id": request.id,
            "prod_faithfulness": await judge_faithful(prod, request.context),
            "shadow_faithfulness": await judge_faithful(shadow, request.context),
            "agreement": semantic_sim(prod.text, shadow.text),
            "prod_cost": prod.cost_usd, "shadow_cost": shadow.cost_usd,
            "prod_latency_ms": prod.latency_ms, "shadow_latency_ms": shadow.latency_ms,
        })
    except Exception as e:
        metrics.count("shadow_error", tags={"kind": type(e).__name__})

Shadow answers three questions before a single user is exposed. Where does the new version disagree with prod, and are those disagreements improvements or regressions when a judge scores them side by side? What does it cost and how slow is it on the real query distribution, not the tidy golden set? And how often does it error or time out on inputs the eval never covered? We let shadow soak for a few days so it sees weekday and weekend traffic. If agreement is high and the disagreements skew better, we promote to canary. This is the same discipline behind retrieval-augmented generation that holds up in production.

Canary: serve it to a sliver, watch, auto-roll-back

Canary routes a small slice of real traffic to the new version and actually serves it. We start at 1 percent, hold, then step to 5, 25, 50, 100 as the signals stay clean. Routing is deterministic per user so nobody flip-flops between versions mid-session.

def pick_version(user_id: str, canary_pct: int) -> str:
    bucket = int(hashlib.sha256(user_id.encode()).hexdigest(), 16) % 100
    return "candidate" if bucket < canary_pct else "prod"

# rollback watcher, runs every 60s against the canary cohort
def check_canary(window):
    c, base = window["candidate"], window["prod"]
    breaches = [
        c["faithfulness"] < base["faithfulness"] - 0.03,
        c["thumbs_down_rate"] > base["thumbs_down_rate"] * 1.25,
        c["p95_latency_ms"] > base["p95_latency_ms"] * 1.3,
        c["error_rate"] > base["error_rate"] + 0.005,
        c["cost_per_req"] > base["cost_per_req"] * 1.2,
    ]
    if any(breaches) and c["n"] > 200:  # wait for a real sample
        set_canary_pct(0)
        page("canary auto-rolled-back", detail=breaches)

The rollback is automatic and it defaults to zero. A human decides to ramp up; a threshold breach decides to ramp down. That asymmetry matters, because at 3am you want the safe direction to be the one that needs no one awake.

What we actually measure

Offline scores lie a little, so the canary leans on live signals too. Faithfulness and answer quality come from an LLM judge scoring against retrieved context. User signals carry the real weight: thumbs, whether the user edited or retried the answer, and deflection or escalation rate for support flows. Then the boring operational trio — cost per request, p95 latency, and error rate. A change that lifts quality 2 percent while doubling cost is not a win, and the canary is where that trade-off shows its real number.

Version everything, name everything

None of this works if you can't say what shipped. Every prompt is a file in git with a semantic version, and the deployed config pins exact strings:

# config/answer.v7.yaml
prompt_id: answer_synthesis
prompt_version: 7.2.0
model: claude-sonnet-4-5-20250929
temperature: 0.2
retriever: hybrid_v3
golden_set: golden_v7

The version string rides along in every logged request. When a metric moves, we can join it to the exact prompt and model that produced it instead of guessing which of last week's four merges did the damage.

The runbook, condensed

1. Change the prompt or model config, bump the version, open the PR.
2. CI runs the eval gate on the golden set; regression blocks the merge.
3. Merge deploys to shadow; soak two to four days across a full traffic cycle.
4. Review shadow: agreement, judge deltas, cost, latency, error rate.
5. Promote to canary at 1 percent; the rollback watcher arms automatically.
6. Ramp 1 to 5 to 25 to 50 to 100 as signals hold; any breach zeroes it out.
7. Old version stays deployable for a week in case a slow signal surfaces late.

The call we'd make

If you only build one of these, build the eval gate — it's cheap and catches the dumb regressions before they ever ship. But the gate alone will lull you, because the failures that actually hurt live in traffic your golden set never saw. Shadow plus a small auto-rollback canary is what turns "we think this is better" into "we watched it be better on real users and nothing broke." That's a slower way to ship a prompt edit. After the Friday incident, slower is exactly what we wanted.