Skip to main content

Eval Tutorials & Guides

8 articles tagged with Eval.

A demo RAG app is easy; one users trust is not. This is the map for reliable retrieval-augmented generation: grounding, evaluation, retrieval quality, guardrails, and safe rollout.
••2 months ago

Production RAG Reliability — Making LLM Answers Trustworthy

A demo RAG app is easy; one users trust is not. This is the map for reliable retrieval-augmented generation: grounding, evaluation, retrieval quality, guardrails, and safe rollout.

KU
Kiril Urbonas·12
Read article
A prompt tweak or model bump can quietly wreck answers everywhere. Ship LLM changes the way you ship risky code: gate, shadow, canary, roll back.
••3 months ago

Shadow Testing and Canary Releases for LLM Changes

A prompt tweak or model bump can quietly wreck answers everywhere. Ship LLM changes the way you ship risky code: gate, shadow, canary, roll back.

KU
Kiril Urbonas·36
Read article
A RAG system that invents facts erodes trust fast. Here is how we ground answers, force citations, and catch the fabrications before users do.
••3 months ago

Hallucination Detection — Grounding and Citations for RAG

A RAG system that invents facts erodes trust fast. Here is how we ground answers, force citations, and catch the fabrications before users do.

KU
Kiril Urbonas·47
Read article
A user got our support bot to recite its system prompt and then draft a refund it wasn't authorized to give. Two layers of guardrails, one on input, one on output, closed both holes.
••3 months ago

Guardrails for Production LLMs: Input and Output Filtering That Holds

A user got our support bot to recite its system prompt and then draft a refund it wasn't authorized to give. Two layers of guardrails, one on input, one on output, closed both holes.

KU
Kiril Urbonas·27
Read article
A prompt tweak that helped one case quietly broke twenty others. Here's the CI eval harness we built so that never ships silently again.
••3 months ago

LLM Evals in CI: Catching Prompt Regressions Before They Ship

A prompt tweak that helped one case quietly broke twenty others. Here's the CI eval harness we built so that never ships silently again.

KU
Kiril Urbonas·23
Read article
You can't improve retrieval you don't measure. The offline eval harness that lets us change embeddings, chunking, and rerankers with confidence instead of vibes — with the metrics that actually predict production quality.
••3 months ago

RAG Retrieval Evaluation — Building an Offline Eval Harness Before You Ship

You can't improve retrieval you don't measure. The offline eval harness that lets us change embeddings, chunking, and rerankers with confidence instead of vibes — with the metrics that actually predict production quality.

KU
Kiril Urbonas·31
Read article
Parsing model output with a regex and a prayer doesn't survive contact with traffic. The validation layers that keep structured LLM output reliable — constrained decoding, schema validation, and the repair loop.
••3 months ago

LLM Output Validation — Schema-Constrained Generation in Production

Parsing model output with a regex and a prayer doesn't survive contact with traffic. The validation layers that keep structured LLM output reliable — constrained decoding, schema validation, and the repair loop.

KU
Kiril Urbonas·40
Read article
Most LLM eval suites correlate poorly with what real users experience. The eval patterns we run that move with prod metrics — and the ones that lied to us.
••4 months ago

LLM Evals That Actually Predict Production Quality

Most LLM eval suites correlate poorly with what real users experience. The eval patterns we run that move with prod metrics — and the ones that lied to us.

KU
Kiril Urbonas·19
Read article