#LLM — Articles | DevOpsNess

#LLM

63 articles tagged with LLM.

Reranking in RAG: When a Cross-Encoder Earns Its Latency

Our RAG answers kept citing the wrong paragraph even when the right one was retrieved. A cross-encoder reranker fixed relevance but added 180ms. Here's when that trade pays off.

Kiril Urbonas·3

Read article

PreviousPage 2 of 6Next

••last week

LLM Evals in CI: Catching Prompt Regressions Before They Ship

A prompt tweak that helped one case quietly broke twenty others. Here's the CI eval harness we built so that never ships silently again.

Kiril Urbonas·1

Read article

••last week

Semantic Caching for LLM Apps: Cutting Cost on Repeated Queries

Users kept asking the same questions in slightly different words, and we paid full price every time. Semantic caching cut our LLM bill by a third.

Kiril Urbonas

Read article

••2 weeks ago

Hybrid Search for RAG: Combining BM25 and Vectors the Right Way

Pure vector search kept missing exact matches like error codes and CLI flags. Adding BM25 back and fusing the two lifted our retrieval recall by 11 points.

Kiril Urbonas·1

Read article

••2 weeks ago

RAG Chunking Strategies: Fixed, Semantic, and Recursive Compared

Our support bot kept citing half a sentence and missing the answer that sat two lines below. The culprit wasn't the model, it was how we split the docs.

Kiril Urbonas·1

Read article

••2 weeks ago

Prompt Caching for Production LLM Apps — Cutting Cost and Latency at the Token Layer

A long, stable system prompt re-billed on every request is money on fire. How prompt caching works, where the cache boundary belongs, and the structuring discipline that got us a big cost and latency cut without changing behavior.

Kiril Urbonas·6

Read article

••3 weeks ago

RAG Retrieval Evaluation — Building an Offline Eval Harness Before You Ship

You can't improve retrieval you don't measure. The offline eval harness that lets us change embeddings, chunking, and rerankers with confidence instead of vibes — with the metrics that actually predict production quality.

Kiril Urbonas·5

Read article

••3 weeks ago

LLM Output Validation — Schema-Constrained Generation in Production

Parsing model output with a regex and a prayer doesn't survive contact with traffic. The validation layers that keep structured LLM output reliable — constrained decoding, schema validation, and the repair loop.

Kiril Urbonas·3

Read article

••0 months ago

RAG vs Fine-Tuning — Picking the Right Tool, Honestly

They solve different problems. RAG injects knowledge; fine-tuning changes behavior. The decision criteria, the hybrid pattern, and what we'd do over.

Kiril Urbonas·8

Read article

••last month

LLM Cost Optimization in Production — What Actually Moves the Bill

Token caching, model routing, prompt compression, and the boring discipline of measuring. The levers that cut our LLM bill 60% without touching feature scope.

Kiril Urbonas·2

Read article

••last month

LLM Evals That Actually Predict Production Quality

Most LLM eval suites correlate poorly with what real users experience. The eval patterns we run that move with prod metrics — and the ones that lied to us.

Kiril Urbonas·5

Read article

••last month

Multi-Provider LLM Routing — Failover, Cost Routing, and Load Balancing

Single-provider LLM apps fail when the provider does. Multi-provider routing isn't just resilience — it's also a cost lever. The patterns we run.