Practical articles on AI, DevOps, Cloud, Linux, and infrastructure engineering.
A 180k-token context window is not a license to stuff everything in. Here's how we cut prompt size 60% without hurting answer quality, and what to trim first.
When our single LLM provider had a 40-minute outage, every AI feature went dark. A gateway with routing and fallback fixed that, and cut spend 30% as a bonus.
Users hit stop, but our server kept paying for tokens for another 40 seconds. Here's how we wired real cancellation and backpressure into an SSE streaming endpoint.
The top model on the MTEB leaderboard made our search worse and our bill bigger. Here's how we actually picked an embedding model for a real RAG system.
Most agents that "need memory" actually need a smaller context window and a database. Here's how we cut a support agent's token bill by 60 percent by deleting memory.
A user got our support bot to recite its system prompt and then draft a refund it wasn't authorized to give. Two layers of guardrails, one on input, one on output, closed both holes.
Our RAG answers kept citing the wrong paragraph even when the right one was retrieved. A cross-encoder reranker fixed relevance but added 180ms. Here's when that trade pays off.
A prompt tweak that helped one case quietly broke twenty others. Here's the CI eval harness we built so that never ships silently again.
Users kept asking the same questions in slightly different words, and we paid full price every time. Semantic caching cut our LLM bill by a third.
Pure vector search kept missing exact matches like error codes and CLI flags. Adding BM25 back and fusing the two lifted our retrieval recall by 11 points.
Our support bot kept citing half a sentence and missing the answer that sat two lines below. The culprit wasn't the model, it was how we split the docs.
A long, stable system prompt re-billed on every request is money on fire. How prompt caching works, where the cache boundary belongs, and the structuring discipline that got us a big cost and latency cut without changing behavior.
