Practical articles on AI, DevOps, Cloud, Linux, and infrastructure engineering.
When our single LLM provider had a 40-minute outage, every AI feature went dark. A gateway with routing and fallback fixed that, and cut spend 30% as a bonus.
Users hit stop, but our server kept paying for tokens for another 40 seconds. Here's how we wired real cancellation and backpressure into an SSE streaming endpoint.
The top model on the MTEB leaderboard made our search worse and our bill bigger. Here's how we actually picked an embedding model for a real RAG system.
Most agents that "need memory" actually need a smaller context window and a database. Here's how we cut a support agent's token bill by 60 percent by deleting memory.
A user got our support bot to recite its system prompt and then draft a refund it wasn't authorized to give. Two layers of guardrails, one on input, one on output, closed both holes.
Our RAG answers kept citing the wrong paragraph even when the right one was retrieved. A cross-encoder reranker fixed relevance but added 180ms. Here's when that trade pays off.
A prompt tweak that helped one case quietly broke twenty others. Here's the CI eval harness we built so that never ships silently again.
Users kept asking the same questions in slightly different words, and we paid full price every time. Semantic caching cut our LLM bill by a third.
Pure vector search kept missing exact matches like error codes and CLI flags. Adding BM25 back and fusing the two lifted our retrieval recall by 11 points.
Our support bot kept citing half a sentence and missing the answer that sat two lines below. The culprit wasn't the model, it was how we split the docs.
They solve different problems. RAG injects knowledge; fine-tuning changes behavior. The decision criteria, the hybrid pattern, and what we'd do over.
Token caching, model routing, prompt compression, and the boring discipline of measuring. The levers that cut our LLM bill 60% without touching feature scope.
