13 articles tagged with Cost Optimization.
A million-token window doesn't retire your retrieval stack. Here's when to stuff the prompt, when to retrieve, and when to do both.
Moving our fleet from x86 to Graviton promised 20% savings. We got 31%, but only after fixing native dependencies, a broken base image, and one nasty perf regression.
A 180k-token context window is not a license to stuff everything in. Here's how we cut prompt size 60% without hurting answer quality, and what to trim first.
A p99 that jumped to 3.4 seconds during traffic ramps turned out to be cold starts. Here's how we measured them properly and cut the tail, with real init timings.
When our single LLM provider had a 40-minute outage, every AI feature went dark. A gateway with routing and fallback fixed that, and cut spend 30% as a bonus.
We moved 40 TB of user media off S3 and cut the bill by 70 percent, mostly by killing egress fees. Here's where R2 won and where we kept S3 anyway.
A single NAT Gateway quietly billed us $2,900 in one month, mostly for data processing on traffic that never needed to leave the VPC. Here's how we found it and cut it.
Our S3 bill tripled in a month with no growth in stored data. The storage line was flat. The cost was in requests and a misconfigured lifecycle rule quietly shredding money.
Users kept asking the same questions in slightly different words, and we paid full price every time. Semantic caching cut our LLM bill by a third.
A long, stable system prompt re-billed on every request is money on fire. How prompt caching works, where the cache boundary belongs, and the structuring discipline that got us a big cost and latency cut without changing behavior.
Three discounting mechanisms, three different commitments. The rules of thumb we use to pick, and the mistakes we made before settling on them.
Token caching, model routing, prompt compression, and the boring discipline of measuring. The levers that cut our LLM bill 60% without touching feature scope.
