Practical articles on AI, DevOps, Cloud, Linux, and infrastructure engineering.
We rotated a leaked AWS access key that a workflow had committed to logs. Switching GitHub Actions to OIDC federation meant no static AWS keys exist to leak in the first place.
A user got our support bot to recite its system prompt and then draft a refund it wasn't authorized to give. Two layers of guardrails, one on input, one on output, closed both holes.
Static service tokens leaked into logs and never rotated. SPIFFE identities plus SPIRE-issued SVIDs gave us short-lived certs and killed the shared-secret sprawl.
A single ALTER TABLE took a lock and stalled every write for 40 seconds during peak traffic. Expand-contract is how we stopped shipping outages.
Our RAG answers kept citing the wrong paragraph even when the right one was retrieved. A cross-encoder reranker fixed relevance but added 180ms. Here's when that trade pays off.
Adding a read replica cut primary load 60%, then support tickets rolled in about users not seeing their own edits. Replication lag turned into a correctness bug we had to route around.
Our proxy topped out at 40k connections while the CPU sat half-idle. The bottleneck was kernel defaults tuned for 2009, not the hardware.
Our node image shipped 240 CVEs, most from OS packages we never called. Moving to distroless dropped the count to single digits and cut image size by 70%.
Users kept asking the same questions in slightly different words, and we paid full price every time. Semantic caching cut our LLM bill by a third.
Our M-series laptops built arm64, our CI built amd64, and prod pulled whichever tag won the race. Buildx and a manifest list ended the chaos.
A prompt tweak that helped one case quietly broke twenty others. Here's the CI eval harness we built so that never ships silently again.
The dashboard said the database was fine. It wasn't. Here's how pg_stat_statements found the query eating 40% of our Postgres CPU.
