DevOps Engineer
376 articles published
Adding a read replica cut primary load 60%, then support tickets rolled in about users not seeing their own edits. Replication lag turned into a correctness bug we had to route around.
Static service tokens leaked into logs and never rotated. SPIFFE identities plus SPIRE-issued SVIDs gave us short-lived certs and killed the shared-secret sprawl.
Our proxy topped out at 40k connections while the CPU sat half-idle. The bottleneck was kernel defaults tuned for 2009, not the hardware.
A single ALTER TABLE took a lock and stalled every write for 40 seconds during peak traffic. Expand-contract is how we stopped shipping outages.
When a service is slow and every dashboard looks green, bpftrace lets you watch the kernel directly. These one-liners found our tail latency.
We had long-lived AWS keys sitting in a datacenter we don't own. IAM Roles Anywhere let us delete every one of them. Here's the real setup.
The dashboard said the database was fine. It wasn't. Here's how pg_stat_statements found the query eating 40% of our Postgres CPU.
Users kept asking the same questions in slightly different words, and we paid full price every time. Semantic caching cut our LLM bill by a third.
Our M-series laptops built arm64, our CI built amd64, and prod pulled whichever tag won the race. Buildx and a manifest list ended the chaos.
A prompt tweak that helped one case quietly broke twenty others. Here's the CI eval harness we built so that never ships silently again.
A cron job silently stopped running for three weeks and nobody knew until the backups were missing. systemd timers give you the logging and status cron never did.
A 900GB events table where every query scanned four years to read one day. Range partitioning by month cut our dashboard queries from 8s to under 200ms.
