36 articles tagged with Reliability.
A single Availability Zone in AWS's Spain region degraded for roughly 2.5 hours, and most "multi-AZ" architectures would have failed anyway.
A slow database made a backend service look dead, and the health check that was supposed to protect it killed it faster. The lesson has nothing to do with AI.
Handing an agent kubectl is a five-minute job. Proving the fix worked and did no harm is the real work, and it belongs in the wrapper, not the prompt.
A green readiness probe means the probe endpoint answered, nothing more. The gaps that cause 502s during rollouts, and the Deployment settings that close them.
AWS says it cannot restore data held only in Bahrain (me-south-1) or in one UAE zone. Multi-AZ gave availability, not recovery, and only cross-region copies survived.
A background cleanup job saturated the database behind GitHub's permission checks, and the safeguard pacing it watched only replica lag. Throttle on more than one signal.
ChatGPT, Claude, and Grok degraded together when Azure East US failed. Gemini stayed up. Multi-provider failover does not help when your providers share a substrate.
A practical guide to dead-letter queues, the pattern that isolates poison messages so one bad payload can't stall your whole pipeline.
Our best engineer quit citing on-call. We rebuilt the whole thing: saner rotations, runbooks that actually help at 3am, and escalation that doesn't punish asking for help.
Reliability arguments used to be shouting matches between SRE and product. An error budget turned them into arithmetic. Here's how we made the number drive the roadmap.
When our single LLM provider had a 40-minute outage, every AI feature went dark. A gateway with routing and fallback fixed that, and cut spend 30% as a bonus.
Our failover config looked perfect in the console and did nothing during a real outage. Here's the health-check design that actually flipped regions when it mattered.
