We used to ship code and turn it on in the same breath, so every deploy was a bet. Feature flags split those two events apart and made rollbacks a config toggle.
The worst deploy of my career was a checkout rewrite that passed every test and then took payments down for 22 minutes at peak. The code was fine in staging. Production traffic hit an edge case staging never had, and because deploying the code and turning the feature on were the same event, our only rollback was a full redeploy of the previous build, which itself takes eight minutes to roll out. We watched revenue drain while a container image slowly rolled back.
That incident is the entire argument for feature flags. Deploy is a technical event, getting code onto servers. Release is a business event, exposing behavior to users. When those are the same action, every deploy is a bet with no cheap way out. Split them and a bad release becomes a config toggle, not a redeploy.
The pattern is simple: deploy the new code disabled, verify it's running clean, then turn it on separately. The new checkout ships behind a flag defaulting to off. It's in production, taking no traffic, until we decide.
if flags.is_enabled("new_checkout_flow", user=current_user):
return new_checkout(cart)
return legacy_checkout(cart)
The old and new paths coexist in the same binary. Deploy touches servers on Monday, release happens Thursday when we choose, and if Thursday goes wrong we flip the flag back in seconds without a redeploy. That's the whole game.
Flags let you release to a slice of users, not everyone at once. We ramp new risky features by percentage and watch the metrics between steps.
{
"key": "new_checkout_flow",
"rules": [
{ "segment": "internal_staff", "value": true },
{ "rollout": { "percentage": 5, "attribute": "user_id" } }
],
"default": false
}
Internal staff first (dogfooding catches the obvious breakage), then 5% of real users. We hold at 5% for an hour watching checkout error rate and p99. If they hold, we go 5 to 25 to 50 to 100 over a day. The rewrite that once took us down? On the redo, the edge case showed up at 5% affecting a handful of users, we saw the error spike, flipped it off, fixed it, and no customer at scale ever hit it. Same bug, 22-minute outage became a 4-minute blip for a few dozen people.
Keying the rollout on user_id matters, it means a given user consistently gets the same experience across requests, instead of flickering between old and new checkout on every page load.
Every risky feature gets an operational kill switch, a flag any on-call engineer can flip without a deploy, without a code review, in the middle of the night. This is different from a rollout flag; it exists purely so a human can stop the bleeding fast.
if flags.is_enabled("kill_switch_recommendations"):
return [] # feature off, degrade gracefully
return recommendation_service.fetch(user)
When our recommendation service started timing out and dragging page loads with it, on-call flipped the kill switch and page latency recovered instantly. The underlying fix shipped the next morning during business hours by rested people. A kill switch turns a 3am firefight into a "we'll deal with it tomorrow."
Here's what the enthusiastic blog posts skip: flags rot. A flag that's been at 100% for six months isn't a flag, it's dead code with a config lookup wrapped around it, and enough of them turn your codebase into a maze of stale conditionals. We once had a flag default flip during a config migration and resurrect a feature we'd "removed" a year earlier.
We treat flags as temporary by default. Each rollout flag gets a removal ticket created the day it's born, due two weeks after it hits 100%.
Flag: new_checkout_flow Created: 2026-05-02
Type: release (temporary)
Remove-by: 2 weeks after 100% rollout
Cleanup ticket: PLAT-4419
Permanent operational flags (kill switches, ops toggles) are tagged separately and exempt. Everything else has an expiry. We run a weekly report of flags older than 90 days still not at 0% or 100%, and those get cleaned up or explained. Without that discipline, the tool that made you safe slowly makes your code unreadable.
A counterweight: not every change needs a flag. Wrapping a one-line copy fix in a flag adds a branch, a config entry, and cleanup debt for zero benefit. We flag things that are risky, hard to reverse, or worth rolling out gradually. A typo fix just ships. Over-flagging is its own mess, you end up with hundreds of flags and no idea which combinations are even tested.
If your deploys and releases are the same event, adopt flags for your risky, user-facing, hard-to-reverse changes first, and build a real kill switch into anything that calls a flaky dependency. But treat flag cleanup as part of the feature, not optional homework, or you'll trade deploy risk for a codebase full of zombie conditionals. The teams that get burned by flags aren't the ones who used them, they're the ones who never deleted them.
Get the latest tutorials, guides, and insights on AI, DevOps, Cloud, and Infrastructure delivered directly to your inbox.
Explore more articles in this category
Our best engineer quit citing on-call. We rebuilt the whole thing: saner rotations, runbooks that actually help at 3am, and escalation that doesn't punish asking for help.
Our early postmortems quietly assigned blame and taught people to hide mistakes. Here's the template and the facilitation rules that finally made them honest and useful.
Most SLI dashboards track things nobody notices. Here's how we picked the handful of signals that map to real user pain, and dropped the vanity metrics.
Evergreen posts worth revisiting.