Feature Flags in Production: Decoupling Release From Deploy

Feature flags split deploy (code on servers) from release (behavior in front of users): ship the new path disabled, turn it on for a slice of users, and turn it off in seconds if it misbehaves. Use them for risky changes and kill switches, evaluate them locally, and delete each release flag within weeks of reaching 100%.

A common worst case is a checkout rewrite that passes every test and still takes payments down at peak. Deploy and release were the same event, so the only rollback was redeploying the previous build. Split them and a bad release becomes a config toggle. We now flag roughly every customer-facing change, and the operational reality is messier than the pitch: flag debt, evaluation latency, fail-open states, and who may flip what.

Ship dark, release later

Deploy the new code disabled, verify it runs clean, then turn it on separately:

if flags.is_enabled("new_checkout_flow", user=current_user):
    return new_checkout(cart)
return legacy_checkout(cart)

Old and new paths coexist in the same binary. Deploy Monday, release Thursday, and flip back in seconds if Thursday goes wrong.

What we use flags for, and what we don't

Gradual rollouts, about 80% of our usage. Release to a slice of users and watch between steps:

{
  "key": "new_checkout_flow",
  "rules": [
    { "segment": "internal_staff", "value": true },
    { "rollout": { "percentage": 5, "attribute": "user_id" } }
  ],
  "default": false
}

Internal staff first, because dogfooding catches the obvious breakage. Then 5% of real users, held for an hour while we watch checkout error rate and p99. If they hold, 5 to 25 to 50 to 100 over a day. If the same edge case shows up at 5%, the error spike is visible, the flag goes off and the fix lands without a redeploy: a full outage becomes a short blip for a small slice of users. Keying on user_id matters: a user gets the same experience on every request instead of flickering between old and new checkout.

Kill switches, about 10%. A flag any on-call engineer can flip without a deploy or a code review, in the middle of the night, on code paths that call something flaky or expensive (payments retry logic, AI feature paths):

if flags.is_enabled("kill_switch_recommendations"):
    return []          # feature off, degrade gracefully
return recommendation_service.fetch(user)

When our recommendation service started timing out, on-call flipped this and page latency recovered at once.

Customer-specific overrides, about 10%. A customer needs a feature ahead of GA, or keeps one after deprecation. Targeting rules handle it.

We do not use flags for configuration values that aren't really toggles (env vars and ConfigMaps do that better), A/B experiments that need statistical rigor (that is an experimentation workflow, even when the same platform runs it), or permanent gating by subscription tier (that's billing logic). Not every change needs a flag either: a one-line copy fix behind a flag is pure cleanup debt. Flag what is risky, hard to reverse, or worth ramping.

Provider choice

We've run two providers in production:

LaunchDarkly. The most mature feature set and broadest SDK coverage, expensive at scale. Check how the bill scales with seats and usage before you commit.

GrowthBook. Open source, self-host or cloud, simpler model, with per-environment SDK connections and rules for targeting, percentage rollouts and experiments. We moved here for cost and simplicity.

Unleash. Similar shape, also open source. We didn't run it, but it's the alternative we'd consider.

Roll your own. A feature_flags table and a small SDK is enough for a small team. It gets real past ~30 flags or once you need consistent bucketing, targeting, audit logs and a UI for non-engineers.

In hindsight we should have started on GrowthBook; the migration cost was real.

Patterns that hold up

Naming. <area>.<feature>.<purpose>, like checkout.express-pay.enabled or payments.retry-v2.kill-switch. Searchable and hard to mix up.

One owner per flag. Usually the team that introduced it. Orphan flags from departed engineers have burned us.

Flag-on-by-default in code. The code default is "on" and the platform default is "off" during rollout. Once a flag is fully rolled out, removing the platform config doesn't break anything.

A removal ticket the day the flag is born. Each release flag is typed as temporary and gets a cleanup ticket due two weeks after it hits 100%. Unleash's own flag-type docs give release flags an expected lifetime of 40 days, which is a sane upper bound:

Flag: new_checkout_flow   Created: 2026-05-02
Type: release (temporary)
Remove-by: 2 weeks after 100% rollout
Cleanup ticket: PLAT-4419

Permanent operational flags (kill switches, ops toggles) are tagged separately and exempt.

The operational gotchas

Evaluation latency. Some SDKs make a network round-trip per check. We use SDKs that fetch flag state at startup and refresh it in the background, so isEnabled() is a local map lookup. Unleash's guidance puts it bluntly: your application shouldn't depend on the flag system being available.

Fail-open vs fail-closed. When the platform is unreachable, what's the default? We decide per flag, at creation. Kill switches fail open, so a platform outage doesn't leave the feature broken and unswitchable. New-feature flags fail closed: better not to ship than to ship a half-tested feature unsupervised.

Bucketing consistency. Switching providers changed the hash function; a user in the 50% bucket on LaunchDarkly might land at 30% on GrowthBook. We migrated users by hand for the dozen flags that needed continuity.

Flag dependencies. Encode "A only when B" logic in code, not in the platform.

Defaults that flip, flags that never leave. During a config migration a flag default flipped and resurrected a feature we'd "removed" a year earlier. Some of our oldest flags have sat at 100% for over a year because the code paths underneath differ subtly. A flag at 100% for six months is dead code wrapped in a config lookup, and removing it is real work that must be scheduled.

Cross-environment drift. Dev, staging and prod flag values are managed independently by default. Code worked in dev with the flag on and broke in prod where it was off. We now keep a synced flag spec per environment with explicit, documented overrides.

Auditing flag debt

A weekly report lists release flags older than 90 days that are not at 0% or 100%; each one gets cleaned up or explained. Each quarter, a 30-minute review covers the rest:

• Flags at 100% for more than 30 days: ready to remove?
• Owner check: is the listed owner still here?
• Coverage: is the flag still referenced in code, or was the path refactored away?

That catches three to five removable flags a quarter. No platform we've used reliably finds dead flag references in the codebase, so we grep for flags that no longer exist in the platform, and we export the audit log weekly for "who flipped what last month." Without this, flag count only grows. Pair flags with automated rollout checks, like canary analysis that auto-aborts on bad metrics, and the ramp watches itself.

Questions people ask

What does it mean to decouple deploy from release?

Deploy puts new code on servers; release exposes it to users. With a feature flag, the new code ships disabled, and you turn it on later for a percentage of users or all of them. If the release goes wrong, you flip the flag off in seconds instead of redeploying the previous build and waiting for the rollout to finish.

What is a kill switch feature flag?

A permanent flag that disables a risky code path, usually one calling a flaky or expensive dependency, without a deploy or code review. On-call flips it to stop the bleeding and the real fix ships later, written by rested people. Unleash lists kill switches as a permanent flag type. It should fail open and be exercised in drills.

How do you prevent feature flag debt?

Create a removal ticket when the flag is created, due a couple of weeks after 100% rollout. Give every flag an owner and an expiry date, tag permanent kill switches separately, and run a weekly report of flags stuck between 0% and 100%. Then grep the codebase for flag keys the platform no longer knows about.

The decision, concretely

• Deploys and releases are still the same event? Put risky, user-facing, hard-to-reverse changes behind flags first.
• Code calls a flaky dependency? Add a kill switch that fails open, and flip it in a drill so you know it works.
• Fewer than ~30 flags and no percentage rollouts? A database table is fine. Past that, use GrowthBook or Unleash; LaunchDarkly when you need its advanced targeting.
• Flag with no removal date? It isn't done. Create the cleanup ticket now.

The call we'd make

Adopt a hosted or self-hosted flag platform once you're past a handful of flags, split release from deploy for every risky customer-facing change, and decide fail-open vs fail-closed when each flag is created rather than during an incident. The caveat is debt: treat flag cleanup as part of the feature, not optional homework. The teams that get burned by flags aren't the ones who used them, they're the ones who never deleted them.

Sources

• Unleash Docs: Feature flags and flag types
• Unleash Docs: Feature flag best practices
• GrowthBook Docs: Feature flag basics
• GrowthBook releases on GitHub