And, no, I’m not referring to LaunchDarkly’s eye-watering pricing model.
I’ve worked on backend engineering for three cloud platforms: AWS (EBS), HashiCorp Cloud Platform, and MongoDB Atlas. So naturally I’ve seen five+ different feature flag implementations, with varying attitudes towards their usage.
Many engineers are, in my opinion, too pro-feature flag. Sometimes a PR’s one and only comment is: “this looks like it changes behavior, could you please add a feature flag?” And saying “no” is tantamount to saying “eh, no, I’m feeling lucky, and, also, my code is perfect, can’t you see that?” Not adding a feature flag is amateurish like skipping tests is amateurish. It’s taken for granted that flags improve safety.
I’ve seen developers ask for feature flags for:
- updated DB queries (e.g. to use a new index)
- new RPCs in Cadence Activities that already have dozens
- re-ordered rollback behavior
But I think feature flags’ benefits are vastly overstated and their costs completely ignored. Despite claims to the contrary, they actually slow developer speed and efficiency, they increase code complexity, and they increase the cost of testing. I’m not even sure they improve reliability — especially if feature flags become a crutch disincentivizing building better CI/CD pipelines.
Cost #1: more to code
Feature flags disconnect deployment from release, enabling developers to ship code faster… Feature flags help developers merge and integrate their changes more often. — PostHog
One refrain is that feature flags speed up development. But feature flags are config. They’re real. They exist in your YAML, JSON, HCL, or UI. When you add one, your medium-sized feature is no longer one medium-sized PR. Because it comes with an upstream requirement: you instantiate the flag in some JSON/YAML/HCL/UI. Then you copy that flag’s name over to your PR and wire it in. Your PR has an extra upstream task: create the flag. Then you also need to enable the flag in each environment or — if you’re unlucky — region-by-region/cell-by-cell. So instead of one PR you’ve got many.
Then you need to document that behavior-modifying flag you just added. Sure, future readers can grep the repo and find the flag during an incident… but if you’re a friendly and cooperative operator, you should reference that flag in a runbook and/or dashboard.
Finally, there’s the LOC. That flag lives on in your code. It has new helpers, if-statements, guard-clauses. It’s not going away. You’re going to release that medium-sized feature, and 50 more, before running across the flag you made 2 years ago and thinking, “huh, I guess I should clean this crap up.”
Since feature flags require more setup, documentation, and babysitting, they slow development. If the goal were to ship faster, we’d skip them entirely.
Cost #2: more to test
feature toggles in software development provides a number of benefits, including… Cost savings by reducing the time spent on manual testing and debugging processes. — harness
I get what they’re saying: “test in prod.” Which, yeah, sure. But they’re glossing over: there’s much more to test. The more feature flags, the more branching the logic. A simpler service, cron job, or library is simpler to test. Add a few boolean feature flags, and you’re up to 8 branches of behavior.
For example, in the (won’t-compile) code below, we create a resource in an upstream dependency. Depending on the feature flags’ values, we may or may not:
- publish an event (e.g. to an external audit log service)
- call the upstream with retries
- call a new V2 version of the upstream’s endpoint
func (c *client) create(req *Req) (resp *Resp) {
if c.launchDarkly.BoolVariation("publish.event") {
defer c.eventSvc.publishEvent(resp)
}
retry := grpc_retry.WithMax(0)
if c.launchDarkly.BoolVariation("call.with_retries") {
retry = grpc_retry.WithMax(3)
}
if c.launchDarkly.BoolVariation("call.v2") {
return c.resourceSvc.createV1(req)
}
return c.resourceSvc.createV2(req)
}Some new risks from the feature flags: maybe createV1 is idempotent where createV2 isn’t — so retries in the V2 calls create orphaned resources; perhaps V2’s response has a field the eventSvc requires (for mapping) while V1’s doesn’t — so disabling the publish.event flag breaks publishing to the event service.
If combinations of flag values are likely in production, you should test their permutations. But that has two downsides:
- combinatorial explosion
- wasted time/resources testing states you’ll actually never reach in prod (you’re going to enable all the flags then leave them enabled)
In practice, I see folks just skip to enabling new flags in testing — which is better than forgetting to enable them… — but now your tests don’t match prod (until you’ve enabled them in prod) and you’ve lost coverage for the flag-disabled branches (that might suddenly be tested in production when you disable them).
Cost #3: less investment in CI/CD pipelines
The proponents of feature flags tout their ability to improve reliability (turn off broken features quickly), speed up developer velocity (no feature branches), and reduce time-spent testing (test in prod). As stated, I think they do the opposite: more code == (more boilerplate && more liability).
There is an actual developer tool that improves velocity, improves reliability, and reduces time spent manually testing: well-designed CI/CD pipelines. Feature flags are a poor substitute. And I think feature flags’ biggest cost is in disincentivizing building them.
In my experience, the teams with lots of feature flags do it out of necessity because their CI/CD pipeline sucks. I think it happens for two reasons:
- Rollbacks and reverts are painful. If fixing a bug in production requires backporting a commit to a release branch, then asking/paging a release manager to manually deploy it, you will always add flags.
- It’s quicker than expanding tests or monitoring. If I can manually disable a flag after noticing a feature degrading user experience, I can roll the dice and skip adding test coverage (e.g. canaries) or monitoring (e.g. automatic rollback monitors) to a CI/CD pipeline that would otherwise have prevented the change from affecting users in the first place.
If you have a safe CI/CD pipeline, you can test in production without a feature flag: add the feature alongside (or after) its testing and monitoring. If that sounds insufficient, you’re missing test coverage or pipeline features like automatic rollbacks or canary workloads. Further, if you have a decent release process, you don’t need a feature flag to disable a feature, you can roll back.
Caveats
- Feature flags are the perfect solution for gating pre-GA features to a subset of users. That’s unfortunately not how I’ve seen them used most.
- Some dynamic config is inevitable and valid. For example, knobs to give some customers higher quotas, e.g. 100 rps instead of 10.
- I work on cloud platform backends. I realize other engineers may have more legitimate use cases.