Worked at a place with a literal settings table of 5K (non-default) records, it was many hundreds of possible settings.
I also seen condensed flags where the flag record embedded which IDs were activated, with a massive overwrite risk anytime someone touched any setting.
Also flag-averse modules where there was so much magic that one one could ever understand exactly what would happen until they loaded up a real or similarly structured set of records. And they didn't trust their results for more than a month or so because changes were frequent as things needed to evolve. The case for this craziness is that too many flags means folks will overlook things, or we'll forget to make the new shiny on-by-default after the roll out period.
IME there is a balance between every feature and code path getting a flag and nothing ever does. Of course the flags themselves introduce complexity and risk. And there's the work to remove them with the vestiges and QA that change for regressions.
"At Google, our philosophy is that “rollbacks are normal.” When an error is found or reasonably suspected in a new release, the releasing team rolls back first and investigates the problem second"
Well this all makes sense now. My enterprise laid off most of our QA team and I see a constant stream of rollback announcements. Who needs regression testing anymore?
Feature flags are great. Working on a crash / memory corruption in an optional subsystem? Just disable the subsystem to unblock coworkers on the same branch while you track down the cause.
Feature flags are terrible. Working on a crash / memory corruption in an optional subsystem? You disabled it previously for your local build, and you'll lose hours failing to repro despite QA giving excellent repro steps.
(For my own gamedev background, I learned to mute audio by setting volume to 0 instead of by disabling the audio subsystem.)
I agree with the entire premise but I do think the cost argument is a bit overblown. Adding "unnecessary" feature flags isn't really that big of a deal imo, feature flags are cheap to add and maintain. Also sometimes flipping feature flags can be faster than doing a rollback, especially if multiple systems are involved.
I think the true cost is that feature flags can cause code bloat and readability issues, since engineers typically aren't great about cleaning up feature flags after things have been rolled out. I think that's an easily solvable problem that doesn't really necessitate a scarcity mindset of "just use less feature flags / only when necessary" though. LaunchDarkly makes it pretty easy to track feature flag usage and remind people to clean up old ones.
> (...) feature flags are cheap to add and maintain.
Not quite. When you add a feature flag, you now have a system that has N separate code flows that you need to verify before launch,and then have to modify and retest to remove the feature flag. A feature flag ends up doubling the workload to test and verify the feature.
> Also sometimes flipping feature flags can be faster than doing a rollback, especially if multiple systems are involved.
This is perhaps the most relevant point that the blogger misses. Feature flags are runtime switches that can extend simultaneously to multiple systems and even clients. You can't pull features from clients with a release, where week-long release processss can barely get you a 60% uptake. Redeploying away features in the backend can easily take half an hour, specially if it's a system that does multi region rolling deployments.
If you want instant switches, your best option is switches flipped at runtime. That's what a feature flag is.
> I liked the article. Every feature flag contributes to hockey stick growth of version variations of your software.
This is specious reasoning. When you add a feature flag, your goal is to purposely introduce version variations. How come is your explicit goal framed as this sneaky gotcha?
And let's be serious for a second. Do you see this "hockey stick growth" as an issue when you look at user settings?
> So cognitive overload is unavoidable after feature flags 5 as you have so many permutations.
Not really. Feature flags have a life cycle which is managed by release managers, and the goal is to always get rid of them asap. They are introduced to manage the risk of a rollout, and when the transition is over you pull the feature flags. Done.
That's a good point about not using feature flags to mitigate risk, and how rollbacks are a better alternative. Teams need to be in the habit of performing a rollback though. Sometimes, the rollback process can be black magic if the engineer handling an incident isn't familiar with that process. Having a bunch of flags in a system is a great way to end up with nondeterministic errors.
And that brings us to another great point, which is too many flags is problematic. So often there's an excuse made in the nature of, "we'll go back and remove this later," but later never comes.
At the end of the day, it's rigor that separates good teams from bad teams. Good teams will rigorously review old code and remove it; it's all too easy to do the opposite.
> That's a good point about not using feature flags to mitigate risk, and how rollbacks are a better alternative. T
I doubt anyone making this sort of claim has any professional experience maintaining any sort of user facing software.
Features that require cross-system support can't easily be pulled out with a revert, particularly in CICD systems where cherry picking a revert can easily be incomplete/miss a bug fix and reverting the whole history will end up inadvertently pulling out some other feature.
Also, a deployment takes time, particularly in services with more than one deployment region. Moreso if n>1 separate systems are involved.
In comparison, runtime config changes are enforced in a fraction of the time, and pulling out the code just involves a roll forward. Simple.
Another issue is launching behind feature flags often skips integration/e2e testing, which couldn't possibly test with every combination of every flag. So your release which is "safer" behind a flag is actually untested until it's launched.
> So your release which is "safer" behind a flag is actually untested until it's launched.
It's only untested if your intention is to not test it.
There is nothing preventing you from testing a feature flag in e2e tests. In fact, testing is perhaps the primary reason why user overrides are supported by feature flag systems.
> Risk management should indeed be a priority for all teams. But there are better ways of doing this than relying on feature flags, especially if your team has control over its own deployments. The vast majority of your bugs should be caught by your automated test suite and/or QA process. And the last few stragglers should be handled using incremental deployments, production alarms and rollbacks.
There is some confusion in this post. For risk management, feature flags are used to mitigate the risk of issues not being caught by the QA process. Wishing for a QA process to be 100% flawless doesn't make it so.
Also, guess what an incremental deployment is.
I think the blogger tried to make sweeping statements but didn't actually spent any time thinking through the arguments.
As an SRE / DBRE, I hate FFs because it means I can be lulled into a false sense of security when something I saw didn’t initially fail, only to start doing so the next week.
Worked at a place with a literal settings table of 5K (non-default) records, it was many hundreds of possible settings.
I also seen condensed flags where the flag record embedded which IDs were activated, with a massive overwrite risk anytime someone touched any setting.
Also flag-averse modules where there was so much magic that one one could ever understand exactly what would happen until they loaded up a real or similarly structured set of records. And they didn't trust their results for more than a month or so because changes were frequent as things needed to evolve. The case for this craziness is that too many flags means folks will overlook things, or we'll forget to make the new shiny on-by-default after the roll out period.
IME there is a balance between every feature and code path getting a flag and nothing ever does. Of course the flags themselves introduce complexity and risk. And there's the work to remove them with the vestiges and QA that change for regressions.
"At Google, our philosophy is that “rollbacks are normal.” When an error is found or reasonably suspected in a new release, the releasing team rolls back first and investigates the problem second"
Well this all makes sense now. My enterprise laid off most of our QA team and I see a constant stream of rollback announcements. Who needs regression testing anymore?
Feature flags are great. Working on a crash / memory corruption in an optional subsystem? Just disable the subsystem to unblock coworkers on the same branch while you track down the cause.
Feature flags are terrible. Working on a crash / memory corruption in an optional subsystem? You disabled it previously for your local build, and you'll lose hours failing to repro despite QA giving excellent repro steps.
(For my own gamedev background, I learned to mute audio by setting volume to 0 instead of by disabling the audio subsystem.)
I agree with the entire premise but I do think the cost argument is a bit overblown. Adding "unnecessary" feature flags isn't really that big of a deal imo, feature flags are cheap to add and maintain. Also sometimes flipping feature flags can be faster than doing a rollback, especially if multiple systems are involved.
I think the true cost is that feature flags can cause code bloat and readability issues, since engineers typically aren't great about cleaning up feature flags after things have been rolled out. I think that's an easily solvable problem that doesn't really necessitate a scarcity mindset of "just use less feature flags / only when necessary" though. LaunchDarkly makes it pretty easy to track feature flag usage and remind people to clean up old ones.
> (...) feature flags are cheap to add and maintain.
Not quite. When you add a feature flag, you now have a system that has N separate code flows that you need to verify before launch,and then have to modify and retest to remove the feature flag. A feature flag ends up doubling the workload to test and verify the feature.
> Also sometimes flipping feature flags can be faster than doing a rollback, especially if multiple systems are involved.
This is perhaps the most relevant point that the blogger misses. Feature flags are runtime switches that can extend simultaneously to multiple systems and even clients. You can't pull features from clients with a release, where week-long release processss can barely get you a 60% uptake. Redeploying away features in the backend can easily take half an hour, specially if it's a system that does multi region rolling deployments.
If you want instant switches, your best option is switches flipped at runtime. That's what a feature flag is.
I liked the article. Every feature flag contributes to hockey stick growth of version variations of your software.
1 feature flag = 2 behaviors 2 feature flags = 4 behaviors
So cognitive overload is unavoidable after feature flags 5 as you have so many permutations.
> I liked the article. Every feature flag contributes to hockey stick growth of version variations of your software.
This is specious reasoning. When you add a feature flag, your goal is to purposely introduce version variations. How come is your explicit goal framed as this sneaky gotcha?
And let's be serious for a second. Do you see this "hockey stick growth" as an issue when you look at user settings?
> So cognitive overload is unavoidable after feature flags 5 as you have so many permutations.
Not really. Feature flags have a life cycle which is managed by release managers, and the goal is to always get rid of them asap. They are introduced to manage the risk of a rollout, and when the transition is over you pull the feature flags. Done.
That's a good point about not using feature flags to mitigate risk, and how rollbacks are a better alternative. Teams need to be in the habit of performing a rollback though. Sometimes, the rollback process can be black magic if the engineer handling an incident isn't familiar with that process. Having a bunch of flags in a system is a great way to end up with nondeterministic errors.
And that brings us to another great point, which is too many flags is problematic. So often there's an excuse made in the nature of, "we'll go back and remove this later," but later never comes.
At the end of the day, it's rigor that separates good teams from bad teams. Good teams will rigorously review old code and remove it; it's all too easy to do the opposite.
> That's a good point about not using feature flags to mitigate risk, and how rollbacks are a better alternative. T
I doubt anyone making this sort of claim has any professional experience maintaining any sort of user facing software.
Features that require cross-system support can't easily be pulled out with a revert, particularly in CICD systems where cherry picking a revert can easily be incomplete/miss a bug fix and reverting the whole history will end up inadvertently pulling out some other feature.
Also, a deployment takes time, particularly in services with more than one deployment region. Moreso if n>1 separate systems are involved.
In comparison, runtime config changes are enforced in a fraction of the time, and pulling out the code just involves a roll forward. Simple.
Another issue is launching behind feature flags often skips integration/e2e testing, which couldn't possibly test with every combination of every flag. So your release which is "safer" behind a flag is actually untested until it's launched.
> So your release which is "safer" behind a flag is actually untested until it's launched.
It's only untested if your intention is to not test it.
There is nothing preventing you from testing a feature flag in e2e tests. In fact, testing is perhaps the primary reason why user overrides are supported by feature flag systems.
how? if you have 100 feature flags in your system each with 3 potential states, which states do you run in your e2es?
> Risk management should indeed be a priority for all teams. But there are better ways of doing this than relying on feature flags, especially if your team has control over its own deployments. The vast majority of your bugs should be caught by your automated test suite and/or QA process. And the last few stragglers should be handled using incremental deployments, production alarms and rollbacks.
There is some confusion in this post. For risk management, feature flags are used to mitigate the risk of issues not being caught by the QA process. Wishing for a QA process to be 100% flawless doesn't make it so.
Also, guess what an incremental deployment is.
I think the blogger tried to make sweeping statements but didn't actually spent any time thinking through the arguments.
As an SRE / DBRE, I hate FFs because it means I can be lulled into a false sense of security when something I saw didn’t initially fail, only to start doing so the next week.
Just use canaries, I’m begging you.
Canaries meaning deploying the new stuff only to certain segments first?
I imagine that's got its own risks and challenges if there is a lot of bottlenecks or dependencies, like multi-tenant data store(s).
good luck rolling back when 20 teams have been waiting for your pipeline to be unblocked for 2 weeks