The feature flag nightmare: a QA perspective.
My former boss, Jerome Dane, wrote a piece I keep coming back to: Feature Flags are Dangerous. His argument is from the engineering and architecture seat: flags blur what's actually running in production, leave landmines of dead code, create multiple code paths, and quietly drift between environments and caches.
I want to add the view from the chair next to his, the QA chair. Everything he says is true on the way in, and it gets worse on the way out. If feature flags are dangerous to write, they are pure hell to test.
The part that doesn't fit in a standup update is that every independent flag doubles the test surface. It doesn't add to it, it doubles it.
The math nobody wants to hear
A single on/off flag means a feature has two states. Two flags, four. Three, eight. The number of distinct configurations is 2ⁿ, and exponential growth gets out of hand faster than intuition expects.
Ten flags (a number any real product blows past) is 1,024 distinct configurations of the same codebase. No QA team on earth is running 1,024 regression passes. So they don't. They test a couple of configurations and ship the rest blind.
This is what that looks like for just three flags:
It's worse than 2ⁿ in the real world
The clean 2ⁿ number assumes flags are independent and the only thing that varies is the flag itself. Neither is true.
Flags interact: flag B behaves differently depending on whether flag A is on, which is exactly the kind of bug that hides in the combinations nobody ran. And flag state is only one axis. The same matrix has to be re-run across the things Jerome warned about: behavior differs across environments, and again across caches (a flag value cached locally can disagree with the distributed source of truth). Add the user roles or segments most B2B products carry, and the real number is a product of all of them.
With only three flags you're already looking at nearly 200 meaningfully different configurations. This is why "it worked in staging" is not a sentence QA can trust when flags are involved: staging was one cell in a very large grid.
What gets tested vs. what ships
The gap between what a developer verified and what actually reaches users is where flag bugs live.
- The flag on, happy path
- On their machine, one environment
- As an admin / their own account
- With a fresh cache
- Every reachable combination of flags
- Across every environment
- Across every role and segment
- With stale, warm, and cold caches
The developer flipped one flag on and saw their feature work. Then a customer hits it with a different flag also on, in production, as a non-admin, behind a stale cache. That's a cell of the grid no one ever opened. To them it's just a bug. To Jerome's point, this is "testing in production" whether you meant to or not.
Why automation doesn't rescue you
The instinct is "we'll just parametrize the tests across flag states." You can, but the matrix is still exponential, so you get to choose between two bad outcomes: run every combination and watch CI time explode past anything usable, or pick a subset and accept the gaps. The test runner has no free lunch hiding in it.
The pragmatic middle ground is combinatorial (pairwise) testing: instead of all 2ⁿ combinations, generate the much smaller set that covers every pair of flag values at least once. It catches the large class of bugs that come from two flags interacting, at a fraction of the cost. It's a mitigation and won't catch a bug that only appears when three specific flags line up, but it's the honest tool for the job.
Take four on/off flags. Exhaustive testing is 2⁴ = 16 runs. Pairwise needs five:
| Run | Flag A | Flag B | Flag C | Flag D |
|---|---|---|---|---|
| 1 | off | off | off | off |
| 2 | off | on | on | on |
| 3 | on | off | on | on |
| 4 | on | on | off | on |
| 5 | on | on | on | off |
Pick any two flags (say B and D) and all four combinations (off/off, off/on, on/off, on/on) appear somewhere in those five rows. That holds for all six pairs. Five is also the floor: no four-row set covers every pair for four flags. The gap widens fast: ten on/off flags is 1,024 exhaustive runs, but every pair can be covered in as few as six.
You don't build these tables by hand. Tools like Microsoft PICT take a list of parameters and values (flags, environments, roles) and generate the pairwise set for you, including constraints like "flag C only exists when flag A is on."
What actually helps, from the QA seat
Jerome's prescription (fewer flags, branches over toggles, small reversible releases, a fast pipeline) is also the best testing strategy, because every one of those choices shrinks the grid. A few things I'd add from the testing side:
A few principles behind that flow:
Treat every live flag as a test parameter. If a flag exists, its states belong in the test plan: default-tested, with pairwise coverage across the rest. A flag that isn't in the test matrix is a flag you're shipping blind.
Kill flags aggressively. Every flag you remove halves the surface you just saw explode. A sunset date on each flag is the single most useful testing decision a team can make, and it costs nothing.
Make flag state observable. When a bug report comes in, the first question is "which flags were on?" If your logs answer that, a failure is reproducible. If they don't, you're debugging one unknown cell in a 200-cell grid.
Pin flag state in test environments. Tests that run against whatever the flag service happens to return today are closer to weather reports. Fix the flag values per test run so a green result means something.
Fewer cells in the grid
Feature flags move risk off the developer's machine and into the tester's matrix, which grows exponentially with every toggle added. Jerome is right that they're dangerous to write. From where I sit, the kindest thing any team can do for the people testing their software is to have fewer flags, register the ones they keep as real test parameters, and kill them fast.