Feature Flags in DevOps

In our previous article A Fresher on Testing in CI/CD, we mentioned that feature flags can be used in Canary Testing. We’ll spend some time to talk more about this technique in this article.

What is Feature Flag

Feature Flag is a technique that enables DevOps teams to turn certain features on and off during runtime. It doesn’t require redeploy new code or roll back to previous building versions. Feature Flag is also known as Feature Toggle or Feature Switch.

In addition to the names mentioned above, Feature Management is also part of the gang. To distinguish these two terms, we can consider Feature Management as a practice and Feature Flag as a technique. Or we can also consider Feature Management as a practice to manage Feature Flag’s lifecycle.

Use Cases

In this section we’ll use go through the feature flag use cases along the DevOps lifecycle, as shown in the diagram below.

Trunk-Based Development

In the Dev bucket of the DevOps lifecycle, there are two popular development styles: Git Flow and Trunk-Based Development. Git Flow practices usually come with quite a number of branches, e.g. master branch, release branch, develop branch, feature branch, etc. It requires multiple pull requests and approvals before a piece of code graduates to production. It can be quite a lengthy, complex and even bureaucracy journey. Due to this reason, it often drives to situations like long-lived feature branches, merge conflicts and aging pull requests.

Trunk-Based Development on the other hand encourages short-lived feature branches. It’s like a lean version of a tree that only has the trunk (e.g. the master branch) and some short and direct connected branches. Developers can merge their code directly to the master branch once their code are “ready”. We’ll explain why we put quotation marks around the word “ready”.

The Trunk-Base Development seems to make the life a lot simpler. However, there is a risk comes with it. The assumption is that the code from the short-live branches compiles and passes all tests before they are merged to the master branch. What if they didn’t? So it comes back to the interpretation of the word “ready” we were talking earlier. It could be that you subjectively think it’s ready but it’s objectively not ready. Without extra layers of branches as protection, like Git Flow, how can we protect the “trunk” from the imperfect code?

Feature Flag provides a good answer to this question. Feature flags enable developers to add an extra gate of protection by wrapping their new code under flags. Sometimes people describe feature flags as a non-negotiable requirement for trunk-based development. It nicely compliment trunk-based development and encourage small batch updates.

Example

Below is an example of coding with feature flags.

//C# example
bool featureFlag = featureFlagRule();

if (featureFlag) {
// Run code in this section if featureFlag value is true
} else {
// Run code in this section if the featureFlag value is false
}

This is a simple IF statement for getting a taste of the feature flag. In real life application of feature flags, the complexity of feature flag decision trees may vary. For more information about the Feature Flag SDK and Libraries, see The Hub for Feature Flag Driven Development.

Continuous Deployment

To explain how feature flags streamline the CD processes, let’s take a half step back by looking at the differences between Continuous Delivery and Continuous Deployment. For questions like “What CD within CI/CD stands for”, we often answer with either Continuous Delivery or Continuous Deployment. The common explanation about the difference between these two are like below.

In Continuous Delivery, the deploy to production part is a manual process, whereas Continuous Deployment automates all the way through. The automate-all-the-way-through process seems to be more streamlined. However, it may raise concerns in scenarios like companies who have well-established product or large development teams with a lot of junior developers. The production environment is less tolerant for imperfect code.

With feature flags introduced in development, it enables this final automation part. DevOps teams can continuously deploy solutions to their production environment with no sweat. The production users won’t be able to interact with the flagged code.

In addition, comparing with the big bang deployment from a long-lived feature branches, DevOps teams can gradually roll out new code and bug fixings from their short-lived branches.

Continuous Testing

As mentioned in the A Fresher on Testing in CI/CD article, testing should not be a standalone stage in the DevOps lifecycle. Instead, it should be integrated into each stage, including testing in production. Canary Testing is a popular testing method used in production. DevOps teams can deploy a new feature to a small portion of the fleet in the production environment and expose it to a small group of users. If there is a defect, only a small percentages of users are affected, and it can be fixed or rolled back.

Feature flags provide a way to quickly roll back the feature during the canary testing. And the beauty of the feature flag is that both the development team and the non-technical stakeholders, e.g. product managers, can handle the “control button”. The non-tech stakeholders usually are at the front line when collecting users’ feedback. Giving them the control of turning on and off new features aligns well with the core concept of DevOps.

In addition to the Canary Testing use case, feature flags can also be utilized in many other testing types, e.g. A/B testing, Integration testing, and so on. It improves the testing efficiency and reduces costs. Having said that, when evaluate the Return on Investment (ROI) for adopting the Feature Flag, the DevOps teams also need to be aware of the technical debt. We’ll talk more about it in the later sections of this article.

Operation Support

We have talked about how feature flags support the CI/CD part of DevOps. If we look at the operation part, feature flags can also play an important role. In addition to the testing in production use cases mentioned in the earlier section, feature flags can also improve the incident management process.

For example, usually when the monitoring system detects an abnormal performance that may be related to the new release feature, with integration to an ITSM system, it’ll create an incident. The operation team will respond to the incident by going through incident response process. Even with SLA defined, it’ll still take some time for the incident ticket to land in an operation staff’s hand. By the time they finished their investigation, escalation and roll back, it may have caused certain level of damages.

If we wrap new features under feature flags and hook them to predefined abnormal service metrics, it will significantly reduce the response time. For example, when the monitoring system picks up any abnormal service metrics related to a new feature, the system can automatically turn off the flag. In the meantime, it can still created an incident ticket automatically for further investigation. This way, it can prevent the incident from would-be disasters

Tools

As we mentioned at the beginning of the article, Feature Flag is a technique and Feature Management is a practice for managing Feature Flag’s lifecycle. One of the core elements in the Feature Management is the tool, i.e. Feature Manager, that handles all feature flags within an application. We’ve listed a few popular feature flag management tools and links to their website below.

There a more feature flag management tools in the market. We will not cover all of them here. Before adopting any tools, it’s recommended to candidate the most viable options, conduct pros and cons analysis, and then adopt the most suitable one.

Technical Debt

As mentioned in the Use Cases section earlier, among all the beauties of feature flags, the DevOps teams also need to be aware of the technical debt. It’s a bit like plastic bags. They are easy to make and handy to use. But if we don’t clean or reuse them properly, they will build up and damage the environment.

To effectively manage feature flags, it’s recommended to group them under categories. One common grouping method is to group feature flags into short-lived and long-lived flags based on how long a flag will remain in the codebase. Using the use cases in the earlier sections as examples. If a feature flag only serves a testing purpose, once the testing is completed, the flag should be removed. If a feature flag serves an operation support purpose, the flag becomes part of your product or platform. It’s better to register these flags or tag the flags to ensure these flags are not interrupted during the flag clean-up process.

Summary

In a nut shell, feature flags are like “buttons”. We can manually flip them on or off, or build a decision tree to automatically decide when to flip. These “buttons” can be used across DevOps lifecycle, which complement continuous integration and deployment as well as operational excellence. And remember to “clean up after yourself” to avoid building up the technical debt when using these “buttons”.