Feature Flags in DevOps

In our previous article A Fresher on Testing in CI/CD, we mentioned that feature flags can be used in Canary Testing. We’ll spend some time to talk more about this technique in this article.

What is Feature Flag

Feature Flag is a technique that enables DevOps teams to turn certain features on and off during runtime. It doesn’t require redeploy new code or roll back to previous building versions. Feature Flag is also known as Feature Toggle or Feature Switch.

In addition to the names mentioned above, Feature Management is also part of the gang. To distinguish these two terms, we can consider Feature Management as a practice and Feature Flag as a technique. Or we can also consider Feature Management as a practice to manage Feature Flag’s lifecycle.

Use Cases

In this section we’ll use go through the feature flag use cases along the DevOps lifecycle, as shown in the diagram below.

Trunk-Based Development

In the Dev bucket of the DevOps lifecycle, there are two popular development styles: Git Flow and Trunk-Based Development. Git Flow practices usually come with quite a number of branches, e.g. master branch, release branch, develop branch, feature branch, etc. It requires multiple pull requests and approvals before a piece of code graduates to production. It can be quite a lengthy, complex and even bureaucracy journey. Due to this reason, it often drives to situations like long-lived feature branches, merge conflicts and aging pull requests.

Trunk-Based Development on the other hand encourages short-lived feature branches. It’s like a lean version of a tree that only has the trunk (e.g. the master branch) and some short and direct connected branches. Developers can merge their code directly to the master branch once their code are “ready”. We’ll explain why we put quotation marks around the word “ready”.

The Trunk-Base Development seems to make the life a lot simpler. However, there is a risk comes with it. The assumption is that the code from the short-live branches compiles and passes all tests before they are merged to the master branch. What if they didn’t? So it comes back to the interpretation of the word “ready” we were talking earlier. It could be that you subjectively think it’s ready but it’s objectively not ready. Without extra layers of branches as protection, like Git Flow, how can we protect the “trunk” from the imperfect code?

Feature Flag provides a good answer to this question. Feature flags enable developers to add an extra gate of protection by wrapping their new code under flags. Sometimes people describe feature flags as a non-negotiable requirement for trunk-based development. It nicely compliment trunk-based development and encourage small batch updates.

Example

Below is an example of coding with feature flags.

//C# example
bool featureFlag = featureFlagRule();

if (featureFlag) {
// Run code in this section if featureFlag value is true
} else {
// Run code in this section if the featureFlag value is false
}

This is a simple IF statement for getting a taste of the feature flag. In real life application of feature flags, the complexity of feature flag decision trees may vary. For more information about the Feature Flag SDK and Libraries, see The Hub for Feature Flag Driven Development.

Continuous Deployment

To explain how feature flags streamline the CD processes, let’s take a half step back by looking at the differences between Continuous Delivery and Continuous Deployment. For questions like “What CD within CI/CD stands for”, we often answer with either Continuous Delivery or Continuous Deployment. The common explanation about the difference between these two are like below.

In Continuous Delivery, the deploy to production part is a manual process, whereas Continuous Deployment automates all the way through. The automate-all-the-way-through process seems to be more streamlined. However, it may raise concerns in scenarios like companies who have well-established product or large development teams with a lot of junior developers. The production environment is less tolerant for imperfect code.

With feature flags introduced in development, it enables this final automation part. DevOps teams can continuously deploy solutions to their production environment with no sweat. The production users won’t be able to interact with the flagged code.

In addition, comparing with the big bang deployment from a long-lived feature branches, DevOps teams can gradually roll out new code and bug fixings from their short-lived branches.

Continuous Testing

As mentioned in the A Fresher on Testing in CI/CD article, testing should not be a standalone stage in the DevOps lifecycle. Instead, it should be integrated into each stage, including testing in production. Canary Testing is a popular testing method used in production. DevOps teams can deploy a new feature to a small portion of the fleet in the production environment and expose it to a small group of users. If there is a defect, only a small percentages of users are affected, and it can be fixed or rolled back.

Feature flags provide a way to quickly roll back the feature during the canary testing. And the beauty of the feature flag is that both the development team and the non-technical stakeholders, e.g. product managers, can handle the “control button”. The non-tech stakeholders usually are at the front line when collecting users’ feedback. Giving them the control of turning on and off new features aligns well with the core concept of DevOps.

In addition to the Canary Testing use case, feature flags can also be utilized in many other testing types, e.g. A/B testing, Integration testing, and so on. It improves the testing efficiency and reduces costs. Having said that, when evaluate the Return on Investment (ROI) for adopting the Feature Flag, the DevOps teams also need to be aware of the technical debt. We’ll talk more about it in the later sections of this article.

Operation Support

We have talked about how feature flags support the CI/CD part of DevOps. If we look at the operation part, feature flags can also play an important role. In addition to the testing in production use cases mentioned in the earlier section, feature flags can also improve the incident management process.

For example, usually when the monitoring system detects an abnormal performance that may be related to the new release feature, with integration to an ITSM system, it’ll create an incident. The operation team will respond to the incident by going through incident response process. Even with SLA defined, it’ll still take some time for the incident ticket to land in an operation staff’s hand. By the time they finished their investigation, escalation and roll back, it may have caused certain level of damages.

If we wrap new features under feature flags and hook them to predefined abnormal service metrics, it will significantly reduce the response time. For example, when the monitoring system picks up any abnormal service metrics related to a new feature, the system can automatically turn off the flag. In the meantime, it can still created an incident ticket automatically for further investigation. This way, it can prevent the incident from would-be disasters

Tools

As we mentioned at the beginning of the article, Feature Flag is a technique and Feature Management is a practice for managing Feature Flag’s lifecycle. One of the core elements in the Feature Management is the tool, i.e. Feature Manager, that handles all feature flags within an application. We’ve listed a few popular feature flag management tools and links to their website below.

There a more feature flag management tools in the market. We will not cover all of them here. Before adopting any tools, it’s recommended to candidate the most viable options, conduct pros and cons analysis, and then adopt the most suitable one.

Technical Debt

As mentioned in the Use Cases section earlier, among all the beauties of feature flags, the DevOps teams also need to be aware of the technical debt. It’s a bit like plastic bags. They are easy to make and handy to use. But if we don’t clean or reuse them properly, they will build up and damage the environment.

To effectively manage feature flags, it’s recommended to group them under categories. One common grouping method is to group feature flags into short-lived and long-lived flags based on how long a flag will remain in the codebase. Using the use cases in the earlier sections as examples. If a feature flag only serves a testing purpose, once the testing is completed, the flag should be removed. If a feature flag serves an operation support purpose, the flag becomes part of your product or platform. It’s better to register these flags or tag the flags to ensure these flags are not interrupted during the flag clean-up process.

Summary

In a nut shell, feature flags are like “buttons”. We can manually flip them on or off, or build a decision tree to automatically decide when to flip. These “buttons” can be used across DevOps lifecycle, which complement continuous integration and deployment as well as operational excellence. And remember to “clean up after yourself” to avoid building up the technical debt when using these “buttons”.

You May Also Like

About the Author: Richard Zhao

My name is Richard Zhao. I'm a solution architect and owner of cloudstudio.com.au. Having built knowledge bases for many companies, I'd like to use this cloud studio to share knowledge and ideas with wider people on the internet.

15 Comments

  1. Heya! I realize this is sort of off-topic however I had to ask.
    Does operating a well-established website like yours require a lot of work?
    I am completely new to blogging but I do write in my journal on a daily
    basis. I’d like to start a blog so I can share my personal experience and views online.

    Please let me know if you have any recommendations or tips for brand new
    aspiring bloggers. Thankyou!

    1. hi there, thanks for the question. I shared the same requirement as yours before I started my blog. There are many options available for bloggers like us. I chose to set up WordPress in my own web hosting account. There are a few steps involved, i.e. choose a web hosting provider (like AWS, Azure or other local SPs) to get a virtual server, install a blog software in the server, and then set up the blog from the UI side. The more straight option is to go with the Software as a Service (SaaS) option. This way you can go straight to the step 3 where step 1 and 2 have been taken care by the SaaS provider. One example (but not limited) is wordpress.com where you can create your blog site directly from the web user interface. Hope this information helps.

    1. Thanks Kristofer. I’ve put the deep dive in feature flag topic on my list. If there are specific detailed area that you are interested, please let me know.

  2. Write more, thats all I have to say. Literally, it seems as though you relied on the video to make your point.

    You obviously know what youre talking about, why throw away your intelligence on just posting videos to your
    blog when you could be giving us something enlightening to read?

    1. Thanks you Russell! I enjoy writing indeed. People have asked me why don’t I post videos. I have purchased a decent microphone try to be physically ready, but mentally I just can’t pull myself out from writing:) Maybe one day I become a full-time blogger, I’ll do both then:)

  3. An impressive share! I have just forwarded this onto a colleague who had been doing a little research on this.
    And he in fact ordered me dinner simply because I stumbled upon it
    for him… lol. So allow me to reword this….
    Thank YOU for the meal!! But yeah, thanks for spending the time to talk about this topic here on your internet site.

    1. Lol, you’re welcome Darrel. Very happy to see that the article can be helpful to your colleague. And spreading the word definitely deserves a reward. Hope you guys have enjoyed the dinner 🙂

  4. What’s up to all, how is the whole thing, I think every one is getting more from
    this web page, and your views are good in support
    of new users.

  5. Hey! Would you mind if I share your blog with my zynga group?
    There’s a lot of people that I think would really enjoy your content.
    Please let me know. Thanks

    1. Thanks Hilton! Please feel free to share the site with your friends. If there are any particular topics you are interested, please let me know as well. I’ll add them to my list.

  6. Link exchange is nothing else however it is simply placing the
    other person’s website link on your page at appropriate place and other person will also do similar for you.

    1. Totally agree Georgia. One article can’t cover everything for a particular topic. If there are good articles or official websites that are relevant to the topic, I will include them as reference links. And like you said, other people can do the same if they found my article can provide more referential content for theirs.

  7. I do consider all the concepts you’ve offered for your post.
    They are very convincing and will certainly work. Still, the posts are too brief for beginners.
    May just you please lengthen them a bit from next time?
    Thanks for the post.

    1. Thanks for the feedback Silke. With interests generated from this high level article, we can certainly drill down further with more detailed content. If there are any particular areas that you are interested, please let me know. I’ll add them to my list.

Leave a Reply

Your email address will not be published.