Feature flags in the development of GitLab

This page explains how developers contribute to the development and operations of the GitLab product through feature flags. To create custom feature flags to show and hide features in your own applications, see Create a feature flag. A complete list of feature flags in GitLab is also available.

Design documents:

• (Latest) Feature Flags usage in GitLab development and operations
• Development Feature Flags Architecture

This document is the subject of continued work as part of an epic to improve internal usage of feature flags. Raise any suggestions as new issues and attach them to the epic.

For an overview of the feature flag lifecycle, or if you need help deciding if you should use a feature flag or not, see the feature flag lifecycle handbook page.

When to use feature flags

Moved to the “When to use feature flags” section in the handbook.

Do not use feature flags in external API consumers

Feature flags are internal implementation details and are not part of the public API contract. External API consumers (such as IDE extensions, Duo CLI, and CI integrations) that query feature flags via the GraphQL metadata.featureFlags field or the deprecated featureFlagEnabled field face a specific risk: when a flag is removed from the monolith, external consumers may not have been updated yet, which can range from a minor UI issue to a customer-impacting incident.

Follow this guidance when working with feature flags in external API consumers:

• Prefer API fields or Application Settings. Where possible, avoid querying a feature flag from an external API consumer. Instead, introduce a dedicated API field or Application Setting that the consumer can query. These values persist after the flag is removed.
• Implement fail-open behavior. If a feature flag must be used in an external API consumer, implement a “fail-open” mechanism: after the rollout milestone is finalized, the consumer should default to treating the flag as enabled. Update the consumer as soon as the rollout milestone is confirmed. See an example in the GitLab Language Server.
• Consider user upgrade patterns before removal. Before removing a flag that is used by an external API consumer, assess how quickly users update their clients and determine the safest timing for removal.

Do not use feature flags for long lived settings

Feature flags are meant to be short lived. If you are intending on adding a feature flag so that something can be enabled per user/group/project for a long period of time, consider introducing Cascading Settings or Application Settings instead. Settings offer a way for customers to enable or disable features for themselves on GitLab.com or GitLab Self-Managed and can remain in the codebase as long as needed. In contrast users have no way to enable or disable feature flags for themselves on GitLab.com and only GitLab Self-Managed admins can change the feature flags. Also, feature flags are not supported in GitLab Dedicated which is another reason you should not use them as a replacement for settings.

Feature flags in GitLab development

The following highlights should be considered when deciding if feature flags should be leveraged:

• The feature flag must be disabled by default.
• Feature flags should remain in the codebase for as short a period as possible to reduce the need for feature flag accounting.
• The person operating the feature flag is responsible for clearly communicating the status of a feature behind the feature flag in the documentation and with other stakeholders. The issue description should be updated with the feature flag name and whether it is defaulted on or off as soon as it is evident that a feature flag is needed.
• Merge requests that introduce a feature flag, update its state, or remove the existing feature flag because a feature is deemed stable must have the ~“feature flag” label assigned.

When the feature implementation is delivered over multiple merge requests:

1. Create a new feature flag which is disabled by default, in the first merge request which uses the flag. Flags should not be added separately.
2. Submit incremental changes via one or more merge requests, ensuring that any new code added can only be reached if the feature flag is enabled. You can keep the feature flag enabled on your local GDK during development.
3. When the feature is ready to be tested by other team members, create the initial documentation. Include details about the status of the feature flag.
4. Enable the feature flag for a specific group/project/user and ensure that there are no issues with the implementation. Do not enable the feature flag for a public project like gitlab-org/gitlab if there is no documentation. Team members and contributors might search for documentation on how to use the feature if they see it enabled in a public project.
5. When the feature is ready for production use, including GitLab Self-Managed instances, open one merge request to:
   • Update the documentation to describe the latest flag status.
   • Add a changelog entry.
   • Remove the feature flag to enable the new behavior, or flip the feature flag to be enabled by default (only for ops and beta feature flags).

When the feature flag removal is delivered over multiple merge requests:

1. The value change of a feature flag should be the only change in a merge request. As long as the feature flag exists in the codebase, both states should be fully functional (when the feature is on and off).
2. After all mentions of the feature flag have been removed, legacy code can be removed. Steps in the feature flag roll-out issue should be followed, and if a step needs to be skipped, a comment should be added to the issue detailing why.

One might be tempted to think that feature flags will delay the release of a feature by at least one month (= one release). This is not the case. A feature flag does not have to stick around for a specific amount of time (for example, at least one release). Instead, it should stick around until the feature is deemed stable. Stable means it works on GitLab.com without causing any problems, such as outages.

Risk of a broken default branch

Feature flags must be used in the MR that introduces them. Not doing so causes a broken default branch scenario due to the rspec:feature-flags job that only runs on the default branch.

Types of feature flags

Choose a feature flag type that matches the expected usage.

gitlab_com_derisk type

gitlab_com_derisk feature flags are short-lived feature flags, used to de-risk GitLab.com deployments. Most feature flags used at GitLab are of the gitlab_com_derisk type.

Constraints

• default_enabled: Must not be set to true. This kind of feature flag is meant to lower the risk on GitLab.com, thus there’s no need to keep the flag in the codebase after it’s been enabled on GitLab.com. default_enabled: true will not have any effect for this type of feature flag.
• Maximum Lifespan: 2 months after it’s merged into the default branch
• Documentation: This type of feature flag doesn’t need to be documented in the All feature flags in GitLab page given they’re short-lived and deployment-related.
• Rollout issue: Must have a rollout issue created from the Feature flag Roll Out template

Usage

The format for gitlab_com_derisk feature flags is Feature.<state>(:<dev_flag_name>).

To enable and disable them, run on the GitLab Rails console:

# To enable it for the instance:
Feature.enable(:<dev_flag_name>)

# To disable it for the instance:
Feature.disable(:<dev_flag_name>)

# To enable for a specific project:
Feature.enable(:<dev_flag_name>, Project.find(<project id>))

# To disable for a specific project:
Feature.disable(:<dev_flag_name>, Project.find(<project id>))

To check a gitlab_com_derisk feature flag’s state:

# Check if the feature flag is enabled
Feature.enabled?(:dev_flag_name)

# Check if the feature flag is disabled
Feature.disabled?(:dev_flag_name)

wip type

Some features are complex and need to be implemented through several MRs. Until they’re fully implemented, they need to be hidden from anyone. In that case, the wip (for “Work In Progress”) feature flag allows merging all the changes to the main branch without actually using the feature yet.

Once the feature is complete, the feature flag type can be changed to the gitlab_com_derisk or beta type depending on how the feature will be presented/documented to customers.

Constraints

• default_enabled: Must not be set to true. If needed, this type can be changed to beta once the feature is complete.
• Maximum Lifespan: 4 months after it’s merged into the default branch
• Documentation: This type of feature flag doesn’t need to be documented in the All feature flags in GitLab page given they’re mostly hiding unfinished code.
• Rollout issue: Likely no need for a rollout issues, as wip feature flags should be transitioned to another type before being enabled

Usage

# Check if feature flag is enabled
Feature.enabled?(:my_wip_flag, project)

# Check if feature flag is disabled
Feature.disabled?(:my_wip_flag, project)

# Push feature flag to Frontend
push_frontend_feature_flag(:my_wip_flag, project)

beta type

We might not be confident we’ll be able to scale, support, and maintain a feature in its current form for every designed use case (example). There are also scenarios where a feature is not complete enough to be considered an MVC. Providing a flag in this case allows engineers and customers to disable the new feature until it’s performant enough.

Constraints

• default_enabled: Can be set to true so that a feature can be “released” to everyone in beta with the possibility to disable it in the case of scalability issues (ideally it should only be disabled for this reason on specific on-premise installations)
• Maximum Lifespan: 6 months after it’s merged into the default branch
• Documentation: This type of feature flag must be documented in the All feature flags in GitLab page. That page is auto-generated during the docs build from the YAML definition files, so no manual edits to that page are needed.
• Rollout issue: Must have a rollout issue created from the Feature flag Roll Out template

Usage

# Check if feature flag is enabled
Feature.enabled?(:my_beta_flag, project)

# Check if feature flag is disabled
Feature.disabled?(:my_beta_flag, project)

# Push feature flag to Frontend
push_frontend_feature_flag(:my_beta_flag, project)

ops type

ops feature flags are long-lived feature flags that control operational aspects of GitLab product behavior. For example, feature flags that disable features that might have a performance impact such as Sidekiq worker behavior.

Remember that using this type should follow a conscious decision not to introduce an instance/group/project/user setting.

While ops type flags have an unlimited lifespan, every 12 months, they must be evaluated to determine if they are still necessary. If so, the milestone field must be updated to the latest milestone to confirm that the ops feature flag is still in use.

Constraints

• default_enabled: Should be set to false in most cases, and only enabled to resolve temporary scalability issues or help debug production issues.
• Maximum Lifespan: Unlimited, but must be evaluated every 12 months
• Documentation: This type of feature flag must be documented in the All feature flags in GitLab page as well as be associated with an operational runbook describing the circumstances when it can be used. That page is auto-generated during the docs build from the YAML definition files, so no manual edits to that page are needed
• Rollout issue: Likely no need for a rollout issues, as it is hard to predict when they are enabled or disabled

Usage

# Check if feature flag is enabled
Feature.enabled?(:my_ops_flag, project)

# Check if feature flag is disabled
Feature.disabled?(:my_ops_flag, project)

# Push feature flag to Frontend
push_frontend_feature_flag(:my_ops_flag, project)

experiment type

experiment feature flags are used for A/B testing on GitLab.com.

An experiment feature flag should conform to the same standards as a beta feature flag, although the interface has some differences. An experiment feature flag should have a rollout issue, created using the Experiment tracking template. More information can be found in the experiment guide.

Constraints

• default_enabled: Must not be set to true.
• Maximum Lifespan: 6 months after it’s merged into the default branch

worker type

worker feature flags are special ops flags that you can use to control Sidekiq worker behavior, such as deferring Sidekiq jobs.

worker feature flags likely do not have any YAML definition as the name could be dynamically generated using the worker name itself, for example, run_sidekiq_jobs_AuthorizedProjectsWorker. Some examples for using worker type feature flags can be found in deferring Sidekiq jobs.

markdown_cache type

markdown_cache feature flags drive the phased rollout of a Gitlab::MarkdownCache::CACHE_COMMONMARK_VERSION bump. Like worker flags, they have no YAML definition: the name is generated from the target cache version, for example, markdown_cache_stochastic_rollout_34. Because every bump uses its own flag, a fresh flag starts disabled, so a percentage left set on a previous bump’s flag cannot carry into the next. For the rollout procedure, see phased rollout of CACHE_COMMONMARK_VERSION bumps.

(Deprecated) development type

The development type is deprecated in favor of the gitlab_com_derisk, wip, and beta feature flag types.

Feature flag definition and validation

During development (RAILS_ENV=development) or testing (RAILS_ENV=test) all feature flag usage is being strictly validated.

This process is meant to ensure consistent feature flag usage in the codebase. All feature flags must:

• Be known. Only use feature flags that are explicitly defined (except for feature flags of the types experiment, worker, and undefined).
• Not be defined twice. They have to be defined either in FOSS or EE, but not both.
• For feature flags that don’t have a definition file, use a valid and consistent type: across all invocations.
• Have an owner.

All feature flags known to GitLab are self-documented in YAML files stored in:

• config/feature_flags
• ee/config/feature_flags

Each feature flag is defined in a separate YAML file consisting of a number of fields:

Create a new feature flag

The GitLab codebase provides bin/feature-flag, a dedicated tool to create new feature flag definitions. The tool asks various questions about the new feature flag, then creates a YAML definition in config/feature_flags or ee/config/feature_flags.

Only feature flags that have a YAML definition file can be used when running the development or testing environments.