Documentation Grafana Cloud Instrument and send data Fleet Management Manage your fleet Guides Minimize risk of configuration changes

Minimize risk by deploying configuration pipelines in stages

In Grafana Fleet Management, configuration pipelines are assigned to collectors using matching attributes. Ensure stability in your production environments with staged releases that leverage these attributes.

In the Fleet Management interface, begin by assigning attributes to your collectors based on their deployment characteristics, such as env=PROD, test=GROUP-A, or deploy=BLUE. You can also automate the assignment of remote attributes with calls to the Collector API.

Once your collectors are categorized, create a configuration pipeline and assign matching attributes, either using the Fleet Management application or the Pipeline API. Matching attributes are combined with an AND operator, so you can customize their application to your setup.

Example: Release a new configuration pipeline

For example, you can use a gradual rollout to test a new configuration pipeline:

1. Add an env remote attribute with values dev, staging, prod-eu, or prod-us to each collector that should receive the pipeline.
2. While creating the new pipeline, add a matching attribute using a regular expression that matches your dev collectors: env=~dev.
3. When you’re satisfied with the pipeline’s performance, modify the matching attribute to include the staging environment: env=~dev|staging.
4. If the pipeline is ready for production, add the prod-eu to the matching attribute: env=~dev|staging|prod-eu.
5. Confirm there are no issues and then add the prod-us to the matching attribute so the pipeline is now deployed across all your environments and production instances: env=~dev|staging|prod-eu|prod-us.

If the pipeline causes a problem at any point during the rollout, deactivate it with a click of the switch in the Remote configuration tab in the Fleet Management application.

Example: Create a new version of a configuration pipeline

With the Grafana Terraform provider or the Pipeline API, it’s possible to integrate version controlled configuration pipelines with Fleet Management. If GitOps is not part of your current observability setup, you can still maintain a version history when testing and rolling out new versions of existing configuration pipelines.

1. Create a copy of the current pipeline and name the copy with its version number (for example, integration_linux_node_metrics_v1_3).
2. Keep the original pipeline running everywhere while you deploy the new version to dev or staging environments using matching attributes.
3. If you’re satisfied with the new version, add the matching attributes to deploy it to the rest of your environments.
4. Deactivate the original pipeline by clicking the switch in the Fleet Management application or setting enabled to false in your API request.

If at any point the new version of the pipeline causes problems, you can deactivate it and reactivate the original pipeline.

Other configuration deployment patterns

Matching attributes can be used for other types of configuration pipeline deployments:

• A/B testing. To conduct A/B testing of different configuration pipelines, assign remote attributes to collectors based on which version of the pipeline they should receive (for example, test=GROUP-A and test=GROUP-B) and then add the matching attribute to the corresponding pipeline. Label the collected telemetry by group as well and then evaluate the performance of each pipeline.
• Canary deployments. Implement a canary deployment by assigning a meaningful attribute to each group of collectors and then matching the new version of the pipeline to each group in succession. Once you’re satisfied with the pipeline’s performance, add a regular expression and the matching attribute for the next group of collectors, and so on. Refer to the staged rollout example for a sample regular expression.
• Blue-green deployments. You can also assign attributes based on blue-green environments, for example deploy=BLUE and deploy=GREEN, and then add matching attributes to control which configuration pipelines are assigned to each environment.