Documentation Grafana Cloud Instrument and send data Fleet Management Set up Infrastructure as code GitOps

GitOps and Fleet Management

If you store your collector configurations in a version control system like Git, you can use the Grafana Terraform provider or the Grafana Fleet Management Pipeline API to integrate your repositories with the remote configuration feature in Fleet Management.

Terraform and GitOps

You can combine a version control system like Git, an infrastructure as code tool like Terraform, and a CI/CD platform like GitHub Actions to create a GitOps workflow for Fleet Management. The Grafana Terraform provider supports the provisioning of Fleet Management resources, including configuration pipelines.

Fleet Management Pipeline API

Create, edit, and delete configuration pipelines in Fleet Management with requests to the Pipeline API from the CI/CD workflow of your GitOps repository. Use the SyncPipelinesRequest endpoint to edit multiple pipelines from a common source in a single atomic operation. Pipelines are associated with their source and namespace, and a sync request affects pipelines from the specified source only. This isolation ensures the correct group of pipelines is updated with each request.

Register your collectors

With Fleet Management, you can remotely configure your collector fleet based on matching attributes. To enable remote configuration, make sure to onboard your collectors to the application.

Make changes in a development branch

Update, remove, or add new configuration pipelines in your repository using a development branch. Make sure to confirm that the configuration works in a test environment before merging to production. You can also use the Fleet Management interface to validate Alloy configuration syntax.

Create a CI/CD job that executes on merge

Use a GitHub Actions workflow, a purpose-built script, or another tool to create a CI/CD job that executes when a development branch is merged to production. This job should make POST requests to the Fleet Management Pipeline API. For bulk edits, make requests to the SyncPipelinesRequest endpoint. Refer to the API examples to learn more about creating request payloads.

Verify the changes

After the merge is complete, check the Fleet Management interface to confirm that the changes have been integrated.

• Check the Remote configuration tab to see a newly created pipeline or confirm changes have been made to an existing pipeline by clicking on the pipeline history icon.
• If you updated matching attributes for a configuration pipeline, you can check if the pipeline was applied to the correct collectors.
  1. Search or filter in the Inventory tab for the attribute.
  2. Click on a collector in the filtered list to open the details view.
  3. Switch to the Configuration tab to see the remote configuration matched to that collector.
  4. Confirm that your changes have been applied. Make sure to give the collector enough time to poll for a new configuration before confirming.

Special considerations

Keep these considerations in mind as you implement a GitOps integration with Fleet Management.

Single source of truth

Using the push workflow described on this page introduces the possibility of drift between your repository configurations and the configuration pipelines stored in Fleet Management. To prevent drift, avoid editing configuration pipelines in the Fleet Management interface. Maintain your version-controlled configurations as the single source of truth.

In addition, watch for transient errors in your CI/CD workflow that might fail to update the remote configurations. Rerun workflows manually to ensure the changes take effect.

Unintended changes to remote configuration

When using SyncPipelinesRequest in a GitOps workflow, be aware of the following behaviors that could result in unintended changes to your Fleet Management configuration pipelines.

• Any pipelines previously managed from the specified source that are not listed in a SyncPipelinesRequest are deleted from the Fleet Management database.
• A pipeline in the Fleet Management database that shares the same name as one in a SyncPipelinesRequest is overwritten, regardless of its source.

To avoid accidentally deleting or overwriting remote configuration pipelines, make sure that all pipelines have a unique name and each sync request accurately reflects the complete and intended set of pipelines for each source and namespace. Always generate your SyncPipelinesRequest from the current, authoritative source of truth and keep your sync automation consistent.

Pipeline limit

The Fleet Management service limits the number of configuration pipelines in a stack to 200. If the number of pipelines in your stack approaches the limit, try deleting your old pipelines before updating or adding pipelines.

Delete old pipelines

The SyncPipelinesRequest endpoint automatically deletes old pipelines from Fleet Management that no longer exist in the specified source. If your workflow doesn’t include calls to SyncPipelinesRequest, follow these steps to delete old pipelines before making changes.

1. Query the ListPipelines endpoint with a ListPipelinesRequest to get a list of pipelines known to Fleet Management.
2. Compare this list to your list of repository pipelines.
3. Make a call to DeletePipeline with a DeletePipelineRequest to remove any pipelines from Fleet Management that don’t exist in the repository.
4. Once the lists match, make another call to UpdatePipeline with a UpdatePipelineRequest or UpsertPipeline with a UpsertPipelineRequest to push your changes to Fleet Management.

If your GitOps-managed configurations exceed the 200 limit, you must be selective about what you update and push to Fleet Management.