Documentationbreadcrumb arrow Grafana Cloudbreadcrumb arrow Instrument and send databreadcrumb arrow Fleet Managementbreadcrumb arrow Manage your fleetbreadcrumb arrow Pipelinesbreadcrumb arrow Sync pipelines from Git
Grafana Cloud

Sync configuration pipelines from a Git repository

You can use the SyncPipelines endpoint of the Pipeline API to synchronize configuration pipelines from external sources like Git repositories. This allows you to manage your pipeline configurations as code and automate the deployment of pipelines to Fleet Management. To learn more about setting up git sync, refer to GitOps and Fleet Management and the fleet-management-sync-action repository.

How pipeline syncing works

When you call the SyncPipelines endpoint, Fleet Management does the following:

  1. Upserts pipelines: Creates new pipelines or updates existing pipelines that match by name. Pipelines are matched by name only, so it is possible to overwrite a pipeline from a different source if you reuse the same name.

  2. Deletes stale pipelines: Removes pipelines that previously came from the same source but are no longer present in the sync request.

    Note

    Source scoping, which checks both source_type and namespace, is used when cleaning up stale pipelines. During this process, Fleet Management only deletes stale pipelines that match both the source_type and namespace provided in the request. This check ensures that syncing from one source does not accidentally delete pipelines from other external sources or those created manually in the Fleet Management application.

Source types and namespaces

Each pipeline can be associated with a source that tracks where it was last updated from. The source has two components:

  • source_type: The type of source system (for example, SOURCE_TYPE_GIT or SOURCE_TYPE_TERRAFORM)
  • namespace: A string identifier for the specific source within that type (for example, the repository name, a unique identifier for your automation, or the workflow name such as instrumentation-hub-init or self-monitoring.)

When you sync pipelines, Fleet Management only deletes pipelines that meet all of the following criteria:

  • Match the same source_type
  • Match the same namespace
  • No pipeline with a matching name is found in the request

Pipeline config types

Each pipeline also has a config_type that tells Fleet Management how to interpret its contents: CONFIG_TYPE_ALLOY for Grafana Alloy configuration or CONFIG_TYPE_OTEL for OpenTelemetry Collector configuration. If you omit config_type in a sync request, Fleet Management assigns CONFIG_TYPE_ALLOY by default.

A single SyncPipelines request can mix pipelines of both config types, so you can manage Alloy and OpenTelemetry Collector fleets from the same Git repository and sync workflow. If you use the fleet-management-sync-action GitHub Action to sync pipelines, set config_type: otel in a pipeline’s YAML metadata to generate an OpenTelemetry Collector pipeline, or omit it (or set it to alloy) for a Grafana Alloy pipeline.

Example: Sync pipelines from a Git repository

The following example shows how the sync operation works for a CI/CD pipeline that syncs configuration from a Git repository to Fleet Management.

Initial state

Your Fleet Management instance has these pipelines:

  • logs_pipeline: source_type: SOURCE_TYPE_GIT, namespace: observability-configs
  • metrics_pipeline: source_type: SOURCE_TYPE_GIT, namespace: observability-configs
  • debug_pipeline: source_type: SOURCE_TYPE_GIT, namespace: observability-configs
  • custom_pipeline: source_type: none, manually created

Sync request

Your Git repository now contains three pipeline configuration files, and your CI/CD system sends a sync request. logs_pipeline and debug_pipeline are Grafana Alloy configurations, and the new traces_pipeline is an OpenTelemetry Collector configuration, so it sets config_type to CONFIG_TYPE_OTEL:

POST pipeline.v1.PipelineService/SyncPipelines

JSON 
{
  "source": {
    "type": "SOURCE_TYPE_GIT",
    "namespace": "observability-configs"
  },
  "pipelines": [
    {
      "name": "logs_pipeline",
      "contents": "// Updated Alloy configuration for logs...",
      "matchers": ["environment=production"]
    },
    {
      "name": "traces_pipeline",
      "contents": "receivers:\n  otlp:\n    protocols: { grpc: {} }\nexporters:\n  otlp:\n    endpoint: <OTLP_ENDPOINT>\nservice:\n  pipelines:\n    traces: { receivers: [otlp], exporters: [otlp] }",
      "matchers": ["environment=production"],
      "config_type": "CONFIG_TYPE_OTEL"
    },
    {
      "name": "debug_pipeline",
      "contents": "// Same Alloy configuration for debug...",
      "matchers": ["environment=staging"]
    }
  ]
}

Result

After the sync operation completes, your remote configurations in Fleet Management look like this:

  1. logs_pipeline: Updated with the new Alloy configuration from Git.
  2. traces_pipeline: Created as a new OpenTelemetry Collector pipeline.
  3. debug_pipeline: Unmodified because the contents are the same.
  4. metrics_pipeline: Deleted because it was previously from this source but is not in the sync request.
  5. custom_pipeline: Preserved because it does not have a source matching the sync request.

Example: Sync pipelines from multiple Git repositories

If you manage pipelines from multiple Git repositories, use different namespace values to isolate them.

Sync from production repository:

JSON 
{
  "source": {
    "type": "SOURCE_TYPE_GIT",
    "namespace": "prod-configs"
  },
  "pipelines": [...]
}

Sync from development repository:

JSON 
{
  "source": {
    "type": "SOURCE_TYPE_GIT",
    "namespace": "dev-configs"
  },
  "pipelines": [...]
}

The cleanup phase is isolated by namespace, so each sync operation only deletes pipelines from its specific namespace.

However, pipelines are matched by name during the upsert phase. If you have multiple repositories and want to avoid pipeline name collisions, consider using a prefix for pipeline names within each repository. For example, prod_logs_pipeline and dev_logs_pipeline.

Example: Implement with curl

The following example uses curl to sync pipelines from Git, including one OpenTelemetry Collector pipeline read from a YAML file:

Bash 
#!/bin/bash

FM_INSTANCE_ID="<your-instance-id>"
FM_API_TOKEN="<your-api-token>"
FM_URL="<your-fleet-management-url>"
GIT_REPO_NAME="observability-configs"

# Read pipeline files from your Git repository
LOGS_CONTENT=$(cat pipelines/logs.alloy)
METRICS_CONTENT=$(cat pipelines/metrics.alloy)
TRACES_CONTENT=$(cat pipelines/traces-otel.yaml)

# Create the sync request
curl -X POST "$FM_URL/pipeline.v1.PipelineService/SyncPipelines" \
  -u "$FM_INSTANCE_ID:$FM_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d @- <<EOF
{
  "source": {
    "type": "SOURCE_TYPE_GIT",
    "namespace": "${GIT_REPO_NAME}"
  },
  "pipelines": [
    {
      "name": "logs_pipeline",
      "contents": $(jq -Rs . <<< "${LOGS_CONTENT}"),
      "matchers": ["environment=production"],
      "enabled": true
    },
    {
      "name": "metrics_pipeline",
      "contents": $(jq -Rs . <<< "${METRICS_CONTENT}"),
      "matchers": ["environment=production"],
      "enabled": true
    },
    {
      "name": "traces_pipeline",
      "contents": $(jq -Rs . <<< "${TRACES_CONTENT}"),
      "matchers": ["environment=production"],
      "enabled": true,
      "config_type": "CONFIG_TYPE_OTEL"
    }
  ]
}
EOF

traces_pipeline sets config_type to CONFIG_TYPE_OTEL because pipelines/traces-otel.yaml contains an OpenTelemetry Collector configuration rather than Alloy configuration.

Refer to the set up documentation for Alloy or OpenTelemetry Collector for how to find your FM_INSTANCE_ID, FM_API_TOKEN, FM_URL, and OTLP_ENDPOINT.