Documentation Grafana Cloud Instrument and send data Fleet Management Manage your fleet Pipelines Sync pipelines from Git

Sync configuration pipelines from a Git repository

You can use the SyncPipelines endpoint of the Pipeline API to synchronize configuration pipelines from external sources like Git repositories. This allows you to manage your pipeline configurations as code and automate the deployment of pipelines to Fleet Management.

How pipeline syncing works

When you call the SyncPipelines endpoint, Fleet Management does the following:

1. Upserts pipelines: Creates new pipelines or updates existing pipelines that match by name. Pipelines are matched by name only, so it is possible to overwrite a pipeline from a different source if you reuse the same name.

2. Deletes stale pipelines: Removes pipelines that previously came from the same source but are no longer present in the sync request.

   Note

   Source scoping, which checks both source_type and namespace, is used when cleaning up stale pipelines. During this process, Fleet Management only deletes stale pipelines that match both the source_type and namespace provided in the request. This check ensures that syncing from one source does not accidentally delete pipelines from other external sources or those created manually in the Fleet Management application.

Understanding source types and namespaces

Each pipeline can be associated with a source that tracks where it was last updated from. The source has two components:

• source_type: The type of source system (for example, SOURCE_TYPE_GIT or SOURCE_TYPE_TERRAFORM)
• namespace: A string identifier for the specific source within that type (for example, the repository name or a unique identifier for your automation)

When you sync pipelines, Fleet Management only deletes pipelines that meet all of the following criteria:

• Match the same source_type
• Match the same namespace
• No pipeline with a matching name is found in the request

Example: Sync pipelines from a Git repository

The following example shows how the sync operation works for a CI/CD pipeline that syncs configuration from a Git repository to Fleet Management.

Initial state

Your Fleet Management instance has these pipelines:

• logs_pipeline: source_type: SOURCE_TYPE_GIT, namespace: observability-configs
• metrics_pipeline: source_type: SOURCE_TYPE_GIT, namespace: observability-configs
• debug_pipeline: source_type: SOURCE_TYPE_GIT, namespace: observability-configs
• custom_pipeline: source_type: none, manually created

Sync request

Your Git repository now contains three pipeline configuration files, and your CI/CD system sends a sync request:

POST pipeline.v1.PipelineService/SyncPipelines

JSON Copy

{
  "source": {
    "type": "SOURCE_TYPE_GIT",
    "namespace": "observability-configs"
  },
  "pipelines": [
    {
      "name": "logs_pipeline",
      "contents": "// Updated Alloy configuration for logs...",
      "matchers": ["environment=production"]
    },
    {
      "name": "traces_pipeline",
      "contents": "// New Alloy configuration for traces...",
      "matchers": ["environment=production"]
    },
    {
      "name": "debug_pipeline",
      "contents": "// Same Alloy configuration for debug...",
      "matchers": ["environment=staging"]
    }
  ]
}

Result

After the sync operation completes, your remote configurations in Fleet Management look like this:

1. logs_pipeline: Updated with the new configuration from Git.
2. traces_pipeline: Created as a new pipeline.
3. debug_pipeline: Unmodified because the contents are the same.
4. metrics_pipeline: Deleted because it was previously from this source but is not in the sync request.
5. custom_pipeline: Preserved because it does not have a source matching the sync request.

Example: Sync pipelines from multiple Git repositories

If you manage pipelines from multiple Git repositories, use different namespace values to isolate them.

Sync from production repository:

JSON Copy

{
  "source": {
    "type": "SOURCE_TYPE_GIT",
    "namespace": "prod-configs"
  },
  "pipelines": [...]
}

Sync from development repository:

JSON Copy

{
  "source": {
    "type": "SOURCE_TYPE_GIT",
    "namespace": "dev-configs"
  },
  "pipelines": [...]
}

The cleanup phase is isolated by namespace, so each sync operation only deletes pipelines from its specific namespace.

However, pipelines are matched by name during the upsert phase. If you have multiple repositories and want to avoid pipeline name collisions, consider using a prefix for pipeline names within each repository. For example, prod_logs_pipeline and dev_logs_pipeline.

Example: Implement with curl

The following example uses curl to sync pipelines from Git:

Bash Copy

#!/bin/bash

FM_INSTANCE_ID="<your-instance-id>"
FM_API_TOKEN="<your-api-token>"
FM_URL="<your-fleet-management-url>"
GIT_REPO_NAME="observability-configs"

# Read pipeline files from your Git repository
LOGS_CONTENT=$(cat pipelines/logs.alloy)
METRICS_CONTENT=$(cat pipelines/metrics.alloy)

# Create the sync request
curl -X POST "$FM_URL/pipeline.v1.PipelineService/SyncPipelines" \
  -u "$FM_INSTANCE_ID:$FM_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d @- <<EOF
{
  "source": {
    "type": "SOURCE_TYPE_GIT",
    "namespace": "${GIT_REPO_NAME}"
  },
  "pipelines": [
    {
      "name": "logs_pipeline",
      "contents": $(jq -Rs . <<< "${LOGS_CONTENT}"),
      "matchers": ["environment=production"],
      "enabled": true
    },
    {
      "name": "metrics_pipeline",
      "contents": $(jq -Rs . <<< "${METRICS_CONTENT}"),
      "matchers": ["environment=production"],
      "enabled": true
    }
  ]
}
EOF

Refer to the set up documentation for how to find your FM_INSTANCE_ID, FM_API_TOKEN, and FM_URL.