Instrument and send data

Fleet Management

Reference

Pipeline HTTP API

Grafana Cloud

The Pipeline HTTP API

The Grafana Fleet Management Pipeline API allows you to perform CRUD operations on configuration pipelines.

For a full definition of the Pipeline API, refer to the .proto file.

Find the base URL

You can find the base URL for the Pipeline API in the Grafana Cloud Fleet Management interface.

In your Grafana Cloud stack, click Connections > Collector > Fleet Management in the left-side menu.
On the Fleet Management interface, switch to the API tab.
Find the URL in the Base URL section. It looks like the following URL, where <CLUSTER_NAME> is the production cluster of your stack:

https://fleet-management-<CLUSTER_NAME>.grafana.net/pipeline.v1.PipelineService/

Note
If you need to secure your API traffic, you can configure an AWS PrivateLink endpoint and use the private DNS name in place of the base URL. You can find the service name for endpoint configuration and the private DNS name on the same API tab where the base URL is found.

Rate limit

Requests to the Pipeline API are limited to 3 requests per second.

PipelineService

The PipelineService defines the RPCs for managing pipelines.

Method Name	Request Type	Response Type	Description
GetPipeline	GetPipelineRequest	Pipeline	Returns a pipeline by ID
GetPipelineID	GetPipelineIDRequest	GetPipelineIDResponse	Returns a pipeline ID by name
ListPipelines	ListPipelinesRequest	Pipelines	Returns all pipelines
CreatePipeline	CreatePipelineRequest	Pipeline	Creates a new pipeline and returns it
UpdatePipeline	UpdatePipelineRequest	Pipeline	Updates an existing pipeline and returns it
UpsertPipeline	UpsertPipelineRequest	Pipeline	Creates a new pipeline or updates an existing one and returns it
DeletePipeline	DeletePipelineRequest	DeletePipelineResponse	Deletes a pipeline by ID
SyncPipelines	SyncPipelinesRequest	SyncPipelinesResponse	Creates, updates, and deletes multiple pipelines from a common source
ListPipelinesRevisions	ListPipelinesRevisionsRequest	PipelineRevisions	Returns all pipeline revisions
ListPipelineRevisions	ListPipelineRevisionsRequest	PipelineRevisions	Returns all pipeline revisions for a single pipeline by ID
GetPipelineRevision	GetPipelineRevisionRequest	PipelineRevision	Returns a pipeline revision by revision ID

Endpoints

Note
Protobuf field names are defined in snake_case. Due to default protojson behavior, field names are converted to camelCase in responses. Both snake_case and camelCase field names are accepted in requests.

CreatePipelineRequest

CreatePipelineRequest is the request to create a new pipeline.

Field	Type	Label	Description
pipeline	Pipeline	required	The pipeline to create
validate_only	bool	optional	If set, validates the request and previews the response, but doesn’t create the actual resource

DeletePipelineRequest

DeletePipelineRequest is the request to delete a pipeline by its ID.

Field	Type	Label	Description
id	string	required	ID of the pipeline to delete

DeletePipelineResponse

DeletePipelineResponse is the response to deleting a pipeline. This message is empty and the results of the deletion are defined by the HTTP status code of the response.

GetPipelineRequest

GetPipelineRequest is the request to retrieve a pipeline by its ID.

Field	Type	Label	Description
id	string	required	ID of the pipeline to get

GetPipelineIDRequest

GetPipelineIDRequest is the request to retrieve a pipeline ID by its name.

Field	Type	Label	Description
name	string	required	Name of the pipeline to get the ID for

GetPipelineIDResponse

GetPipelineIDResponse is the response to retrieving a pipeline ID.

Field	Type	Label	Description
id	string		ID of the pipeline

ListPipelinesRequest

ListPipelinesRequest is the request to get the full list of pipelines, including their contents and matchers.

Pipeline

A Pipeline is a self-contained piece of configuration that can be assigned to collectors based on matchers.

Field	Type	Label	Description
name	string	required	Name of the pipeline which is the unique identifier for the pipeline
contents	string	required	Configuration contents of the pipeline to be used by collectors
matchers	list(string)		Used to match against collectors and assign pipelines to them; follows the syntax of Prometheus Alertmanager matchers; 200 character limit per matcher
created_at	google.protobuf.Timestamp	optional	Timestamp when the pipeline was created
updated_at	google.protobuf.Timestamp	optional	Timestamp when the pipeline was last updated
enabled	bool	optional	Whether the pipeline is enabled for collectors
id	string	optional	Server-assigned ID of the pipeline
source	PipelineSource	optional	Source of the pipeline

Pipelines

Pipelines represents a list of pipelines.

Field	Type	Label	Description
pipelines	list(Pipeline)		List of pipelines

PipelineSource

PipelineSource indicates the external source of a pipeline. Although the two fields type and namespace are optional, if type is specified, then namespace must also be specified. If type is set to UNSPECIFIED, namespace must be an empty string. Requests that don’t follow these rules fail with a 400 BAD REQUEST validation error.

Field	Type	Label	Description
type	PipelineSource_SourceType	optional	Source of the pipeline
namespace	string	optional	Namespace of the pipeline

PipelineSource_SourceType

Value	Description
SOURCE_TYPE_UNSPECIFIED	Default value applied when `source` is not specified; do not set this value manually
SOURCE_TYPE_GIT	The pipeline originates from a Git workflow
SOURCE_TYPE_TERRAFORM	The pipeline originates from a Terraform workflow

UpdatePipelineRequest

UpdatePipelineRequest is the request to update an existing pipeline. This contents supplied in this request replace the existing pipeline contents, so any fields that are not set are removed. If the pipeline does not already exist, this request returns a 404 ‘NOT FOUND’ error.

Field	Type	Label	Description
pipeline	Pipeline	required	Contents of the pipeline to update
validate_only	bool	optional	If set, validates the request and previews the response, but doesn’t update the actual resource

UpsertPipelineRequest

UpsertPipelineRequest is the request to create a new pipeline or update an existing one. If the pipeline already exists, it is updated and like UpdatePipelineRequest, any fields that are not set are removed.

Field	Type	Label	Description
pipeline	Pipeline	required	Pipeline to create or update

ListPipelinesRevisionsRequest

ListPipelinesRevisionsRequest is the request to get the full list of pipelines revisions, excluding contents.

ListPipelineRevisionsRequest

ListPipelineRevisionsRequest is the request to retrieve pipeline revisions by pipeline ID.

Field	Type	Label	Description
id	string	required	ID of the pipeline to get

GetPipelineRevisionRequest

GetPipelineRevisionRequest is the request to retrieve a pipeline revision by revision ID.

Field	Type	Label	Description
revision_id	string	required	ID of the pipeline revision to get

PipelineRevision

A PipelineRevision represents a point in time for a Pipeline.

Field	Type	Label	Description
revision_id	string	required	Server-assigned ID of the pipeline revision
snapshot	Pipeline	required	The pipeline at the point in time of the revision
created_at	google.protobuf.Timestamp	required	Timestamp when the pipeline revision was created
operation	PipelineRevision.Operation	required	The operation that was performed on the pipeline which created the revision

PipelineRevision.Operation

Value	Description
UNSPECIFIED	Unknown revision operation
INSERT	The pipeline was created at the time of this revision
UPDATE	The pipeline was updated at the time of this revision
DELETE	The pipeline was deleted at the time of this revision

PipelineRevisions

PipelineRevisions represents a list of pipeline revisions.

Field	Type	Label	Description
pipeline_revisions	list(PipelineRevision)		List of pipeline revisions

SyncPipelinesRequest

SyncPipelinesRequest is the request to bulk update pipelines from a common source. If a source pipeline does not exist in the Fleet Management database, it is created. If a source pipeline already exists in the database, it is updated with any changes, the same as if it were updated with an UpsertPipelineRequest. If a pipeline exists in the database but not in the source, it is deleted.

Field	Type	Label	Description
source	PipelineSource	required	Source of the pipeline(s)
pipelines	list(Pipeline)	required	List of pipelines

SyncPipelinesResponse

SyncPipelinesResponse is the response to bulk updating pipelines. This message is empty and the results of the sync are defined by the HTTP status code of the response. An HTTP status code of 400 indicates a validation error in the source field of the request or in one or more of the pipelines.

Examples

Pipeline API requests and responses require proper escaping of the pipeline contents. For this reason, some use of jq is helpful. In this example, the contents of the pipeline are saved in a file named config.alloy.

--- config.alloy ---
prometheus.exporter.self "alloy" { }

prometheus.scrape "alloy" {
  targets    = prometheus.exporter.self.alloy.targets
  forward_to = [prometheus.remote_write.grafanacloud.receiver]

  scrape_interval = "60s"
}

prometheus.remote_write "grafanacloud" {
  // Must match the uniquely-identifiable ID set up in the remotecfg block.
  external_labels = {"collector_id" = constants.hostname}

  endpoint {
    url = <SERVICE_URL>

    basic_auth {
      username      = <USERNAME>
      password_file = <PASSWORD_FILE>
    }
  }
}

To build a payload for the CreatePipeline and UpsertPipeline requests with properly escaped content, the following command is a good starting point. It redirects the content to a create-pipeline.json file for easier handling.

jq --arg contents "$(cat config.alloy)" \
   --arg name "myname" \
   --argjson matchers '["collector.os=linux", "team!=team-a"]' \
   --argjson enabled true \
   '.pipeline = {name: $name, contents: $contents, matchers: $matchers, enabled: $enabled}' \
   <<< '{}' > create-pipeline.json

The CreatePipeline response verifies what was created and also lists the unique ID for this pipeline.

curl -q \
    -d @create-pipeline.json \
    -u "user:pass" \
    --header "Content-Type: application/json" \
    -X POST https://fleet-management-prod-001.grafana.net/pipeline.v1.PipelineService/CreatePipeline
{
  "name": "myname",
  "contents": " ... pipeline contents ...",
  "matchers": [
    "collector.os=\"linux\"",
    "team!=\"team-a\""
  ],
  "enabled": true,
  "id": "22234"
}

The ListPipelines request takes an empty payload and returns all available pipelines.

$  curl -q \
    -d '{}' \
    -u "user:pass" \
    --header "Content-Type: application/json" \
    -X POST https://fleet-management-prod-001.grafana.net/pipeline.v1.PipelineService/ListPipelines
{
  "pipelines": [
    {
      "name": "pipeline_1",
      "contents": " ... pipeline contents ...",
      "matchers": [
        "team=\"team-a\""
      ],
      "createdAt": "2024-09-02T12:31:47Z",
      "updatedAt": "2024-09-02T14:08:10Z",
      "enabled": false,
      "id": "22231"
    },
    {
      "name": "pipeline_2",
      "contents": " ... pipeline contents ...",
      "matchers": [
        "collector.os=\"linux\"",
        "team!=\"team-a\""
      ],
      "createdAt": "2024-09-02T16:42:26Z",
      "updatedAt": "2024-09-02T16:42:26Z",
      "enabled": true,
      "id": "22234"
    }
  ]
}

You can use a similar jq command to build an UpdatePipeline request redirected to update-pipeline.json, by adding the unique pipeline ID.

jq --arg contents "$(cat config.alloy)" \
   --arg id "22234" \
   --arg name "myname" \
   --argjson matchers '["collector.os=linux", "team!=team-a"]' \
   --argjson enabled true \
   '.pipeline = {id: $id, name: $name, contents: $contents, matchers: $matchers, enabled: $enabled}' \
   <<< '{}' > update-pipeline.json

The UpdatePipeline response verifies what was updated.

curl -q \
    -d @update-pipeline.json \
    -u "user:pass" \
    --header "Content-Type: application/json" \
    -X POST https://fleet-management-prod-001.grafana.net/pipeline.v1.PipelineService/UpdatePipeline
{
  "name": "myname",
  "contents": " ... new pipeline contents ...",
  "matchers": [
    "collector.os=\"linux\"",
    "team!=\"team-a\""
  ],
  "enabled": true,
  "id": "22234"
}

The GetPipeline and DeletePipeline requests require only an ID for accessing and removing a configuration.

curl -q \
    -d '{"id": "22234"}' \
    -u "user:pass" \
    --header "Content-Type: application/json" \
    -X POST https://fleet-management-prod-001.grafana.net/pipeline.v1.PipelineService/GetPipeline
{
  "name": "myname",
  "contents": "... pipeline contents ...",
  "matchers": [
    "collector.os=\"linux\"",
    "team!=\"team-a\""
  ],
  "createdAt": "2024-09-02T16:42:26Z",
  "updatedAt": "2024-09-02T16:42:26Z",
  "enabled": true,
  "id": "22234"
}

curl -q \
    -d '{"id": "22234"}' \
    -u "user:pass" \
    --header "Content-Type: application/json" \
    -X POST https://fleet-management-prod-001.grafana.net/pipeline.v1.PipelineService/DeletePipeline
{}

The revision RPCs allow for revisiting the history of changes to pipeline objects for a user.

ListPipelinesRevisions lists all changes in chronological order alongside the operation that initiated them. For brevity, the response omits the configuration contents of the pipeline in each snapshot.

Similarly, ListPipelineRevisions allows for viewing the state of a single pipeline ID at every point in time it was changed, along with a full snapshot of the pipeline object.

curl -q \
    -d '{}' \
    -u "user:pass" \
    --header "Content-Type: application/json" \
    -X POST https://fleet-management-prod-001.grafana.net/pipeline.v1.PipelineService/ListPipelinesRevisions
{
  "pipelineRevisions": [
    {
      "revisionId": "1",
      "snapshot": {
        "name": "myname",
        "matchers": [
          "collector.os=\"linux\"",
          "team!=\"team-a\""
        ],
        "enabled": true,
        "id": "1"
      },
      "createdAt": "2025-02-14T09:35:54Z",
      "operation": "INSERT"
    },
    {
      "revisionId": "2",
      "snapshot": {
        "name": "otlp_pipeline",
        "matchers": [
          "collector.os=\"linux\"",
          "team=\"team-b\""
        ],
        "enabled": true,
        "id": "2"
      },
      "createdAt": "2025-02-14T09:36:54Z",
      "operation": "INSERT"
    },
    {
      "revisionId": "3",
      "snapshot": {
        "name": "myname",
        "matchers": [
          "collector.os=\"linux\"",
          "team=\"team-a\""
        ],
        "enabled": true,
        "id": "1"
      },
      "createdAt": "2025-02-14T09:37:49Z",
      "operation": "UPDATE"
    },
    {
      "revisionId": "4",
      "snapshot": {
        "name": "myname",
        "matchers": [
          "collector.os=\"linux\"",
          "team=\"team-a\""
        ],
        "enabled": true,
        "id": "1"
      },
      "createdAt": "2025-02-14T09:38:55Z",
      "operation": "DELETE"
    },
    <...>
  ]
}

curl -q \
    -d '{"id": "2"}' \
    -u "user:pass" \
    --header "Content-Type: application/json" \
    -X POST https://fleet-management-prod-001.grafana.net/pipeline.v1.PipelineService/ListPipelineRevisions
{
  "pipelineRevisions": [
    {
      "revisionId": "2",
      "snapshot": {
        "name": "myname",
        "contents": "prometheus.exporter.self \"alloy\" { }\n\nprometheus.scrape \"alloy\" {\n\ttargets    = prometheus.exporter.self.alloy.targets\n\tforward_to = [prometheus.remote_write.grafanacloud.receiver]\n\n\tscrape_interval = \"60s\"\n}\n\nprometheus.remote_write \"grafanacloud\" {\n\t// Must match the uniquely-identifiable ID set up in the remotecfg block.\n\texternal_labels = {\"collector_id\" = constants.hostname}\n\n\tendpoint {\n\t\turl = \"http://localhost:8081\"\n\n\t\tbasic_auth {\n\t\t\tusername      = \"user\"\n\t\t\tpassword_file = \"/path/to/my/pass\"\n\t\t}\n\t}\n}",
        "matchers": [
          "collector.os=\"linux\"",
          "team=\"team-a\""
        ],
        "enabled": true,
        "id": "2"
      },
      "createdAt": "2025-02-14T09:37:49Z",
      "operation": "UPDATE"
    }
  ]
}

Finally, the GetPipelineRevision request returns a singular revision ID, alongside the contents of the pipeline at the given point in time.

curl -q \
    -d '{"revision_id": "2"}' \
    -u "user:pass" \
    --header "Content-Type: application/json" \
    -X POST https://fleet-management-prod-001.grafana.net/pipeline.v1.PipelineService/GetPipelineRevision
{
  "revisionId": "4",
  "snapshot": {
    "name": "myname",
    "contents": "prometheus.exporter.self \"alloy\" { }\n\nprometheus.scrape \"alloy\" {\n\ttargets    = prometheus.exporter.self.alloy.targets\n\tforward_to = [prometheus.remote_write.grafanacloud.receiver]\n\n\tscrape_interval = \"60s\"\n}\n\nprometheus.remote_write \"grafanacloud\" {\n\t// Must match the uniquely-identifiable ID set up in the remotecfg block.\n\texternal_labels = {\"collector_id\" = constants.hostname}\n\n\tendpoint {\n\t\turl = \"http://localhost:8081\"\n\n\t\tbasic_auth {\n\t\t\tusername      = \"user\"\n\t\t\tpassword_file = \"/path/to/my/pass\"\n\t\t}\n\t}\n}",
    "matchers": [
      "collector.os=\"linux\"",
      "team=\"team-a\""
    ],
    "enabled": true,
    "id": "1"
  },
  "createdAt": "2025-02-14T09:38:55Z",
  "operation": "DELETE"
}

Was this page helpful?

Email docs@grafana.com

Help and support

Community

The Pipeline HTTP API

Find the base URL

Rate limit

PipelineService

Endpoints

CreatePipelineRequest

DeletePipelineRequest

DeletePipelineResponse

GetPipelineRequest

GetPipelineIDRequest

GetPipelineIDResponse

ListPipelinesRequest

Pipeline

Pipelines

PipelineSource

PipelineSource_SourceType

UpdatePipelineRequest

UpsertPipelineRequest

ListPipelinesRevisionsRequest

ListPipelineRevisionsRequest

GetPipelineRevisionRequest

PipelineRevision

PipelineRevision.Operation

PipelineRevisions

SyncPipelinesRequest

SyncPipelinesResponse

Examples

Was this page helpful?

Related resources from Grafana Labs