Increase efficiency with reusable configuration pipelines

Save time and effort by reusing Grafana Fleet Management configuration pipelines across your fleet.

Set environment variables for properties that vary and create a pipeline that reads those variables.
Inject collector attributes into configuration pipelines to label telemetry.
Use collector attributes to customize how pipelines are applied.

Configuration properties with different values

Configuring a fleet of collectors can be cumbersome, especially when configuration contexts, such as host credentials or scrape targets, differ. To avoid creating multiple versions of the same pipeline, Fleet Management can help you efficiently reuse standard configuration pipelines across deployments, with the help of environment variables. Set variables where your collector runs and read those variables in the pipeline. Grafana Alloy uses the sys.env configuration function. The OpenTelemetry Collector uses environment variable substitution.

Example: Scrape different targets

For example, if you want to collect metrics on all deployments but you need to scrape different targets, set a target environment variable on each host, refer to the variable in the pipeline, and then roll out the pipeline to your collectors:

Alloy scrape configuration

Use sys.env to read the target location from the host environment:

prometheus.scrape "example" {
  targets = [
    {"__address__" = sys.env("TARGET_LOCATION")},
  ]
  forward_to = [prometheus.remote_write.staging.receiver]
}

OpenTelemetry Collector scrape configuration

Set the TARGET_LOCATION environment variable on each collector, reference it in the Prometheus receiver configuration, and then roll out the same configuration to all collectors:

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: example
          static_configs:
            - targets:
                - ${env:TARGET_LOCATION}

exporters:
  prometheus_remote_write:
    endpoint: "https://<your-endpoint>"

service:
  pipelines:
    metrics:
      receivers: [prometheus]
      exporters: [prometheus_remote_write]

Telemetry labels based on collector attributes

When you assign attributes to a collector, you are categorizing them based on meaningful characteristics, such as team ownership or environment. You might also want to persist those attributes in the labels you apply to your collected data.

Alloy supports injecting collector attributes into pipelines, allowing you to reuse a single pipeline rather than creating separate pipelines for each set of labels.

Example: Add telemetry labels for owner and environment

For example, if you’ve assigned attributes to collectors for owner and env and want to export metrics with corresponding labels, relabel the telemetry in the pipeline and then roll it out to your collectors.

Alloy attribute injection

If you’re configuring an Alloy instance, use argument.attributes.value to read collector attributes and add them as telemetry labels:

discovery.relabel "example" {
  targets = prometheus.exporter.self.example.targets

  rule {
    source_labels = ["__address__"]
    target_label  = "owner"
    replacement   = argument.attributes.value["owner"]
  }

  rule {
    source_labels = ["__address__"]
    target_label  = "env"
    replacement   = argument.attributes.value["env"]
  }
}

Customized pipelines by application workload or technology

Match configuration pipelines to your collectors based on the applications or technologies that are running on the host. You can add attributes to the collector that match to each workload or technology. If you have standard telemetry that needs to be collected from all hosts, such as server or database monitoring, create a single default configuration pipeline that applies to all collectors on all machines with a universal matching attribute. If you remove an application from a host, remove the attribute from that host’s collector to disable the pipeline for that instance.

Example: Apply the Node.js integration to hosts running your application

Collect metrics from all hosts where your Node.js application is running.

Add the attribute application=MYAPP to each collector where the application is hosted. You can add a local attribute directly to the remotecfg block of the local configuration or add a remote attribute in the Fleet Management application or with an API request.
Follow the instructions to create a new configuration pipeline using an integration template and select the Node.js integration.
Add the matching attribute application=MYAPP to the pipeline.
Activate the pipeline to begin collecting telemetry from your Node.js applications.

Example: Apply the MySQL integration to every collector

If all of your machines are running MySQL, use a default configuration pipeline to collect metrics and logs from them.

Add the attribute default=MYSQL to every collector by selecting all collectors in the Fleet Management application and clicking the bulk edit tool. Make sure to scroll through your entire fleet to load all your collectors before selecting them. You can also make a BulkUpdateCollectorsRequest to the Collector API.
Follow the instructions to create a new configuration pipeline using an integration template and select the MySQL integration.
Review the configuration and set up any parameters required. For MySQL, the default configuration expects the connection string to be available in a file at /var/lib/alloy/mysql-secret.
Add the matching attribute default=MYSQL to the pipeline.
Activate the pipeline to begin collecting telemetry from your MySQL databases.