Documentation Grafana Cloud Instrument and send data Fleet Management Set up Configuration pipelines Self-monitoring pipelines

Self-monitoring pipelines

The first time you access the Grafana Fleet Management interface in Grafana Cloud, a set of configuration pipelines is autogenerated. These autogenerated pipelines, whose names begin with self_monitoring_*, collect important internal metrics and real-time logs from registered collectors and send them to Grafana Cloud. This internal telemetry populates the preinstalled dashboards and log displays in Fleet Management so you can centrally monitor the health of your fleet.

Update self-monitoring pipelines

Fleet Management creates self-monitoring pipelines in your stack, but it does not overwrite them when newer default versions become available. To update to the latest default versions, delete all autogenerated pipelines whose names begin with self_monitoring_* from the Remote configuration tab. When the last autogenerated self-monitoring pipeline is deleted, Fleet Management recreates them all with the latest versions.

With the release of the Kubernetes Monitoring Helm chart v4.1, a new default version of the self-monitoring logs pipeline was also released. Use the following guidance to decide if you should update the self-monitoring pipelines in your stack:

Environment variables

The self-monitoring pipelines rely on environment variables to authenticate to the Fleet Management service and assign data to the correct collector.

• If you installed Alloy and enabled Fleet Management using the Grafana Cloud Alloy Install workflow, the workflow command exports the necessary variables for you.
• If you enabled Fleet Management on existing collectors or installed your collector using other methods, make sure to export the following environment variables wherever your collector is running.

GCLOUD_RW_API_KEY (Alloy)

The GCLOUD_RW_API_KEY variable represents an access token that authenticates to Fleet Management, reads configuration, and writes to the relevant database.

The token should have the following scopes for health metrics and logs to appear in Fleet Management:

• fleet-management:read
• metrics:write
• logs:write

Add these additional scopes if you want to use the same access token for all telemetry:

• traces:write
• profiles:write

GCLOUD_BASIC_AUTH_BASE64 (OpenTelemetry Collector)

The GCLOUD_BASIC_AUTH_BASE64 variable is used by the OpenTelemetry self-monitoring pipelines in place of GCLOUD_RW_API_KEY. OpenTelemetry Collectors authenticate to Grafana Cloud using an HTTP Basic Auth header, which requires a base64-encoded credential string rather than a raw token.

To generate the value, combine your Grafana Cloud instance ID and access token, then base64-encode the result:

sh Copy

echo -n "<INSTANCE_ID>:<API_KEY>" | openssl base64 -A

Replace <INSTANCE_ID> with your Grafana Cloud instance ID and <API_KEY> with the access token you created for Fleet Management. Set the output as the value of GCLOUD_BASIC_AUTH_BASE64 wherever your OpenTelemetry Collector is running.

The token should have the set:otel-data-write scope, which includes:

• fleet-management:read
• metrics:write
• logs:write
• traces:write
• profiles:write

GCLOUD_FM_LOG_PATH (MacOS only)

The GCLOUD_FM_LOG_PATH variable is used in the self_monitoring_logs_macos pipeline. If you are collecting logs from a collector running on MacOS, set this variable to the absolute path of the Alloy log file.

Kubernetes-specific variables (Alloy)

The self_monitoring_logs_kubernetes pipeline uses Kubernetes pod discovery to collect logs from the Alloy pod. If you run Alloy in Kubernetes, set the following environment variables wherever Alloy is running:

• GCLOUD_RW_API_KEY: Set this variable to a Grafana Cloud access token that includes write access to Loki.
• NAMESPACE: Set this variable to the Kubernetes namespace of the running pod.
• POD_NAME: Set this variable to the name of the running pod.
• HOSTNAME: Set this variable to the name of the Kubernetes node hosting the pod.

If you deploy Alloy with the Kubernetes Monitoring Helm chart v4.1 or later, the chart sets these variables for you. If you use an older Kubernetes Monitoring Helm chart or the Grafana Alloy Helm chart, you might need to set these variables for the pipeline to work.

Self-monitoring logs

Fleet Management locates and matches logs differently depending on the collector type.

Alloy

The Fleet Management application finds your Alloy collector’s logs by their job label, then narrows the results to the current collector. Because the application matches collector_id after locating the log streams, you can send collector_id as structured metadata instead of as an indexed Loki label. Use structured metadata to avoid stream cardinality issues.

Fleet Management looks for logs whose job label is one of these values:

• integrations/alloy: used by standalone Alloy installs and by Kubernetes installs that use the Kubernetes Monitoring Helm chart, which configures its Alloy pods to use this value.
• <namespace>/alloy: the default value on Kubernetes when Alloy is deployed another way and not with the Kubernetes Monitoring Helm chart. The namespace value comes from the collector’s reported attribute.

For Kubernetes collectors, the Fleet Management application matches either value, so it doesn’t matter which deployment method you used. This match relies on the collector’s reported namespace attribute matching its pod’s actual namespace in Kubernetes. If you’ve overridden the namespace attribute to a different value in Fleet Management, or haven’t set it at all, only the integrations/alloy value can be used to match logs to collectors.

If you customize the job label on your Alloy log streams, the application log views can’t find them. You can still query the streams directly in Explore using your custom labels.

OpenTelemetry Collector

OpenTelemetry Collectors export their internal logs directly to Grafana Cloud over OTLP using the GCLOUD_BASIC_AUTH_BASE64 environment variable for authentication. Fleet Management finds and matches OpenTelemetry Collector logs by the service_instance_id stream label, which is derived from the service.instance.id resource attribute that the Collector sets automatically. No job label configuration is required.

Troubleshoot lack of self-monitoring data

If you’ve registered collectors and exported the necessary environment variables, but you don’t see data in the dashboards or logs display in Fleet Management, refer to the following pages for tips on how to troubleshoot:

• Health dashboards have no data (Alloy)
• Health dashboards have no data (OpenTelemetry Collector)
• Logs don’t appear in app (Alloy)
• Logs don’t appear in app (OpenTelemetry Collector)