Documentation Grafana Cloud Instrument and send data Fleet Management Set up Configuration pipelines Self-monitoring pipelines

Self-monitoring pipelines

The first time you access the Grafana Fleet Management interface in Grafana Cloud, a set of configuration pipelines is autogenerated. These autogenerated pipelines, whose names begin with self_monitoring_*, collect important internal metrics and real-time logs from registered collectors and send them to Grafana Cloud. This internal telemetry populates the preinstalled dashboards and log displays in Fleet Management so you can centrally monitor the health of your fleet.

Update self-monitoring pipelines

Fleet Management creates self-monitoring pipelines in your stack, but it does not overwrite them when newer default versions become available. To update to the latest default versions, delete all autogenerated pipelines whose names begin with self_monitoring_* from the Remote configuration tab. When the last autogenerated self-monitoring pipeline is deleted, Fleet Management recreates them all with the latest versions.

With the release of the Kubernetes Monitoring Helm chart v4.1, a new default version of the self-monitoring logs pipeline was also released. Use the following guidance to decide if you should update the self-monitoring pipelines in your stack:

Environment variables

The self-monitoring pipelines rely on environment variables to authenticate to the Fleet Management service and assign data to the correct collector.

• If you installed Alloy and enabled Fleet Management using the Grafana Cloud Alloy Install workflow, the workflow command exports the necessary variables for you.
• If you enabled Fleet Management on existing collectors or installed Alloy using other methods, make sure to export the following environment variables wherever Alloy is running.

GCLOUD_RW_API_KEY

The GCLOUD_RW_API_KEY variable represents an access token that authenticates to Fleet Management, reads Alloy configuration, and writes to the relevant database.

The token should have the following scopes for health metrics and logs to appear in Fleet Management:

• fleet-management:read
• metrics:write
• logs:write

Add these additional scopes if you want to use the same access token for all telemetry:

• traces:write
• profiles:write

GCLOUD_FM_LOG_PATH (MacOS only)

The GCLOUD_FM_LOG_PATH variable is used in the self_monitoring_logs_macos pipeline. If you are collecting logs from a collector running on MacOS, set this variable to the absolute path of the Alloy log file.

Kubernetes-specific variables

The self_monitoring_logs_kubernetes pipeline uses Kubernetes pod discovery to collect logs from the Alloy pod. If you run Alloy in Kubernetes, set the following environment variables wherever Alloy is running:

• GCLOUD_RW_API_KEY: Set this variable to a Grafana Cloud access token that includes write access to Loki.
• NAMESPACE: Set this variable to the Kubernetes namespace of the running pod.
• POD_NAME: Set this variable to the name of the running pod.
• HOSTNAME: Set this variable to the name of the Kubernetes node hosting the pod.

If you deploy Alloy with the Kubernetes Monitoring Helm chart v4.1 or later, the chart sets these variables for you. If you use an older Kubernetes Monitoring Helm chart or the Grafana Alloy Helm chart, you might need to set these variables for the pipeline to work.

Self-monitoring logs

The Fleet Management application finds your collector’s logs by their job label, then narrows the results to the current collector. Because the application matches collector_id after locating the log streams, you can send collector_id as structured metadata instead of as an indexed Loki label. Use structured metadata to avoid stream cardinality issues.

Fleet Management looks for logs whose job label is one of these values:

• integrations/alloy: emitted by the autogenerated self_monitoring_logs_* pipelines for standalone Alloy installs.
• <namespace>/alloy: the value used by the Kubernetes Monitoring Helm chart, where namespace comes from the collector’s reported attribute.

If you customize the job label on your Alloy log streams, the application log views can’t find them. You can still query the streams directly in Explore using your custom labels.

Troubleshoot lack of self-monitoring data

If you’ve registered collectors and exported the necessary environment variables, but you don’t see data in the dashboards or logs display in Fleet Management, refer to the following pages for tips on how to troubleshoot:

• Health dashboards have no data
• Logs don’t appear in app