Note
Fleet Management is currently in public preview. Grafana Labs offers limited support, and breaking changes might occur prior to the feature being made generally available. For bug reports or questions, fill out our feedback form.
Troubleshoot onboarding issues
This page describes how to address issues you might encounter during the collector onboarding process.
Collector isn’t in Inventory
If you’ve registered a collector with Fleet Management, but it doesn’t appear in your Inventory, you can test if the remotecfg
block is working properly.
Check that the collector is running.
If it’s not running, start the collector.
If the collector is running, ping the collector’s root
/metrics
endpoint and look forremotecfg_*
metrics.curl --no-progress-meter http://localhost:12345/metrics | grep remotecfg_
If you see
remotecfg_*
metrics, but your collector still doesn’t appear in the inventory, look for the metricsremotecfg_last_load_successful
andremotecfg_load_failures_total
to see if failures are reported.If
remotecfg
failed to load, make sure the configuration block has the correct syntax, arguments, and values.If you still need more debugging information, review the log output for the collector, looking for lines with the label
service=remotecfg
.
Collector was deleted from Inventory
If you have mistakenly deleted a collector from the Fleet Management interface, you can recover it by restarting the collector.
As long as you haven’t removed the remotecfg
block from its local configuration and you have explicitly set the id
value, the collector re-registers itself with the Fleet Management service on startup.
Deleting a collector deletes any custom attributes you assigned to it in the Fleet Management application. You need to recreate these custom attributes so that the correct configuration pipelines are matched to the recovered collector.
Restarting a collector could result in lost custom attributes if the collector’s id
changes.
For example, if you have not explicitly set the id
field in the collector’s remotecfg
block, Alloy generates a universally unique identifier (UUID) for the collector.
The UUID is reset if the Alloy-generated seed file is not persisted, so when the collector restarts, it registers with a new UUID and appears as a new collector.
Health dashboards have no data
If the health tab of your collector’s details view says there is no data, make sure the collector is configured correctly for internal metrics.
Note
There is a delay in data availability after registering a new collector. If you’ve just completed enrollment and configuration, wait a few more minutes to see if data begins to flow.
If several minutes have passed and you still don’t see data, click on the Remote configuration tab in the Fleet Management app.
Check if the
self_monitoring_metrics
configuration pipeline exists and is enabled.If the pipeline doesn’t exist, you can make the application recreate it by deleting all of the
self_monitoring_*
autogenerated pipelines. When the last pipeline is deleted, Fleet Management recreates them all. After the pipeline is created, wait a few minutes for the collector to poll for an updated configuration and for data to begin flowing. The polling frequency is set in theremotecfg
block.If the pipeline exists but is not enabled, click the switch to turn it on. Wait for the collector to poll for an updated configuration.
If the pipeline is enabled, make sure the collector has the necessary matching attributes to receive the configuration pipeline. You can see a collector’s attributes by clicking on the collector in the Inventory tab.
If the collector has the necessary attributes, make sure that all of the required environment variables are set.
GCLOUD_FM_COLLECTOR_ID
must match the value of theid
argument in the collector’sremotecfg
block.GCLOUD_RW_API_KEY
must match thebasic_auth
password in the collector’sremotecfg
block.
If the environment variables are set, check the remote configuration assigned to the collector.
- On the Inventory tab, click the collector’s row to open the details view.
- Switch to the Configuration tab and look for a
declare
block that names the self-monitoring metrics pipeline. For example:
declare "self_monitoring_metrics" {
You can also check the collector’s data storage to confirm it received the self-monitoring pipeline.
- Navigate to Alloy’s data storage.
The default path is
data-alloy/
. If you set a different path using the--storage.path
flag, make sure to look there. - Change to the directory named
remotecfg
and list its contents. Each file in this directory contains a complete remote configuration that the collector received from Fleet Management based on a givenremotecfg
block. If you change anyremotecfg
arguments, a new file is created. - Open the files and look for a
declare
block that names the self-monitoring metrics pipeline. For example:
declare "self_monitoring_metrics" {
- Navigate to Alloy’s data storage.
The default path is
If you still need help, refer to Debug Grafana Alloy.
Logs don’t appear in app
If you don’t see your collector’s logs in the Fleet Management application, make sure the collector is configured correctly for internal logs.
Note
There is a delay in data availability after registering a new collector. If you’ve just completed enrollment and configuration, wait a few more minutes to see if data begins to flow.
If several minutes have passed and you still don’t see data, click on the Remote configuration tab in the Fleet Management app.
Check if the
self_monitoring_logs_*
configuration pipeline exists and is enabled. The logs pipeline assigned to your collector depends on its operating system.If the pipeline doesn’t exist, you can make the application recreate it by deleting all of the
self_monitoring_*
autogenerated pipelines. When the last pipeline is deleted, Fleet Management recreates them all. After the pipeline is created, wait a few minutes for the collector to poll for an updated configuration and for data to begin flowing. The polling frequency is set in theremotecfg
block.If the pipeline exists but is not enabled, click the switch to turn it on. Wait for the collector to poll for an updated configuration.
If the pipeline is enabled, make sure the collector has the necessary matching attributes to receive the configuration pipeline. You can see a collector’s attributes by clicking on the collector in the Inventory tab.
If the collector has the necessary attributes, make sure that the required environment variables are set.
GCLOUD_FM_COLLECTOR_ID
must match the value of theid
argument in the collector’sremotecfg
block.GCLOUD_RW_API_KEY
must match thebasic_auth
password in the collector’sremotecfg
block.GCLOUD_FM_LOG_PATH
(on macOS systems) must be set to the path of the log file. For example,$PWD/alloy.log
.
If the environment variables are set, check the remote configuration assigned to the collector.
- On the Inventory tab, click the collector’s row to open the details view.
- Switch to the Configuration tab and look for a
declare
block that names the self-monitoring logs pipeline. For example:
declare "self_monitoring_logs_linux" {
You can also check the collector’s data storage to confirm it received the self-monitoring pipeline.
- Navigate to Alloy’s data storage.
The default path is
data-alloy/
. If you set a different path using the--storage.path
flag, make sure to look there. - Change to the directory named
remotecfg
and list its contents. Each file in this directory contains a complete remote configuration that the collector received from Fleet Management based on a givenremotecfg
block. If you change anyremotecfg
arguments, a new file is created. - Open the files and look for a
declare
block that names the self-monitoring logs pipeline. For example:
declare "self_monitoring_logs_linux" {
- Navigate to Alloy’s data storage.
The default path is
If you still need help, refer to Debug Grafana Alloy.
Logs in Kubernetes
Support for Kubernetes in Fleet Management is under active development. Internal log collection is not yet supported out of the box for Kubernetes deployments.
Alerts don’t appear in app
Active alerts appear in the Fleet Management application in the collector details view. If you know an alert is active, but you don’t see it there, check to make sure the alert is properly labeled.