Documentationbreadcrumb arrow Grafana Cloudbreadcrumb arrow Instrument and send databreadcrumb arrow Fleet Managementbreadcrumb arrow Manage your fleetbreadcrumb arrow Collectorsbreadcrumb arrow Troubleshootbreadcrumb arrow Internal logs
Grafana Cloud

Internal logs

When you visit the Grafana Fleet Management interface in Grafana Cloud, a self-monitoring pipeline is automatically created and assigned to collect internal logs:

  • For Alloy collectors, this is a self_monitoring_logs_* pipeline assigned based on the collector’s operating system or platform.
  • For OpenTelemetry Collectors, this is the self_monitoring_otelcol pipeline.

View logs for a registered collector

The Fleet Management service displays internal logs in the application interface.

  1. In your Grafana Cloud instance, click Connections > Fleet Management in the left navigation menu.
  2. Search, filter, or sort to find the collector in your Inventory.
  3. Click on the collector to open the details view drawer and switch to the Logs tab.
  4. Browse the logs or use the case-insensitive search box to look for specific log lines or error messages.

Note

If you don’t see any log records, refer to Logs don’t appear in app (Alloy or OpenTelemetry Collector) for help.

Error-level logs

The following log messages can help you diagnose collector errors.

Alloy

Log messageCauseFix
failed to fetch remote configuration from the APIAlloy cannot reach the Fleet Management API or could not load the latest configuration.Check if the API is operational. Check for authentication errors, such as an invalid token. Check configuration for invalid syntax or missing components and revert if necessary.
failed to fetch remote configVariant of previous message that is logged when the error occurs during Alloy startup or when the remotecfg block arguments are changed.Check if the API is operational. Check for authentication errors, such as an invalid token. Check configuration for invalid syntax or missing components and revert if necessary.
failed to read from cacheAlloy cannot access or read the locally cached remote configuration file.If logged at startup, check if the file exists. Alternatively, check for a permissions issue.
failed to load from cacheAlloy cannot load the contents it read from the locally cached remote configuration file.Check if the file has been truncated or corrupted.
failed to flush remote configuration contents to the on-disk cacheAlloy cannot write new configuration to the locally cached remote configuration file.Check for a permissions issue.

OpenTelemetry Collector

Log messageCauseFix
error found during service initializationCollector service initialization failed after logger creation.Search nearby startup errors for the root cause. This line is useful as a startup-failure marker, but the wrapped error usually carries the actionable detail.
Config watch failedCollector cannot continue watching for configuration changes.Check Fleet Management API reachability, auth/token validity, and remote configuration retrieval health. If this appears after a config change, inspect the latest assigned pipeline revision.
Asynchronous error received, terminating processA component reported a fatal asynchronous error and the Collector is exiting.Use adjacent log lines and component attributes to identify the component. Check recent pipeline changes, component credentials, endpoints, and resource limits.
Failed to start componentA receiver, processor, exporter, or connector failed during startup.Check the type and id fields to identify the failing component, then validate that component’s configuration, endpoint, credentials, storage extension, or network dependency.

Debug-level logs

The following log messages can help you identify the source and context of events to debug collector issues.

Alloy

Log messageEvent
fetching remote configurationAlloy reaches a polling interval and requests the current remote configuration from the API.
skipping over API response since it has not been modified since last fetchThe API response to a GetConfig request is the same as the most recent locally cached remote configuration file.
skipping over API response since it matched the last loaded oneThe API response to a GetConfig request is the same as the last loaded remote configuration. The last loaded remote configuration might differ from the most recent locally cached remote configuration if the cached file had an error. Alloy always loads and runs the last known good configuration if it encounters an error. If the error is corrected with a rollback in the UI, an API response with the corrected configuration might match the last known good configuration, meaning there’s no need to reload the configuration.

OpenTelemetry Collector

Log messageEvent
Preparing to make HTTP requestThe OTLP HTTP exporter is about to send to a specific URL.
Currently used memory.Memory limiter sampled current heap allocation. Use with memory pressure errors or refusal events to see whether memory is approaching configured limits.

Query for pipeline logs

For Alloy collectors, you can isolate logs from a specific pipeline using queries in the Grafana Cloud Explore application.

  1. In your Grafana Cloud instance, click Explore in the left navigation menu.
  2. Select your logs datasource from the dropdown menu. The datasource name should be similar to grafanacloud-<instance_name>-logs.
  3. Switch the query editor from Builder to Code.
  4. Copy and paste one of the following queries into the box. Make sure to substitute your own pipeline name for the placeholder. Click Run query.

Find component logs for a specific pipeline:

LogQL 
{job="integrations/alloy"} | collector_id=~".+" |= "component_path=/remotecfg/<pipeline_name>.default"

Find logs emitted when a pipeline is refreshed or has an issue loading:

LogQL 
{job="integrations/alloy"} | collector_id=~".+" |= "controller_path=/remotecfg" |=  "controller_id=<pipeline_name>.default"

Find logs emitted when a pipeline can’t be loaded at all:

LogQL 
{job="integrations/alloy"} | collector_id=~".+" |= "controller_path=/" |= "controller_id=remotecfg" |= "node=<pipeline_name>.default"

Note

If you’re running Alloy on Kubernetes without the Kubernetes Monitoring Helm chart, replace job="integrations/alloy" with job="<namespace>/alloy", substituting your namespace name. If you’re not sure which value your logs use, match both with job=~"integrations/alloy|<namespace>/alloy" instead. If you’ve customized the job label on your Alloy log streams, use your custom value.

The logging block

Remote configuration does not support top-level configuration blocks, which means that the logging block cannot be used in Fleet Management configuration pipelines.

If you want to use the logging block, you must do so by adding the block to the collector’s local configuration file.

Log levels

The default log level and how to change it differ between collector types.

Alloy

The self_monitoring_logs* pipelines do not support a change in log level. If you wish to change log levels for debugging, you must use the logging block in the collector’s local configuration.

OpenTelemetry Collector

The self_monitoring_otelcol pipeline configures a debug log level. To change the log level, update the service::telemetry::logs::level key to the self-monitoring configuration pipeline:

YAML 
service:
  telemetry:
    logs:
      level: error