Troubleshoot Grafana Alloy on Grafana Labs

Monitor the Grafana Alloy component controller

Mon, 30 Mar 2026 15:47:22 +0000

Monitor the Grafana Alloy component controller

The Alloy component controller exposes Prometheus metrics which you can use to investigate the controller state.

Metrics for the controller are exposed at the /metrics HTTP endpoint of the Alloy HTTP server, which defaults to listening on http://localhost:12345.

The documentation for the alloy run command describes how to modify the address Alloy listens on for HTTP traffic.

The controller exposes the following metrics:

alloy_component_controller_evaluating (Gauge): Set to 1 whenever the component controller is evaluating components. This value may be misrepresented depending on how fast evaluations complete or how often evaluations occur.
alloy_component_controller_running_components (Gauge): The current number of running components by health. The health is represented in the health_type label.
alloy_component_evaluation_seconds (Histogram): The time it takes to evaluate components after one of their dependencies is updated.
alloy_component_dependencies_wait_seconds (Histogram): Time spent by components waiting to be evaluated after one of their dependencies is updated.
alloy_component_evaluation_queue_size (Gauge): The current number of component evaluations waiting to be performed.

Monitor components

Mon, 30 Mar 2026 15:47:22 +0000

Monitor components

Alloy components may optionally expose Prometheus metrics which can be used to investigate the behavior of that component. These component-specific metrics are only generated when an instance of that component is running.

Component-specific metrics are different than any metrics processed by the component. Component-specific metrics are used to expose the state of a component for observability, alerting, and debugging.

Component-specific metrics are exposed at the /metrics HTTP endpoint of the Alloy HTTP server, which defaults to listening on http://localhost:12345.

The documentation for the alloy run command describes how to modify the address Alloy listens on for HTTP traffic.

Component-specific metrics have a component_id label matching the component ID generating those metrics. For example, component-specific metrics for a prometheus.remote_write component labeled production has a component_id label with the value prometheus.remote_write.production.

The reference documentation for each component described the list of component-specific metrics that the component exposes. Not all components expose metrics.

Import rendered mixin dashboards

Thu, 02 Apr 2026 19:39:57 +0000

Import rendered mixin dashboards

The Grafana Alloy mixin defines dashboards you can use to monitor collector health, resource use, and internal pipeline metrics. You can import these dashboards from the Grafana Alloy mixin directly into your Grafana instance. The dashboards are available as rendered JSON files in the source repository or in the release archive.

Before you begin

Before you begin, ensure you have the following:

Access to a Grafana instance where you can import dashboards.
Access to rendered mixin files from one of these sources:
- The Alloy source repository at operations/alloy-mixin/rendered/dashboards/.
- The release archive alloy-mixin-dashboards-<RELEASE_TAG>.zip.
A configured Prometheus data source in Grafana for imported dashboards.

Get rendered mixin dashboard files

Choose one source for dashboards:

Source repository: Use the rendered dashboard files located in operations/alloy-mixin/rendered/dashboards/ in your copy or clone of the Alloy repository.
Release archive: Download alloy-mixin-dashboards-<RELEASE_TAG>.zip from an Alloy release artifact and extract the archive.

Import dashboards from JSON files

Use the Grafana dashboard import UI to import each rendered dashboard JSON file:

Open Grafana and go to Dashboards.
Click New and then click Import.
Upload a JSON file from operations/alloy-mixin/rendered/dashboards/ or from the extracted zip archive.
Select the target data source mappings.
Click Import.
Repeat for each dashboard file you want to import.

For full UI details and options, refer to importing dashboards in the Grafana documentation.

Next steps

Refer to Troubleshoot Grafana Alloy for additional troubleshooting tasks.
Refer to the Alloy mixin source for rendered files and customization options.

Generate a support bundle

Mon, 30 Mar 2026 15:47:22 +0000

Generate a support bundle

Note
Generate support bundle isn’t available on Alloy v1.4 and older.

The /-/support?duration=N endpoint returns a support bundle, a compressed file that contains information about a running Alloy instance, and can be used as a baseline of information when trying to debug an issue.

This feature isn’t covered by the backward-compatibility guarantees.

Note
This endpoint is enabled by default, but may be disabled using the --disable-support-bundle runtime flag.

The duration parameter is optional, must be less than or equal to the configured HTTP server write timeout, and if not provided, defaults to it. The endpoint is only exposed to the Alloy HTTP server listen address, which defaults to localhost:12345.

The support bundle contains all information in plain text, so you can inspect it before sharing to verify that no sensitive information has leaked.

In addition, you can inspect the support bundle implementation to verify the code used to generate these bundles.

A support bundle contains the following data:

alloy-components.json contains information about the components running on this Alloy instance, generated by the /api/v0/web/components endpoint.
alloy-environment.txt contains the values of several environment variables relevant to the golang runtime.
alloy-logs.txt contains the logs during the bundle generation.
alloy-metadata.yaml contains the Alloy build version and the installation’s operating system, architecture, and uptime.
alloy-metrics-sample-start.txt contains a snapshot of the internal metrics for Alloy at the start of the bundle collection.
alloy-metrics-sample-end.txt contains a snapshot of the internal metrics for Alloy at the end of the bundle collection.
alloy-peers.json contains information about the identified cluster peers of this Alloy instance, generated by the /api/v0/web/peers endpoint.
alloy-runtime-flags.txt contains the values of the runtime flags available in Alloy.
The pprof/ directory contains Go runtime profiling data (CPU, heap, goroutine, mutex, block profiles) as exported by the pprof package. Refer to the profile documentation for more details on how to use this information.
The sources/ directory contains copies of the local configuration files used to configure Alloy.
sources/remote-config/remote.alloy contains a copy of the last received remote configuration.

Profile Grafana Alloy resource consumption

Mon, 30 Mar 2026 15:47:22 +0000

Profile Grafana Alloy resource consumption

Alloy is written in the Go programming language, which offers built-in support for profiling. Like other applications written in Go, you can profile Alloy by sending an HTTP request, which returns a pprof Go profile.

After you have the pprof file, visualize it as a flame graph in Grafana Pyroscope. Alternatively, you could visualize it on Grafana Play or locally by using Go’s built-in pprof tool.

Note
A profile may contain sensitive information about your environment. You may not want to upload your profiles to a public location.

The port you use to send the HTTP request is controlled by the --server.http.listen-addr command line argument. It’s set to 127.0.0.1:12345 by default.

Obtain a single profile

Different types of HTTP requests retrieve different profiles.

Memory consumption

Goroutine leaks often cause memory leaks. This is why obtaining a goroutine profile is usually necessary when investigating memory issues. For example:

curl localhost:12345/debug/pprof/heap -o heap.pprof
curl localhost:12345/debug/pprof/goroutine -o goroutine.pprof

It’s often helpful to collect profiles both when memory usage is low and when it’s high. You can compare the profiles, and it may be easier to identify what caused the memory consumption to increase.

CPU consumption

If you are experiencing high CPU consumption, you can collect a CPU profile:

curl http://localhost:12345/debug/pprof/profile?seconds=30 -o cpu.pprof

The ?seconds=30 part of the URL above means the profiling continues for 30 seconds.

Continuous profiling

You don’t have to send manual curl commands each time you want to collect profiles. You can also profile continuously using the pyroscope components in Alloy.

If you have very few Alloy instances, you can even configure them to profile themselves. However, if you have a large cluster of collectors, it’s best to set up Alloy instances whose sole job is to profile other Alloy instances.

The following is an example of an Alloy instance profiling itself:

pyroscope.scrape "default" {
  targets    = [{"__address__" = "localhost:12345", "service_name"="alloy"}]
  forward_to = [pyroscope.write.default.receiver]
}

pyroscope.write "default" {
  endpoint {
    url = "https://profiles-prod-014.grafana.net"
    basic_auth {
      username = sys.env("PYROSCOPE_USERNAME")
      password = sys.env("PYROSCOPE_PASSWORD")
    }
  }
}

Expected resource consumption

Refer to Estimate resource usage for more information about the expected resource consumption in Alloy.

Alloy consumes an abnormally large amount of resources

If Alloy consumes an abnormally large amount of resources, you can open an issue in the Alloy repository. Attach your pprof files and your Alloy configuration file. Make sure you redact any secrets in the attachments.

Debug Grafana Alloy

Mon, 30 Mar 2026 15:47:22 +0000

Debug Grafana Alloy

Follow these steps to debug issues with Alloy:

Use the Alloy UI to debug issues.
If the Alloy UI doesn’t help with debugging an issue, you can examine the logs.

Alloy UI

Alloy includes an embedded UI viewable from the Alloy HTTP server, which defaults to listening at http://localhost:12345.

Note
For security reasons, installations of Alloy on non-containerized platforms default to listening on localhost. This default prevents other machines on the network from being able to view the UI.

To expose the UI to other machines on the network on non-containerized platforms, refer to the documentation for how you installed Alloy.

If you are running a custom installation of Alloy, refer to the documentation for the run command to learn how to change the HTTP listen address and pass the appropriate flag when running Alloy.

Home page

The home page shows a table of components defined in the configuration file and their health.

Click View on a row in the table to navigate to the Component detail page for that component.

Click the Alloy logo to navigate back to the home page.

Graph page

EXPERIMENTAL: This is an experimental feature. Experimental features are subject to frequent breaking changes, and may be removed with no equivalent replacement. To enable and use an experimental feature, you must set the stability.level flag to experimental.

The Graph page shows a graph view of components defined in the configuration file. Clicking a component in the graph navigates to the Component detail page for that component.

You can click and drag the components to move them around.

To access the graph page of a module, click on the Graph button on the module’s detail page.

The amount of data that exits a component that supports live debugging is shown on the outgoing edges of the component. The data is refreshed according to the window parameter.

Component detail page

The component detail page shows the following information for each component:

The health of the component with a message explaining the health.
The current evaluated arguments for the component.
The current exports for the component.
The current debug info for the component if the component has debug info.

From there you can also go to the component documentation or to its corresponding Live Debugging page.

Note
Values marked as a secret display only as the text (secret).

Clustering page

The clustering page shows the following information for each cluster node:

The node’s name.
The node’s advertised address.
The node’s current state: Viewer, Participant, or Terminating.
The local node that serves the UI.

Live Debugging page

Live debugging provides a real-time stream of debugging data from a component. You can access this page from the corresponding Component detail page.

Caution
Live debugging defaults to disabled to avoid accidentally displaying sensitive telemetry data. To enable live debugging, configure the livedebugging block.

Live debugging allows you to do the following:

Pause and clear the data stream.
Sample data and disable auto-scrolling to handle heavy loads.
Search through the data using keywords.
Copy the entire data stream to the clipboard.

The format and content of the debugging data vary depending on the component type.

Note
Live debugging isn’t yet available in all components.

Supported components:

loki.process

loki.relabel

loki.secretfilter

otelcol.connector.*

otelcol.processor.*

otelcol.receiver.*

prometheus.remote_write

prometheus.relabel

discovery.*

prometheus.scrape

Debug using the UI

To debug using the UI:

Ensure that no component reports as unhealthy.
Ensure that the arguments and exports for misbehaving components appear correct.
Ensure that the live debugging data meets your expectations.

Examine logs

Alloy provides different log levels that help you determine the root cause of issues. You can configure the log level using the logging configuration block in your Alloy configuration file.

Logs from Alloy are written in logfmt format by default. You can configure the log format to be either logfmt or json. You can retrieve the logs in different ways depending on your platform and installation method.

Common log messages

The following log messages are normal during Alloy operation:

Component lifecycle messages

During normal startup and operation, you’ll see messages about component lifecycle:

Component startup and initialization:

level=info msg="starting controller"
level=info msg="starting server"
level=info msg="starting server" addr=localhost:8080
level=info msg="started scheduled components"

Note
The starting server messages refer to the built-in HTTP server that hosts the debugging UI, /metrics endpoint, and other debugging endpoints.

Component updates and configuration changes:

level=info msg="configuration loaded"
level=info msg="module content loaded"
level=info msg="started scheduled components"
level=info msg="terminating server"

Component health reporting:

level=info msg="started scheduled components"
level=warn msg="failed to start scheduled component" err="connection refused"
level=warn msg="the discovery.process component only works on linux; enabling it otherwise will do nothing"

Cluster operation messages

If you enable clustering, you’ll see messages about cluster operations:

Normal startup and peer discovery:

level=info msg="starting cluster node" peers_count=2 peers=192.168.1.10:12345,192.168.1.11:12345 advertise_addr=192.168.1.12:12345
level=info msg="using provided peers for discovery" join_peers="192.168.1.10:12345, 192.168.1.11:12345"
level=info msg="discovered peers" peers_count=3 peers=192.168.1.10:12345,192.168.1.11:12345,192.168.1.12:12345
level=info msg="rejoining peers" peers_count=2 peers=192.168.1.10:12345,192.168.1.11:12345

Cluster size management:

level=info msg="minimum cluster size reached, marking cluster as ready to admit traffic" minimum_cluster_size=3 peers_count=3
level=warn msg="minimum cluster size requirements are not met - marking cluster as not ready for traffic" minimum_cluster_size=3 minimum_size_wait_timeout=5m0s peers_count=2
level=warn msg="deadline passed, marking cluster as ready to admit traffic" minimum_cluster_size=3 minimum_size_wait_timeout=5m0s peers_count=2

Normal cluster operations:

level=debug msg="found an IP cluster join address" addr=192.168.1.10:12345
level=debug msg="received DNS query response" addr=cluster.example.com record_type=A records_count=3

Expected warnings

Some warnings are normal during startup or cluster changes:

level=warn msg="failed to get peers to join at startup; will create a new cluster" err="no peers available"
level=warn msg="failed to connect to peers; bootstrapping a new cluster" err="connection refused"
level=warn msg="failed to resolve provided join address" addr=unavailable-node:12345

Problematic messages

These messages indicate issues that require attention:

level=error msg="failed to bootstrap a fresh cluster with no peers" err="bind: address already in use"
level=error msg="failed to rejoin list of peers" err="connection timeout"
level=warn msg="failed to refresh list of peers" err="dns resolution failed"

Debug clustering issues

To debug issues when using clustering, check for the following symptoms.

Cluster not converging: The cluster peers aren’t converging on the same view of their peers’ status. This is most likely due to network connectivity issues between the cluster nodes. Use the Alloy UI of each running peer to understand which nodes aren’t picked up correctly.
Cluster split brain: The cluster peers aren’t aware of one another, thinking they’re the only node present. Again, check for network connectivity issues. Check that the addresses or DNS names given to the node to join are correctly formatted and reachable.
Configuration drift: Clustering assumes that all nodes are running with the same configuration file at roughly the same time. Check the logs for issues with the reloaded configuration file as well as the graph page to verify that the changes are applied.
Node name conflicts: Clustering assumes all nodes have unique names. The cluster rejects nodes with conflicting names and they won’t join the cluster. Look at the clustering UI page for the list of current peers with their names, and check the logs for any reported name conflict events.
Node stuck in terminating state: The node attempted to gracefully shut down and set its state to Terminating, but it hasn’t completely gone away. Check the clustering page to view the state of the peers and verify that the terminating Alloy has been shut down.

Note
Some issues that appear to be clustering issues may be symptoms of other issues, for example, problems with scraping or service discovery can result in missing metrics for an Alloy instance that you can interpret as a node not joining the cluster.