Documentation Grafana Alloy Troubleshoot Debug

Debug Grafana Alloy

Follow these steps to debug issues with Alloy:

1. Use the Alloy UI to debug issues.
2. If the Alloy UI doesn’t help with debugging an issue, logs can be examined instead.

Alloy UI

Alloy includes an embedded UI viewable from the Alloy HTTP server, which defaults to listening at http://localhost:12345.

Home page

The home page shows a table of components defined in the configuration file and their health.

Click View on a row in the table to navigate to the Component detail page for that component.

Click the Alloy logo to navigate back to the home page.

Graph page

The Graph page shows a graph view of components defined in the configuration file and their health. Clicking a component in the graph navigates to the Component detail page for that component.

Component detail page

The component detail page shows the following information for each component:

• The health of the component with a message explaining the health.
• The current evaluated arguments for the component.
• The current exports for the component.
• The current debug info for the component (if the component has debug info).

From there you can also go to the component documentation or to its corresponding Live Debugging page.

Clustering page

The clustering page shows the following information for each cluster node:

• The node’s name.
• The node’s advertised address.
• The node’s current state (Viewer/Participant/Terminating).
• The local node that serves the UI.

Live Debugging page

Live debugging provides a real-time stream of debugging data from a component. You can access this page from the corresponding Component detail page.

Live debugging allows you to do the following:

• Pause and clear the data stream.
• Sample data and disable auto-scrolling to handle heavy loads.
• Search through the data using keywords.
• Copy the entire data stream to the clipboard.

The format and content of the debugging data vary depending on the component type.

Debugging using the UI

To debug using the UI:

• Ensure that no component is reported as unhealthy.
• Ensure that the arguments and exports for misbehaving components appear correct.
• Ensure that the live debugging data meets your expectations.

Examining logs

Logs may also help debug issues with Alloy.

To reduce logging noise, many components hide debugging info behind debug-level log lines. It is recommended that you configure the logging block to show debug-level log lines when debugging issues with Alloy.

The location of Alloy logs is different based on how it’s deployed. Refer to the logging block page to see how to find logs for your system.

Debugging clustering issues

To debug issues when using clustering, check for the following symptoms.

• Cluster not converging: The cluster peers aren’t converging on the same view of their peers’ status. This is most likely due to network connectivity issues between the cluster nodes. Use the Alloy UI of each running peer to understand which nodes aren’t being picked up correctly.
• Cluster split brain: The cluster peers aren’t aware of one another, thinking they’re the only node present. Again, check for network connectivity issues. Check that the addresses or DNS names given to the node to join are correctly formatted and reachable.
• Configuration drift: Clustering assumes that all nodes are running with the same configuration file at roughly the same time. Check the logs for issues with the reloaded configuration file as well as the graph page to verify changes have been applied.
• Node name conflicts: Clustering assumes all nodes have unique names. Nodes with conflicting names are rejected and won’t join the cluster. Look at the clustering UI page for the list of current peers with their names, and check the logs for any reported name conflict events.
• Node stuck in terminating state: The node attempted to gracefully shut down and set its state to Terminating, but it has not completely gone away. Check the clustering page to view the state of the peers and verify that the terminating Alloy has been shut down.