Troubleshoot Grafana Agent

Despite our best attempts and practices, sometimes things go wrong. This page provides some ideas to help you troubleshoot and find solutions for things we have been able to anticipate. If you encounter an issue not listed, your next step is to ask for help using the appropriate method, according to the support options for your account type.

Troubleshoot an agent installation

There are multiple ways to install the agent on your system. Look for the section that corresponds to the method you are using to find troubleshooting tips.

Install via the script method

If you are using the install script for Debian-based and Red Hat-based distributions, by following the instructions in the onboarding walkthrough, and an error appears when you try to check the connection, here are things to check.

Check that the agent is actually running on the target system using sudo systemctl status grafana-agent.service.
If the agent is not running, then check whether you can start it using sudo systemctl start grafana-agent.service. If the agent starts and you can check status and see it running, go back to Grafana Cloud and retest your connection. If all is well, instead of an error message a success message will appear.
If you cannot start the agent, check that the install script was successfully downloaded. If it was not downloaded, check that curl is installed on the system and attempt the download again.
If the download works but the script does not complete properly, check that the user account being used on the target system has sudo privileges and is able to run scripts and install software.

Download and install the binary

If you are trying to install the agent as a single binary, by following the instructions in the onboarding walkthrough, and encounter an error, here are some things to try.

When attemping to download the file you run into command not found: curl - Ensure that you are running commands on a Linux terminal and that curl is installed on your operating system.
When attempting to extract the binary you run into command not found: unzip- Ensure that unzip is installed on your operating system.
When attemping to make the file executable you run into cannot find or open grafana-agent-linux-amd64.zip - Ensure that you have followed the Download file step correctly and that no errors have been returned in the terminal window.
When attemping to make the file executable you run into no such file or directory - Ensure that you have followed the Download file and Extract the binary steps correctly and that no errors have been returned in the terminal window.

Run the agent

If the agent is installed correctly, but is not running, here are some tips specific to error messages you may encounter.

When attemping to run the agent you run into no such file or directory: grafana-agent-linux-amd64 - Ensure that you have followed the Download file step correctly and that no errors have been returned in the terminal window.
When attempting to run the agent you run into permission denied: grafana-agent-linux-amd64- Ensure that you have followed the Make sure it’s executable step correctly and that no errors have been returned in the terminal window.
When attempting to run the agent you run into error loading config file agent-config.yaml - Ensure that you have followed the Generate config file step correctly and that no errors have been returned in the terminal window. Additionally, ensure that the command to Run the agent is being run in the same directory as the command used to Generate the config file.
When attempting to run the agent you run into error creating WAL: create dir: mkdir /tmp/grafana-agent-wal: permission denied - Stop the agent from running by pressing with Ctrl-C in the terminal window. Ensure that your user has the correct permissions to create directories inside of /tmp. You may need to contact your system administrator to get access to this directory. Then, follow the Run the agent step again.
When attempting to run the agent you run into server returned HTTP status 404 Not Found: 404 page not found - The Generate config file step may not have been copied successfully. Stop the agent from running by pressing Ctrl-C in the terminal window and then repeat the Generate config file step, followed by Run the agent, again.
When attempting to run the agent you run into listen tcp :12345: bind: permission deniedl - Stop the agent from running by pressing with Ctrl-C in the terminal window. You may need to contact your system administrator to get access to run processes with permission to bind on port 12345. Then, follow the Run the agent step again.
When attempting to run the agent you run into server returned HTTP status 401 Unauthorized: {\"status\": \"error\", \"error\": \"Invalid API Key\"} - The Generate config file step may not have been copied successfully. Stop the agent from running by pressing Ctrl-C in the terminal window and then repeat the Generate config file step, followed by Run the agent, again. If you still run into this error, please contact Grafana support for more help.
When attempting to run the agent you run into server returned HTTP status 429 Too Many Requests: ingestion rate limit exceeded - You may have exceeded your current active series limits. Please contact Grafana support for more help.

Troubleshoot Kubernetes script installation

The following is a list of common errors that may occur when trying to install the the agent using the Kubernetes instsall script, by following the instructions in the onboarding walkthrough.

Download and install the Kubernetes install script

Command not found: kubectl - Ensure that you are running commands on a Linux terminal with bash and that kubectl installed. Once you’ve verified that kubectl is installed, ensure that you are currently using the proper Kubernetes context to install the Grafana Agent.

No errors but Check Connection fails

Run kubectl -ndefault get po -lname=grafana-agent in a terminal window. You should see several pods running, and a table similar to below:

Name	Ready	Status	Restarts	Age
grafana-agent-25hd5	1/1	Running	0	25h
grafana-agent-2tjd2	1/1	Running	1	26h
grafana-agent-4wtzn	1/1	Running	0	25h
grafana-agent-5nk2p	1/1	Running	0	11h
grafana-agent-8r4mf	1/1	Running	0	27h

The values under Name will be different, but the values under Status should all be Running. If any of the statuses are Pending, ensure that you have correctly run the Generate config file step and that no errors were returned. It may take up to a few minutes for pods to change from a status of Pending to Running after running this step.

Agent log errors

If the agent is running but Check Connection is still failing, try retrieving logs for your agent by running kubectl -ndefault logs daemonset/grafana-agent | grep level=err in a terminal window to retrieve all error-level agent logs. The following is a list of common log errors:

server returned HTTP status 404 Not Found: 404 page not found - The Generate config file step may not have been copied successfully. Stop the agent from running by pressing Ctrl-C in the terminal window and then repeat the Generate config file step, followed by Run the agent, again.
server returned HTTP status 401 Unauthorized: {\"status\": \"error\", \"error\": \"Invalid API Key\"} - The Generate config file step may not have been copied successfully. Stop the agent from running by pressing Ctrl-C in the terminal window and then repeat the Generate config file step, followed by Run the agent, again. If you still run into this error, please contact Grafana support for more help.
server returned HTTP status 429 Too Many Requests: ingestion rate limit exceeded - You may have exceeded your current active series limits. Please contact Grafana support for more help.

Other errors

If you are still running into other error codes not listed, experiencing a Check Connection failure not described, or not seeing any errors in agent logs but still can’t connect, please contact Grafana support for more help and provide any specific error messages you are encountering.

An established connection was aborted by the software in your host machine

If you encounter an error in the Windows Event Viewer stating wsasend: An established connection was aborted by the software in your host machine, there might be a timeout. To fix the error, try increasing the scrape_timeout value. You can either change this value globally or, if you suspect that a specific integration such as the windows_exporter is timing out, you can configure this setting in the specific integration’s configuration section.