Troubleshoot Grafana Agent
Despite our best attempts and practices, sometimes things go wrong. This page provides some ideas to help you troubleshoot and find solutions for things we have been able to anticipate. If you encounter an issue not listed, your next step is to ask for help using the appropriate method, according to the support options for your account type.
Troubleshoot an agent installation
There are multiple ways to install the agent on your system. Look for the section that corresponds to the method you are using to find troubleshooting tips.
Install via the script method
If you are using the install script for Debian-based and Red Hat-based distributions, by following the instructions in the onboarding walkthrough, and an error appears when you try to check the connection, here are things to check.
Check that the agent is actually running on the target system using
sudo systemctl status grafana-agent.service
.If the agent is not running, then check whether you can start it using
sudo systemctl start grafana-agent.service
. If the agent starts and you can check status and see it running, go back to Grafana Cloud and retest your connection. If all is well, instead of an error message a success message will appear.If you cannot start the agent, check that the install script was successfully downloaded. If it was not downloaded, check that
curl
is installed on the system and attempt the download again.If the download works but the script does not complete properly, check that the user account being used on the target system has sudo privileges and is able to run scripts and install software.
Download and install the binary
If you are trying to install the agent as a single binary, by following the instructions in the onboarding walkthrough, and encounter an error, here are some things to try.
When attemping to download the file you run into
command not found: curl
- Ensure that you are running commands on a Linux terminal and thatcurl
is installed on your operating system.When attempting to extract the binary you run into
command not found: unzip
- Ensure thatunzip
is installed on your operating system.When attemping to make the file executable you run into
cannot find or open grafana-agent-linux-amd64.zip
- Ensure that you have followed the Download file step correctly and that no errors have been returned in the terminal window.When attemping to make the file executable you run into
no such file or directory
- Ensure that you have followed the Download file and Extract the binary steps correctly and that no errors have been returned in the terminal window.
Run the agent
If the agent is installed correctly, but is not running, here are some tips specific to error messages you may encounter.
When attemping to run the agent you run into
no such file or directory: grafana-agent-linux-amd64
- Ensure that you have followed the Download file step correctly and that no errors have been returned in the terminal window.When attempting to run the agent you run into
permission denied: grafana-agent-linux-amd64
- Ensure that you have followed the Make sure it’s executable step correctly and that no errors have been returned in the terminal window.When attempting to run the agent you run into
error loading config file agent-config.yaml
- Ensure that you have followed the Generate config file step correctly and that no errors have been returned in the terminal window. Additionally, ensure that the command to Run the agent is being run in the same directory as the command used to Generate the config file.When attempting to run the agent you run into
error creating WAL: create dir: mkdir /tmp/grafana-agent-wal: permission denied
- Stop the agent from running by pressing with Ctrl-C in the terminal window. Ensure that your user has the correct permissions to create directories inside of /tmp. You may need to contact your system administrator to get access to this directory. Then, follow the Run the agent step again.When attempting to run the agent you run into
server returned HTTP status 404 Not Found: 404 page not found
- The Generate config file step may not have been copied successfully. Stop the agent from running by pressing Ctrl-C in the terminal window and then repeat the Generate config file step, followed by Run the agent, again.When attempting to run the agent you run into
listen tcp :12345: bind: permission deniedl
- Stop the agent from running by pressing with Ctrl-C in the terminal window. You may need to contact your system administrator to get access to run processes with permission to bind on port 12345. Then, follow the Run the agent step again.When attempting to run the agent you run into
server returned HTTP status 401 Unauthorized: {\"status\": \"error\", \"error\": \"Invalid API Key\"}
- The Generate config file step may not have been copied successfully. Stop the agent from running by pressing Ctrl-C in the terminal window and then repeat the Generate config file step, followed by Run the agent, again. If you still run into this error, please contact Grafana support for more help.When attempting to run the agent you run into
server returned HTTP status 429 Too Many Requests: ingestion rate limit exceeded
- You may have exceeded your current active series limits. Please contact Grafana support for more help.
Troubleshoot Kubernetes script installation
The following is a list of common errors that may occur when trying to install the the agent using the Kubernetes instsall script, by following the instructions in the onboarding walkthrough.
Download and install the Kubernetes install script
Command not found: kubectl
-
Ensure that you are running commands on a Linux terminal with bash and that kubectl
installed. Once you’ve verified that kubectl
is installed, ensure that you are currently using the proper Kubernetes context to install the Grafana Agent.
No errors but Check Connection fails
Run kubectl -ndefault get po -lname=grafana-agent
in a terminal window. You should see several pods running, and a table similar to below:
Name | Ready | Status | Restarts | Age |
---|---|---|---|---|
grafana-agent-25hd5 | 1/1 | Running | 0 | 25h |
grafana-agent-2tjd2 | 1/1 | Running | 1 | 26h |
grafana-agent-4wtzn | 1/1 | Running | 0 | 25h |
grafana-agent-5nk2p | 1/1 | Running | 0 | 11h |
grafana-agent-8r4mf | 1/1 | Running | 0 | 27h |
The values under Name will be different, but the values under Status should all be Running
. If any of the statuses are Pending
, ensure that you have correctly run the Generate config file step and that no errors were returned. It may take up to a few minutes for pods to change from a status of Pending
to Running
after running this step.
Agent log errors
If the agent is running but Check Connection is still failing, try retrieving logs for your agent by running kubectl -ndefault logs daemonset/grafana-agent | grep level=err
in a terminal window to retrieve all error-level agent logs. The following is a list of common log errors:
server returned HTTP status 404 Not Found: 404 page not found
- The Generate config file step may not have been copied successfully. Stop the agent from running by pressing Ctrl-C in the terminal window and then repeat the Generate config file step, followed by Run the agent, again.server returned HTTP status 401 Unauthorized: {\"status\": \"error\", \"error\": \"Invalid API Key\"}
- The Generate config file step may not have been copied successfully. Stop the agent from running by pressing Ctrl-C in the terminal window and then repeat the Generate config file step, followed by Run the agent, again. If you still run into this error, please contact Grafana support for more help.server returned HTTP status 429 Too Many Requests: ingestion rate limit exceeded
- You may have exceeded your current active series limits. Please contact Grafana support for more help.
Other errors
If you are still running into other error codes not listed, experiencing a Check Connection
failure not described, or not seeing any errors in agent logs but still can’t connect, please contact Grafana support for more help and provide any specific error messages you are encountering.
An established connection was aborted by the software in your host machine
If you encounter an error in the Windows Event Viewer stating wsasend: An established connection was aborted by the software in your host machine
, there might be a timeout. To fix the error, try increasing the scrape_timeout
value. You can either change this value globally or, if you suspect that a specific integration such as the windows_exporter
is timing out, you can configure this setting in the specific integration’s configuration section.