Grafana Cloud AgentTroubleshooting Grafana Cloud Agent

Troubleshooting Grafana Cloud Agent

Troubleshooting binary installation

The following is a list of common errors that may occur when trying to install the agent as a single binary, by following the instructions in the integrations walkthrough.

Downloading and installing the binary

  • When attemping to download the file you run into command not found: curl - Ensure that you are running commands on a Linux terminal and that curl is installed on your operating system.

  • When attempting to extract the binary you run into command not found: unzip- Ensure that unzip is installed on your operating system.

  • When attemping to make the file executable you run into cannot find or open agent-linux-amd64.zip - Ensure that you have followed the Download file step correctly and that no errors have been returned in the terminal window.

  • When attemping to make the file executable you run into no such file or directory - Ensure that you have followed the Download file and Extract the binary steps correctly and that no errors have been returned in the terminal window.

Running the agent

  • When attemping to run the agent you run into no such file or directory: agent-linux-amd64 - Ensure that you have followed the Download file step correctly and that no errors have been returned in the terminal window.

  • When attempting to run the agent you run into permission denied: agent-linux-amd64- Ensure that you have followed the Make sure it’s executable step correctly and that no errors have been returned in the terminal window.

  • When attempting to run the agent you run into error loading config file agent-config.yaml - Ensure that you have followed the Generate config file step correctly and that no errors have been returned in the terminal window. Additionally, ensure that the command to Run the agent is being run in the same directory as the command used to Generate the config file.

  • When attempting to run the agent you run into error creating WAL: create dir: mkdir /tmp/grafana-agent-wal: permission denied - Stop the agent from running by pressing with Ctrl-C in the terminal window. Ensure that your user has the correct permissions to create directories inside of /tmp. You may need to contact your system administrator to get access to this directory. Then, follow the Run the agent step again.

  • When attempting to run the agent you run into server returned HTTP status 404 Not Found: 404 page not found - The Generate config file step may not have been copied successfully. Stop the agent from running by pressing Ctrl-C in the terminal window and then repeat the Generate config file step, followed by Run the agent, again.

  • When attempting to run the agent you run into listen tcp :12345: bind: permission deniedl - Stop the agent from running by pressing with Ctrl-C in the terminal window. You may need to contact your system administrator to get access to run processes with permission to bind on port 12345. Then, follow the Run the agent step again.

  • When attempting to run the agent you run into server returned HTTP status 401 Unauthorized: {\"status\": \"error\", \"error\": \"Invalid API Key\"} - The Generate config file step may not have been copied successfully. Stop the agent from running by pressing Ctrl-C in the terminal window and then repeat the Generate config file step, followed by Run the agent, again. If you still run into this error, please contact Grafana support for more help.

  • When attempting to run the agent you run into server returned HTTP status 429 Too Many Requests: ingestion rate limit exceeded - You may have exceeded your current active series limits. Please contact Grafana support for more help.

Troubleshooting Kubernetes script installation

The following is a list of common errors that may occur when trying to install the the agent using the Kubernetes instsall script, by following the instructions in the integrations walkthrough.

Downloading and installing the Kubernetes install script

Command not found: kubectl - Ensure that you are running commands on a Linux terminal with bash and that kubectl installed. Once you’ve verified that kubectl is installed, ensure that you are currently using the proper Kubernetes context to install the Grafana Cloud Agent.

No errors but Check Connection fails

Run kubectl -ndefault get po -lname=grafana-agent in a terminal window. You should see several pods running, and a table similar to below:

Name Ready Status Restarts Age
grafana-agent-25hd5 1/1 Running 0 25h
grafana-agent-2tjd2 1/1 Running 1 26h
grafana-agent-4wtzn 1/1 Running 0 25h
grafana-agent-5nk2p 1/1 Running 0 11h
grafana-agent-8r4mf 1/1 Running 0 27h

The values under Name will be different, but the values under Status should all be Running. If any of the statuses are Pending, ensure that you have correctly run the Generate config file step and that no errors were returned. It may take up to a few minutes for pods to change from a status of Pending to Running after running this step.

Agent log errors

If the agent is running but Check Connection is still failing, try retrieving logs for your agent by running kubectl -ndefault logs daemonset/grafana-agent | grep level=err in a terminal window to retrieve all error-level agent logs. The following is a list of common log errors:

  • server returned HTTP status 404 Not Found: 404 page not found - The Generate config file step may not have been copied successfully. Stop the agent from running by pressing Ctrl-C in the terminal window and then repeat the Generate config file step, followed by Run the agent, again.

  • server returned HTTP status 401 Unauthorized: {\"status\": \"error\", \"error\": \"Invalid API Key\"} - The Generate config file step may not have been copied successfully. Stop the agent from running by pressing Ctrl-C in the terminal window and then repeat the Generate config file step, followed by Run the agent, again. If you still run into this error, please contact Grafana support for more help.

  • server returned HTTP status 429 Too Many Requests: ingestion rate limit exceeded - You may have exceeded your current active series limits. Please contact Grafana support for more help.

Other errors

If you are still running into other error codes not listed, experiencing a Check Connection failure not described, or not seeing any errors in agent logs but still can’t connect, please contact Grafana support for more help and provide any specific error messages you are encountering.