Grafana Agent

The Grafana Agent collects observability data and sends it to Grafana Cloud. Once the agent is deployed to your hosts, it collects and sends Prometheus-style metrics and log data using a pared-down Prometheus collector.

NOTE: The initial version of official docs for the Grafana Agent have just been released. As those mature, information included here in the Grafana Cloud docs will likely become stale and will certainly be removed in favor of linking to the agent docs so that there is only one canonical source of information about the agent. These cloud docs will ultimately retain only the information specific to the cloud side of using the agent.

Installing the Grafana Agent

There are several ways to install and run the agent. This page first describes the preferred way where we install using a script for Debian- and Red Hat-based systems, starting with v0.9.1 of the agent. For other Linux distributions, or if you prefer to install the binary directly rather than as a package, see Installing the binary directly without our script for all other use cases.

For both methods, begin in the Grafana Cloud walkthrough, as shown in the quickstart.

1. In the Grafana Cloud walkthrough, select an integration to install. Click Next step.
2. Choose your OS, either Debian/Red Hat if you want to use our script to automate the install or Other Distribution for other use cases.

Installing the preferred way with our script for Debian- and Red Hat-based systems

This method exists to enable automating the installation for systems that use DEB or RPM packages, such as with Debian- or Red Hat-based systems. Our script will check and find which is appropriate for your system and then use your package manager for the Grafana Agent install. It will also set up the agent to be managed and run as a systemd service.

This method will run the agent using a newly-created user, grafana-agent. If you are collecting logs, for example, you may need to give this user access to log-containing directories.

NOTE: The configuration for the agent is stored in /etc/grafana-agent.yaml. This file is overwritten each time the script is run, to add/remove any integration changes made in the Grafana Cloud UI. If you have custom configurations that you want to use, you can create and maintain your own custom configuration file by changing the environment variable in /etc/default/grafana-agent to point to your custom configuration file instead of the default location.

1. Copy and run the provided script.
2. Start the service using the provided systemd command.
3. Click Check Connection to confirm metrics are being received.

NOTE: This method also makes it easy to retrieve and review Grafana Agent logs using systemd.

To retrieve service logs for the agent, enter:

sudo journalctl -b -u grafana-agent.service

Installing the binary directly without our script for all other use cases

• Use the Kubernetes install script with preconfigured manifests
• Run the agent with Docker
• Run the agent directly on the host
• Modify the example Kubernetes manifests
• Build the agent from source
• Use our production Tanka configs

The full instructions for installing and running the Grafana Agent are updated frequently as new options are made available.

Note: Users who install and run the agent directly on the host may want the agent to automatically restart when its process is killed or the host is rebooted. To do this, create a systemd service to manage the agent.

Uninstalling the Grafana agent

How you uninstall depends on what Linux distribution you’re running and how you installed the agent.

The script method installs a system package called grafana-agent. This means that for:

• Debian-based systems you can remove the agent using sudo apt remove grafana-agent
• Red Hat-based sytems you can remove the agent using sudo dnf uninstall grafana-agent

If you installed using the “other distribution” step, the agent is not persisted anywhere. The binary is only downloaded to the directory you ran the install steps in. Stop any running process and then delete the binary to uninstall.

Benefits and trade-offs

The Grafana Agent is designed for easy installation and updates. It uses a subset of Prometheus code features to interact with hosted metrics, specifically:

• Service discovery
• Scraping
• Write ahead log (WAL)
• Remote writing

Along with this, the agent typically uses less memory than scraping metrics using Prometheus.

This optimization of Prometheus for remote write and resource reduction in the agent has resulted in some trade-offs:

• Metrics are first pulled by the agent, then pushed (forwarded) to a central storage location
• You can’t query the agent directly; you can only query metrics from the remote write storage
• Recording rules are not supported
• Alerts from the agent are not supported

Configuration of the agent

The agent is configured in a YAML file, named agent.yaml by default. For the most part, you can follow the specific instructions that come with various integrations and edit as instructed. You don’t have to think much about the details. If you are an experienced user or just want to dive in to the configuration options and explore, see the Configuration Reference in the agent GitHub repo.

Agent IP address

By default, the agent listens on 0.0.0.0. This location provides an HTTP server using port 12345 that exposes only the agent’s internal metrics, metrics for any enabled integrations, and an API for the agent’s status. It also provides a gRPC server using port 9095 for agents to communicate with one another (if you have installed multiple instances of the agent across nodes, all of which are connected to the same central location, such as Grafana Cloud). This does not allow any other access to agent metrics.

If you would like to change the IP and port settings, you can do so in the above mentioned agent YAML file by changing or adding lines like these (using IP addresses and ports of your choosing in the place of the samples we show, making sure the ports used for each server are unique while the IP addresses may be unique or shared):

server:
  http_listen_address: 127.0.0.1
  http_listen_port: 12345
  grpc_listen_address: 127.0.0.1
  grpc_listen_port: 9095

NOTE: It is not possible to turn these servers off as they are needed for proper agent functioning.

Monitoring the monitoring system

We all want to know whether our monitoring and observability systems are working as intended. There are some unique attributes of using our agent that you must consider when setting this up.

Historically, while gathering metrics Prometheus pulls metrics from the source into its local storage. The Grafana Agent instead acts as a forwarder, meaning the agent that is installed on a node both scrapes (pulls) metrics and also forwards (pushes) those metrics to the remote monitoring and observability system. This is different from a remote monitoring system only pulling metrics from a set of defined targets.

NOTE: Sometimes it is necessary or just useful for there to be a local Prometheus instance that scrapes metrics that the agent is then configured to remote write to Grafana Cloud. In this configuration, Prometheus scrapes the metrics from the source and the agent is a forwarder from that Prometheus instance to the central time series database (TSDB) that is part of Grafana Cloud.

With a pull model, it is straightforward to determine whether a node is available using an up metric with a value of 1 when the target is reachable and 0 when it is not. With the agent’s push model, the up metric has a value of 1 when the agent is running and no value at all when it is not. This distinction is important when monitoring whether your monitoring system is running as expected. To learn more, please see Monitoring Grafana Agent

NOTE: While the agent itself can’t use recording rules and alerts, remote write systems like Grafana Cloud support server-side rules and alerts. This means that the reliability of Grafana Cloud Alerts is tied to the reliability of the remote system and that alerts will be delayed at least by the time it takes for samples to reach the remote system.

Integrations

An integration is the combination of:

• A built-in metrics exporter with sane agent configuration defaults for a given metrics source, like MySQL
• A set of curated dashboards and alerts on Grafana Cloud for a given metrics source, like MySQL

On the Grafana Cloud side, you can see all the integrations available in the Currently available Integrations page.

When the Grafana Cloud team builds Integration for users to install, code is also added to the agent, so that the agent and Grafana Cloud send and receive metrics for the data source. You can’t see which Integrations are supported for your agent version. This makes it important to update your agent version any time you want to use a new Integration you have not previously enabled.

On the host side, where the agent is installed, you may need to edit the agent configuration file to define which data is remote written to Grafana Cloud for use in querying and dashboards. The installation instructions for each individual Integration will tell you if you must do so.