Grafana Agent
The Grafana Agent collects observability data and sends it to Grafana Cloud. Once the agent is deployed to your hosts, it collects and sends Prometheus-style metrics and log data using a pared-down Prometheus collector.
NOTE: The initial version of official docs for the Grafana Agent have just been released. As those mature, information included here in the Grafana Cloud docs will likely become stale and will certainly be removed in favor of linking to the agent docs so that there is only one canonical source of information about the agent. These cloud docs will ultimately retain only the information specific to the cloud side of using the agent.
Installing the Grafana Agent
There are several ways to install and run the agent. This page first describes the preferred way where we install using a script for Debian- and Red Hat-based systems, starting with v0.9.1 of the agent. For other Linux distributions, or if you prefer to install the binary directly rather than as a package, see Installing the binary directly without our script for all other use cases.
For both methods, begin in the Grafana Cloud walkthrough, as shown in the quickstart.
- In the Grafana Cloud walkthrough, select an integration to install. Click Install Integration.
- Choose your OS, either Debian/Red Hat if you want to use our script to automate the install or Other Distribution for other use cases.
Installing the preferred way with our script for Debian- and Red Hat-based systems
This method exists to enable automating the installation for systems that use DEB or RPM packages, such as with Debian- or Red Hat-based systems. Our script will check and find which is appropriate for your system and then use your package manager for the Grafana Agent install. It will also set up the agent to be managed and run as a systemd service.
This method will run the agent using a newly-created user, grafana-agent
. If you are collecting logs, for example, you may need to give this user access to log-containing directories.
NOTE: The configuration for the agent is stored in
/etc/grafana-agent.yaml
. This file is overwritten each time the script is run, to add/remove any integration changes made in the Grafana Cloud UI. If you have custom configurations that you want to use, you can create and maintain your own custom configuration file by changing the environment variable in/etc/default/grafana-agent
to point to your custom configuration file instead of the default location.
- Copy and run the provided script.
- Start the service using the provided systemd command.
- Click Check Connection to confirm metrics are being received.
NOTE: This method also makes it easy to retrieve and review Grafana Agent logs using systemd.
To retrieve service logs for the agent, enter:
sudo journalctl -b -u grafana-agent.service
Installing the binary directly without our script for all other use cases
- Use the Kubernetes install script with preconfigured manifests
- Run the agent with Docker
- Run the agent directly on the host
- Modify the example Kubernetes manifests
- Build the agent from source
- Use our production Tanka configs
The full instructions for installing and running the Grafana Agent are updated frequently as new options are made available.
Note: Users who install and run the agent directly on the host may want the agent to automatically restart when its process is killed or the host is rebooted. To do this, create a systemd service to manage the agent.
Uninstalling the Grafana agent
How you uninstall depends on what Linux distribution you’re running and how you installed the agent.
The script method installs a system package called grafana-agent. This means that for:
- Debian-based systems you can remove the agent using
sudo apt remove grafana-agent
- Red Hat-based sytems you can remove the agent using
sudo dnf uninstall grafana-agent
If you installed using the “other distribution” step, the agent is not persisted anywhere. The binary is only downloaded to the directory you ran the install steps in. Stop any running process and then delete the binary to uninstall.
Benefits and trade-offs
The Grafana Agent is designed for easy installation and updates. It uses a subset of Prometheus code features to interact with hosted metrics, specifically:
- Service discovery
- Scraping
- Write ahead log (WAL)
- Remote writing
Along with this, the agent typically uses less memory than scraping metrics using Prometheus.
This optimization of Prometheus for remote write and resource reduction in the agent has resulted in some trade-offs:
- Metrics are first pulled by the agent, then pushed (forwarded) to a central storage location
- You can’t query the agent directly; you can only query metrics from the remote write storage
- Recording rules are not supported
- Alerts from the agent are not supported
Configuration of the agent
The agent is configured in a YAML file, named agent.yaml
by default. For the most part, you can follow the specific instructions that come with various integrations and edit as instructed. You don’t have to think much about the details. If you are an experienced user or just want to dive in to the configuration options and explore, see the Configuration Reference in the agent GitHub repo.
Agent IP address
By default, the agent listens on 0.0.0.0. This location provides an HTTP server using port 12345 that exposes only the agent’s internal metrics, metrics for any enabled integrations, and an API for the agent’s status. It also provides a gRPC server using port 9095 for agents to communicate with one another (if you have installed multiple instances of the agent across nodes, all of which are connected to the same central location, such as Grafana Cloud). This does not allow any other access to agent metrics.
If you would like to change the IP and port settings, you can do so in the above mentioned agent YAML file by changing or adding lines like these (using IP addresses and ports of your choosing in the place of the samples we show, making sure the ports used for each server are unique while the IP addresses may be unique or shared):
server:
http_listen_address: 127.0.0.1
http_listen_port: 12345
grpc_listen_address: 127.0.0.1
grpc_listen_port: 9095
NOTE: It is not possible to turn these servers off as they are needed for proper agent functioning.
Monitoring the monitoring system
We all want to know whether our monitoring and observability systems are working as intended. There are some unique attributes of using our agent that you must consider when setting this up.
Historically, while gathering metrics Prometheus pulls metrics from the source into its local storage. The Grafana Agent instead acts as a forwarder, meaning the agent that is installed on a node both scrapes (pulls) metrics and also forwards (pushes) those metrics to the remote monitoring and observability system. This is different from a remote monitoring system only pulling metrics from a set of defined targets.
NOTE: Sometimes it is necessary or just useful for there to be a local Prometheus instance that scrapes metrics that the agent is then configured to remote write to Grafana Cloud. In this configuration, Prometheus scrapes the metrics from the source and the agent is a forwarder from that Prometheus instance to the central time series database (TSDB) that is part of Grafana Cloud.
With a pull model, it is straightforward to determine whether a node is available using an up metric with a value of 1 when the target is reachable and 0 when it is not. With the agent’s push model, the up metric has a value of 1 when the agent is running and no value at all when it is not. This distinction is important when monitoring whether your monitoring system is running as expected. To learn more, please see Monitoring Grafana Agent
NOTE: While the agent itself can’t use recording rules and alerts, remote write systems like Grafana Cloud support server-side rules and alerts. This means that the reliability of Grafana Alerting is tied to the reliability of the remote system and that alerts will be delayed at least by the time it takes for samples to reach the remote system.
Integrations
An integration is the combination of:
- A built-in metrics exporter with sane agent configuration defaults for a given metrics source, like MySQL
- A set of curated dashboards and alerts on Grafana Cloud for a given metrics source, like MySQL
On the Grafana Cloud side, you can see all the integrations available in the Currently available Integrations page.
When the Grafana Cloud team builds Integration for users to install, code is also added to the agent, so that the agent and Grafana Cloud send and receive metrics for the data source. You can’t see which Integrations are supported for your agent version. This makes it important to update your agent version any time you want to use a new Integration you have not previously enabled.
On the host side, where the agent is installed, you may need to edit the agent configuration file to define which data is remote written to Grafana Cloud for use in querying and dashboards. The installation instructions for each individual Integration will tell you if you must do so.
Related Grafana Cloud resources
Intro to Prometheus and Grafana Cloud
Prometheus is taking over the monitoring world! In this webinar, we will start with a quick introduction to the open source project that’s the de facto standard for monitoring modern, cloud native systems.
How to set up and visualize synthetic monitoring at scale with Grafana Cloud
Learn how to use Kubernetes, Grafana Loki, and Grafana Cloud’s synthetic monitoring feature to set up your infrastructure's checks in this GrafanaCONline session.
Using Grafana Cloud to drive manufacturing plant efficiency
This GrafanaCONline session tells how Grafana helps a 75-year-old manufacturing company with product quality and equipment maintenance.