About Grafana Agent
Grafana Agent is a telemetry collector for sending metrics, logs, and trace data to Grafana Cloud. Grafana Agent uses a subset of Prometheus code features to interact with hosted metrics, specifically:
- Service discovery
- Scraping
- Write ahead log (WAL)
- Remote writing
Agent typically uses less memory than scraping metrics using Prometheus.
This optimization of Prometheus for remote write and resource reduction in Agent has resulted in some trade-offs:
- Metrics are first pulled by Agent, then pushed (forwarded) to a central storage location.
- You can’t query the agent directly; you can only query metrics from the remote write storage.
- Recording rules are not supported.
- Alerts from Agent are not supported.
Configuration of Grafana Agent
Grafana Agent is configured in a YAML file, named agent.yaml by default. For the most part, you can follow the specific instructions that come with various integrations and edit as instructed. You don’t have to think much about the details. If you are an experienced user or just want to dive in to the configuration options and explore, see Configure Grafana Agent.
Agent IP address
By default, the agent listens on 0.0.0.0. This location provides an HTTP server using port 12345 that exposes only the agent’s internal metrics, metrics for any enabled integrations, and an API for the agent’s status. It also provides a gRPC server using port 9095 for agents to communicate with one another (if you have installed multiple instances of the agent across nodes, all of which are connected to the same central location, such as Grafana Cloud). This does not allow any other access to agent metrics.
If you would like to change the IP and port settings, you can do using the following server command-line flags -server.http.address and -server.grpc.address.
Note: It is not possible to turn these servers off as they are needed for proper agent functioning.
Integrations
An integration is the combination of:
- A built-in metrics exporter with sane agent configuration defaults for a given metrics source, like MySQL
- A set of curated dashboards and alerts on Grafana Cloud for a given metrics source, like MySQL
On the Grafana Cloud side, you can see all the integrations available in the Integrations reference.
When the Grafana Cloud team builds an integration for users to install, code is also added to Grafana Agent, so that Grafana Agent and Grafana Cloud send and receive metrics for the data source. You can’t see which integrations are supported for your Agent version. This makes it important to update your Agent version any time you want to use a new integration you have not previously enabled.
On the host side, where Agent is installed, you may need to edit the Agent configuration file to define which data is remote written to Grafana Cloud for use in querying and dashboards. The installation instructions for each individual integration will tell you if you must do so.
Monitor the monitoring system
We all want to know whether our monitoring and observability systems are working as intended. There are some unique attributes of using Grafana Agent that you must consider when setting this up.
Historically, while gathering metrics Prometheus pulls metrics from the source into its local storage. Grafana Agent instead acts as a forwarder, meaning the Agent that is installed on a node both scrapes (pulls) metrics and also forwards (pushes) those metrics to the remote monitoring and observability system. This is different from a remote monitoring system that pulls metrics only from a set of defined targets.
NOTE: Sometimes it is necessary or just useful to have a local Prometheus instance that scrapes metrics that Grafana Agent then remote-writes to Grafana Cloud. In this configuration, Prometheus scrapes the metrics from the source and Grafana Agent is a forwarder from that Prometheus instance to the central time series database (TSDB) that is part of Grafana Cloud.
With a pull model, it is straightforward to determine whether a node is available using an up metric with a value of 1 when the target is reachable and 0 when it is not. With Agent’s push model, the up metric has a value of 1 when Agent is running and no value at all when it is not. This distinction is important when monitoring whether your monitoring system is running as expected. To learn more, see Monitor Grafana Agent
NOTE: While Grafana Agent itself can’t use recording rules and alerts, remote write systems like Grafana Cloud support server-side rules and alerts. This means that the reliability of Grafana Alerting is tied to the reliability of the remote system and that alerts are delayed at least by the time it takes for samples to reach the remote system.
