Install Grafana Agent Flow

Wed, 11 Sep 2024 17:43:35 +0000

Install Grafana Agent Flow

You can install Grafana Agent Flow on Docker, Kubernetes, Linux, macOS, or Windows.

The following architectures are supported:

Linux: AMD64, ARM64
Windows: AMD64
macOS: AMD64 (Intel), ARM64 (Apple Silicon)
FreeBSD: AMD64

Note
Installing Grafana Agent Flow on other operating systems is possible, but isn’t recommended or supported.

Data collection

By default, Grafana Agent Flow sends anonymous usage information to Grafana Labs. Refer to data collection for more information about what data is collected and how you can opt-out.

Run Grafana Agent Flow

Wed, 11 Sep 2024 17:43:35 +0000

Run Grafana Agent Flow

Use the following pages to learn how to start, restart, and stop Grafana Agent Flow after it is installed. For installation instructions, refer to Install Grafana Agent Flow.

Grafana Agent Flow deployment topologies

Wed, 11 Sep 2024 17:43:35 +0000

Deploy Grafana Agent

Grafana Agent is a flexible, vendor-neutral telemetry collector. This flexibility means that Grafana Agent doesn’t enforce a specific deployment topology but can work in multiple scenarios.

This page lists common topologies used for deployments of Grafana Agent, when to consider using each topology, issues you may run into, and scaling considerations.

As a centralized collection service

Deploying Grafana Agent as a centralized service is recommended for collecting application telemetry. This topology allows you to use a smaller number of agents to coordinate service discovery, collection, and remote writing.

Using this topology requires deploying the Agent on separate infrastructure, and making sure that agents can discover and reach these applications over the network. The main predictor for the size of the agent is the number of active metrics series it is scraping; a rule of thumb is approximately 10 KB of memory for each series. We recommend you start looking towards horizontal scaling around the 1 million active series mark.

Using Kubernetes StatefulSets

Deploying Grafana Agent as a StatefulSet is the recommended option for metrics collection. The persistent pod identifiers make it possible to consistently match volumes with pods so that you can use them for the WAL directory.

You can also use a Kubernetes deployment in cases where persistent storage is not required, such as a traces-only pipeline.

Pros

Straightforward scaling using clustering or hashmod sharding
Minimizes the “noisy neighbor” effect
Easy to meta-monitor

Cons

Requires running on separate infrastructure

Use for

Scalable telemetry collection

Don’t use for

Host-level metrics and logs

As a host daemon

Deploying one Grafana Agent per machine is required for collecting machine-level metrics and logs, such as node_exporter hardware and network metrics or journald system logs.

Each Grafana Agent requires you to open an outgoing connection for each remote endpoint it’s shipping data to. This can lead to NAT port exhaustion on the egress infrastructure. Each egress IP can support up to (65535 - 1024 = 64511) outgoing connections on different ports. So, if all agents are shipping metrics and log data, an egress IP can support up to 32,255 agents.

Using Kubernetes DaemonSets

The simplest use case of the host daemon topology is a Kubernetes DaemonSet, and it is required for node-level observability (for example cAdvisor metrics) and collecting pod logs.

Pros

Doesn’t require running on separate infrastructure
Typically leads to smaller-sized agents
Lower network latency to instrumented applications

Cons

Requires planning a process for provisioning Grafana Agent on new machines, as well as keeping configuration up to date to avoid configuration drift
Not possible to scale agents independently when using Kubernetes DaemonSets
Scaling the topology can strain external APIs (like service discovery) and network infrastructure (like firewalls, proxy servers, and egress points)

Use for

Collecting machine-level metrics and logs (for example, node_exporter hardware metrics, Kubernetes pod logs)

Don’t use for

Scenarios where Grafana Agent grows so large it can become a noisy neighbor
Collecting an unpredictable amount of telemetry

As a container sidecar

Deploying Grafana Agent as a container sidecar is only recommended for short-lived applications or specialized agent deployments.

Using Kubernetes pod sidecars

In a Kubernetes environment, the sidecar model consists of deploying Grafana Agent as an extra container on the pod. The pod’s controller, network configuration, enabled capabilities, and available resources are shared between the actual application and the sidecar agent.

Pros

Doesn’t require running on separate infrastructure
Straightforward networking with partner applications

Cons

Doesn’t scale separately
Makes resource consumption harder to monitor and predict
Agents do not have a life cycle of their own, making it harder to reason about things like recovering from network outages

Use for

Serverless services
Job/batch applications that work with a push model
Air-gapped applications that can’t be otherwise reached over the network

Don’t use for

Long-lived applications
Scenarios where the agent size grows so large it can become a noisy neighbor

Processing different types of telemetry in different Grafana Agent instances

If the load on Grafana Agent is small, it is recommended to process all necessary telemetry signals in the same Grafana Agent process. For example, a single Grafana Agent can process all of the incoming metrics, logs, traces, and profiles.

However, if the load on the Grafana Agents is big, it may be beneficial to process different telemetry signals in different deployments of Grafana Agents.

This provides better stability due to the isolation between processes. For example, an overloaded Grafana Agent processing traces won’t impact an Grafana Agent processing metrics. Different types of signal collection require different methods for scaling:

“Pull” components such as prometheus.scrape and pyroscope.scrape are scaled using hashmod sharing or clustering.
“Push” components such as otelcol.receiver.otlp are scaled by placing a load balancer in front of them.

Traces

Scaling Grafana Agent instances for tracing is very similar to scaling OpenTelemetry Collector instances. This similarity is because most Grafana Agent Flow components used for tracing are based on components from the OTel Collector.

When to scale

To decide whether scaling is necessary, check metrics such as:

receiver_refused_spans_ratio_total from receivers such as otelcol.receiver.otlp.
processor_refused_spans_ratio_total from processors such as otelcol.processor.batch.
exporter_send_failed_spans_ratio_total from exporters such as otelcol.exporter.otlp and otelcol.exporter.loadbalancing.

Stateful and stateless components

In the context of tracing, a “stateful component” is a component that needs to aggregate certain spans to work correctly. A “stateless Grafana Agent” is a Grafana Agent which does not contain stateful components.

Scaling stateful Grafana Agents is more difficult, because spans must be forwarded to a specific Grafana Agent according to a span property such as trace ID or a service.name attribute. You can forward spans with otelcol.exporter.loadbalancing.

Examples of stateful components:

otelcol.processor.tail_sampling
otelcol.connector.spanmetrics
otelcol.connector.servicegraph

A “stateless component” does not need to aggregate specific spans to work correctly - it can work correctly even if it only has some of the spans of a trace.

A stateless Grafana Agent can be scaled without using otelcol.exporter.loadbalancing. For example, you could use an off-the-shelf load balancer to do a round-robin load balancing.

Examples of stateless components:

otelcol.processor.probabilistic_sampler
otelcol.processor.transform
otelcol.processor.attributes
otelcol.processor.span

Get started with Grafana Agent Flow on Grafana Labs

Install Grafana Agent Flow

Install Grafana Agent Flow

Data collection

Run Grafana Agent Flow

Run Grafana Agent Flow

Grafana Agent Flow deployment topologies

Deploy Grafana Agent

As a centralized collection service

Using Kubernetes StatefulSets

Pros

Cons

Use for

Don’t use for

As a host daemon

Using Kubernetes DaemonSets

Pros

Cons

Use for

Don’t use for

As a container sidecar

Using Kubernetes pod sidecars

Pros

Cons

Use for

Don’t use for

Processing different types of telemetry in different Grafana Agent instances

Traces

When to scale

Stateful and stateless components