Tasks with Grafana Agent Flow on Grafana Labs

Configure Grafana Agent Flow

Wed, 29 Jan 2025 11:00:55 +0000

Configure Grafana Agent Flow

You can configure Grafana Agent Flow after it is installed. The default River configuration file for Grafana Agent Flow is located at:

Linux: /etc/grafana-agent-flow.river
macOS: $(brew --prefix)/etc/grafana-agent-flow/config.river
Windows: C:\Program Files\Grafana Agent Flow\config.river

This section includes information that helps you configure Grafana Agent Flow.

Migrate to Grafana Agent Flow

Wed, 29 Jan 2025 11:00:55 +0000

How to migrate to Grafana Agent Flow

This section details how to migrate to Grafana Agent Flow from other common solutions.

Monitor Grafana Agent Flow

Wed, 29 Jan 2025 11:00:55 +0000

How to monitor Grafana Agent Flow

This section details various ways to monitor and debug Grafana Agent Flow.

Collect and forward Prometheus metrics

Wed, 29 Jan 2025 11:00:55 +0000

Collect and forward Prometheus metrics

You can configure Grafana Agent Flow to collect Prometheus metrics and forward them to any Prometheus-compatible database.

This topic describes how to:

Configure metrics delivery.
Collect metrics from Kubernetes Pods.

Components used in this topic

Before you begin

Ensure that you have basic familiarity with instrumenting applications with Prometheus.
Have a set of Prometheus exports or applications exposing Prometheus metrics that you want to collect metrics from.
Identify where you will write collected metrics. Metrics can be written to Prometheus or Prometheus-compatible endpoints such as Grafana Mimir, Grafana Cloud, or Grafana Enterprise Metrics.
Be familiar with the concept of Components in Grafana Agent Flow.

Configure metrics delivery

Before components can collect Prometheus metrics, you must have a component responsible for writing those metrics somewhere.

The prometheus.remote_write component is responsible for delivering Prometheus metrics to one or Prometheus-compatible endpoints. After a prometheus.remote_write component is defined, you can use other Grafana Agent Flow components to forward metrics to it.

To configure a prometheus.remote_write component for metrics delivery, complete the following steps:

Add the following prometheus.remote_write component to your configuration file.
Alloy
```
prometheus.remote_write "<LABEL>" {
  endpoint {
    url = "<PROMETHEUS_URL>"
  }
}
```
Replace the following:
- <LABEL>: The label for the component, such as default. The label you use must be unique across all prometheus.remote_write components in the same configuration file.
- <PROMETHEUS_URL> The full URL of the Prometheus-compatible endpoint where metrics will be sent, such as https://prometheus-us-central1.grafana.net/api/prom/push.
If your endpoint requires basic authentication, paste the following inside the endpoint block.
Alloy
```
basic_auth {
  username = "<USERNAME>"
  password = "<PASSWORD>"
}
```
Replace the following:
- <USERNAME>: The basic authentication username.
- <PASSWORD>: The basic authentication password or API key.
If you have more than one endpoint to write metrics to, repeat the endpoint block for additional endpoints.

The following example demonstrates configuring prometheus.remote_write with multiple endpoints and mixed usage of basic authentication, and a prometheus.scrape component which forwards metrics to it.

prometheus.remote_write "default" {
  endpoint {
    url = "http://localhost:9090/api/prom/push"
  }

  endpoint {
    url = "https://prometheus-us-central1.grafana.net/api/prom/push"

    // Get basic authentication based on environment variables.
    basic_auth {
      username = env("<REMOTE_WRITE_USERNAME>")
      password = env("<REMOTE_WRITE_PASSWORD>")
    }
  }
}

prometheus.scrape "example" {
  // Collect metrics from the default listen address.
  targets = [{
    __address__ = "127.0.0.1:12345",
  }]

  forward_to = [prometheus.remote_write.default.receiver]
}

For more information on configuring metrics delivery, refer to prometheus.remote_write.

Collect metrics from Kubernetes Pods

Grafana Agent Flow can be configured to collect metrics from Kubernetes Pods by:

Discovering Kubernetes Pods to collect metrics from.
Collecting metrics from those discovered Pods.

To collect metrics from Kubernetes Pods, complete the following steps:

Follow Configure metrics delivery to ensure collected metrics can be written somewhere.
Discover Kubernetes Pods:
1. Add the following discovery.kubernetes component to your configuration file to discover every Pod in the cluster across all Namespaces.
  Alloy
```
discovery.kubernetes "<DISCOVERY_LABEL>" {
  role = "pod"
}
```
  Replace the following
  - <DISCOVERY_LABEL>: The label for the component, such as pods. The label you use must be unique across all discovery.kubernetes components in the same configuration file.
  This generates one Prometheus target for every exposed port on every discovered Pod.
2. To limit the Namespaces that Pods are discovered in, add the following block inside the discovery.kubernetes component.
  Alloy
```
namespaces {
  own_namespace = true
  names         = [<NAMESPACE_NAMES>]
}
```
  Replace the following:
  - <NAMESPACE_NAMES>: A comma-delimited list of strings representing Namespaces to search. Each string must be wrapped in double quotes. For example, "default","kube-system".
  If you don’t want to search for Pods in the Namespace Grafana Agent Flow is running in, set own_namespace to false.
3. To use a field selector to limit the number of discovered Pods, add the following block inside the discovery.kubernetes component.
  Alloy
```
selectors {
  role  = "pod"
  field = "<FIELD_SELECTOR>"
}
```
  Replace the following:
  - <FIELD_SELECTOR>: The Kubernetes field selector to use, such as metadata.name=my-service. For more information on field selectors, refer to the Kubernetes documentation on Field Selectors.
  Create additional selectors blocks for each field selector you want to apply.
4. To use a label selector to limit the number of discovered Pods, add the following block inside the discovery.kubernetes component.
  Alloy
```
selectors {
  role  = "pod"
  label = "LABEL_SELECTOR"
}
```
  Replace the following:
  - <LABEL_SELECTOR>: The Kubernetes label selector, such as environment in (production, qa). For more information on label selectors, refer to the Kubernetes documentation on Labels and Selectors.
  Create additional selectors blocks for each label selector you want to apply.
Collect metrics from discovered Pods:
1. Add the following prometheus.scrape component to your configuration file.
  Alloy
```
prometheus.scrape "<SCRAPE_LABEL>" {
  targets    = discovery.kubernetes.<DISCOVERY_LABEL>.targets
  forward_to = [prometheus.remote_write.<REMOTE_WRITE_LABEL>.receiver]
}
```
  Replace the following:
  - <SCRAPE_LABEL>: The label for the component, such as pods. The label you use must be unique across all prometheus.scrape components in the same configuration file.
  - <DISCOVERY_LABEL>: The label for the discovery.kubernetes component.
  - <REMOTE_WRITE_LABEL>: The label for your existing prometheus.remote_write component.

The following example demonstrates configuring Grafana Agent Flow to collect metrics from running production Kubernetes Pods in the default Namespace.

discovery.kubernetes "pods" {
  role = "pod"

  namespaces {
    own_namespace = false

    names = ["default"]
  }

  selectors {
    role  = "pod"
    label = "environment in (production)"
  }
}

prometheus.scrape "pods" {
  targets    = discovery.kubernetes.pods.targets
  forward_to = [prometheus.remote_write.default.receiver]
}

prometheus.remote_write "default" {
  endpoint {
    url = "http://localhost:9090/api/prom/push"
  }
}

For more information on configuring Kubernetes service delivery and collecting metrics, refer to discovery.kubernetes and prometheus.scrape.

Collect metrics from Kubernetes Services

You can configure Grafana Agent Flow to collect metrics from Kubernetes Services by:

Discovering Kubernetes Services to collect metrics from.
Collecting metrics from those discovered Services.

To collect metrics from Kubernetes Services, complete the following steps.

Follow Configure metrics delivery to ensure collected metrics can be written somewhere.
Discover Kubernetes Services:
1. Add the following discovery.kubernetes component to your configuration file to discover every Services in the cluster across all Namespaces.
  Alloy
```
discovery.kubernetes "<DISCOVERY_LABEL>" {
  role = "service"
}
```
  Replace the following:
  - <DISCOVERY_LABEL>: A label for the component, such as services. The label you use must be unique across all discovery.kubernetes components in the same configuration file.
  This will generate one Prometheus target for every exposed port on every discovered Service.
2. To limit the Namespaces that Services are discovered in, add the following block inside the discovery.kubernetes component.
  Alloy
```
namespaces {
  own_namespace = true
  names         = [<NAMESPACE_NAMES>]
}
```
  Replace the following:
  - <NAMESPACE_NAMES>: A comma-delimited list of strings representing Namespaces to search. Each string must be wrapped in double quotes. For example, "default","kube-system".
  If you don’t want to search for Services in the Namespace Grafana Agent Flow is running in, set own_namespace to false.
3. To use a field selector to limit the number of discovered Services, add the following block inside the discovery.kubernetes component.
  Alloy
```
selectors {
  role  = "service"
  field = "<FIELD_SELECTOR>"
}
```
  Replace the following:
  - <FIELD_SELECTOR>: The Kubernetes field selector, such as metadata.name=my-service. For more information on field selectors, refer to the Kubernetes documentation on Field Selectors.
  Create additional selectors blocks for each field selector you want to apply.
4. To use a label selector to limit the number of discovered Services, add the following block inside the discovery.kubernetes component.
  Alloy
```
selectors {
  role  = "service"
  label = "<LABEL_SELECTOR>"
}
```
  Replace the following:
  - <LABEL_SELECTOR>: The Kubernetes label selector, such as environment in (production, qa). For more information on label selectors, refer to the Kubernetes documentation on Labels and Selectors.
  Create additional selectors blocks for each label selector you want to apply.
Collect metrics from discovered Services:
1. Add the following prometheus.scrape component to your configuration file.
  Alloy
```
prometheus.scrape "<SCRAPE_LABEL>" {
  targets    = discovery.kubernetes.<DISCOVERY_LABEL>.targets
  forward_to = [prometheus.remote_write.<REMOTE_WRITE_LABEL>.receiver]
}
```
  Replace the following:
  - <SCRAPE_LABEL>: The label for the component, such as services. The label you use must be unique across all prometeus.scrape components in the same configuration file.
  - <DISCOVERY_LABEL>: The label for the discovery.kubernetes component.
  - <REMOTE_WRITE_LABEL>: The label for your existing prometheus.remote_write component.

The following example demonstrates configuring Grafana Agent Flow to collect metrics from running production Kubernetes Services in the default Namespace.

discovery.kubernetes "services" {
  role = "service"

  namespaces {
    own_namespace = false

    names = ["default"]
  }

  selectors {
    role  = "service"
    label = "environment in (production)"
  }
}

prometheus.scrape "services" {
  targets    = discovery.kubernetes.services.targets
  forward_to = [prometheus.remote_write.default.receiver]
}

prometheus.remote_write "default" {
  endpoint {
    url = "http://localhost:9090/api/prom/push"
  }
}

For more information on configuring Kubernetes service delivery and collecting metrics, refer to discovery.kubernetes and prometheus.scrape.

Collect metrics from custom targets

You can configure Grafana Agent Flow to collect metrics from a custom set of targets without the need for service discovery.

To collect metrics from a custom set of targets, complete the following steps.

Follow Configure metrics delivery to ensure collected metrics can be written somewhere.
Add the following prometheus.scrape component to your configuration file:
Alloy
```
prometheus.scrape "<SCRAPE_LABEL>" {
  targets    = [<TARGET_LIST>]
  forward_to = [prometheus.remote_write.<REMOTE_WRITE_LABEL>.receiver]
}
```
Replace the following:
- _<SCRAPE_LABEL>: The label for the component, such as custom_targets. The label you use must be unique across all prometheus.scrape components in the same configuration file.
- <TARGET_LIST>: A comma-delimited list of Objects denoting the Prometheus target. Each object must conform to the following rules:
  - There must be an __address__ key denoting the HOST:PORT of the target to collect metrics from.
  - To explicitly specify which protocol to use, set the __scheme__ key to "http" or "https". If the __scheme__ key isn’t provided, the protocol to use is inherited by the settings of the prometheus.scrape component. The default is "http".
  - To explicitly specify which HTTP path to collect metrics from, set the __metrics_path__ key to the HTTP path to use. If the __metrics_path__ key isn’t provided, the path to use is inherited by the settings of the prometheus.scrape component. The default is "/metrics".
  - Add additional keys as desired to inject extra labels to collected metrics. Any label starting with two underscores (__) will be dropped prior to scraping.
- <REMOTE_WRITE_LABEL>: The label for your existing prometheus.remote_write component.

The following example demonstrates configuring prometheus.scrape to collect metrics from a custom set of endpoints.

prometheus.scrape "custom_targets" {
  targets = [
    {
      __address__ = "prometheus:9090",
    },
    {
      __address__ = "mimir:8080",
      __scheme__  = "https",
    },
    {
      __address__      = "custom-application:80",
      __metrics_path__ = "/custom-metrics–path",
    },
    {
      __address__ = "grafana-agent:12345",
      application = "grafana-agent",
      environment = "production",
    },
  ]

  forward_to = [prometheus.remote_write.default.receiver]
}

prometheus.remote_write "default" {
  endpoint {
    url = "http://localhost:9090/api/prom/push"
  }
}

Set up meta-monitoring

Wed, 29 Jan 2025 11:00:55 +0000

Set up meta-monitoring

You can configure Grafana Agent Flow to collect its own telemetry and forward it to the backend of your choosing.

This topic describes how to collect and forward Grafana Agent Flow’s metrics, logs and traces data.

Components and configuration blocks used in this topic

Before you begin

Identify where to send Grafana Agent Flow’s telemetry data.
Be familiar with the concept of Components in Grafana Agent Flow.

Meta-monitoring metrics

Grafana Agent Flow exposes its internal metrics using the Prometheus exposition format.

In this task, you will use the prometheus.exporter.self and prometheus.scrape components to scrape Grafana Agent Flow’s internal metrics and forward it to compatible Grafana Agent Flow components.

Add the following prometheus.exporter.self component to your configuration. The component accepts no arguments.
Alloy
```
prometheus.exporter.self "<SELF_LABEL>" {
}
```
Add the following prometheus.scrape component to your configuration file.
Alloy
```
prometheus.scrape "<SCRAPE_LABEL>" {
  targets    = prometheus.exporter.<SELF_LABEL>.default.targets
  forward_to = [<METRICS_RECEIVER_LIST>]
}
```
Replace the following:
- <SELF_LABEL>: The label for the component such as default or metamonitoring. The label must be unique across all prometheus.exporter.self components in the same configuration file.
- <SCRAPE_LABEL>: The label for the scrape component such as default. The label must be unique across all prometheus.scrape components in the same configuration file.
- <METRICS_RECEIVER_LIST>: A comma-delimited list of component receivers to forward metrics to. For example, to send to an existing remote write component, use prometheus.remote_write.WRITE_LABEL.receiver. Similarly, to send data to an existing relabeling component, use prometheus.relabel.PROCESS_LABEL.receiver. To use data in the OTLP format, you can send data to an existing converter component, like otelcol.receiver.prometheus.OTEL.receiver.

The following example demonstrates configuring a possible sequence of components.

prometheus.exporter.self "default" {
}

prometheus.scrape "metamonitoring" {
  targets    = prometheus.exporter.self.default.targets
  forward_to = [prometheus.remote_write.default.receiver]
}

prometheus.remote_write "default" {
  endpoint {
    url = "http://mimir:9009/api/v1/push"
  }
}

Meta-monitoring logs

The logging block defines the logging behavior of Grafana Agent Flow.

In this task, you will use the logging block to forward Grafana Agent Flow’s logs to a compatible component. The block is specified without a label and can only be provided once per configuration file.

Add the following logging configuration block to the top level of your configuration file.
Alloy
```
logging {
  level    = "<LOG_LEVEL>"
  format   = "<LOG_FORMAT>"
  write_to = [<LOGS_RECEIVER_LIST>]
}
```
Replace the following:
- <LOG_LEVEL>: The log level to use for Grafana Agent Flow’s logs. If the attribute isn’t set, it defaults to info.
- <LOG_FORMAT>: The log format to use for Grafana Agent Flow’s logs. If the attribute isn’t set, it defaults to logfmt.
- <LOGS_RECEIVER_LIST>: A comma-delimited list of component receivers to forward logs to. For example, to send to an existing processing component, use loki.process.PROCESS_LABEL.receiver. Similarly, to send data to an existing relabeling component, use loki.relabel.PROCESS_LABEL.receiver. To use data in the OTLP format, you can send data to an existing converter component, like otelcol.receiver.loki.OTEL.receiver.

The following example demonstrates configuring the logging block and sending to a compatible component.

logging {
  level    = "warn"  
  format   = "json"
  write_to = [loki.write.default.receiver]
}

loki.write "default" {
  endpoint {
    url = "http://loki:3100/loki/api/v1/push"
  }
}

Meta-monitoring traces

The tracing block defines the tracing behavior of Grafana Agent Flow.

In this task you will use the tracing block to forward Grafana Agent Flow internal traces to a compatible component. The block is specified without a label and can only be provided once per configuration file.

Add the following tracing configuration block to the top level of your configuration file.
Alloy
```
tracing {
  sampling_fraction = <SAMPLING_FRACTION>
  write_to          = [<TRACES_RECEIVER_LIST>]
}
```
Replace the following:
- <SAMPLING_FRACTION>: The fraction of traces to keep. If the attribute isn’t set, it defaults to 0.1.
- <TRACES_RECEIVER_LIST>: A comma-delimited list of component receivers to forward traces to. For example, to send to an existing OpenTelemetry exporter component use otelcol.exporter.otlp.EXPORT_LABEL.input.

The following example demonstrates configuring the tracing block and sending to a compatible component.

tracing {
  sampling_fraction = 0.1
  write_to          = [otelcol.exporter.otlp.default.input]
}

otelcol.exporter.otlp "default" {
    client {
        endpoint = "tempo:4317"
    }
}

Collect OpenTelemetry data

Wed, 29 Jan 2025 11:00:55 +0000

Collect OpenTelemetry data

Grafana Agent Flow can be configured to collect OpenTelemetry-compatible data and forward it to any OpenTelemetry-compatible endpoint.

This topic describes how to:

Configure OpenTelemetry data delivery.
Configure batching.
Receive OpenTelemetry data over OTLP.

Components used in this topic

Before you begin

Ensure that you have basic familiarity with instrumenting applications with OpenTelemetry.
Have a set of OpenTelemetry applications ready to push telemetry data to Grafana Agent Flow.
Identify where Grafana Agent Flow writes received telemetry data.
Be familiar with the concept of Components in Grafana Agent Flow.

Configure an OpenTelemetry Protocol exporter

Before components can receive OpenTelemetry data, you must have a component responsible for exporting the OpenTelemetry data. An OpenTelemetry exporter component is responsible for writing (exporting) OpenTelemetry data to an external system.

In this task, you will use the otelcol.exporter.otlp component to send OpenTelemetry data to a server using the OpenTelemetry Protocol (OTLP). After an exporter component is defined, you can use other Grafana Agent Flow components to forward data to it.

Refer to the list of available Components for the full list of otelcol.exporter components that you can use to export OpenTelemetry data.

To configure an otelcol.exporter.otlp component for exporting OpenTelemetry data using OTLP, complete the following steps:

Add the following otelcol.exporter.otlp component to your configuration file:
Alloy
```
otelcol.exporter.otlp "<EXPORTER_LABEL>" {
  client {
    url = "<HOST>:<PORT>"
  }
}
```
Replace the following:
- <EXPORTER_LABEL>: The label for the component, such as default. The label you use must be unique across all otelcol.exporter.otlp components in the same configuration file.
- <HOST>: The hostname or IP address of the server to send OTLP requests to.
- <PORT>: The port of the server to send OTLP requests to.
If your server requires basic authentication, complete the following:
1. Add the following otelcol.auth.basic component to your configuration file:
  Alloy
```
otelcol.auth.basic "<BASIC_AUTH_LABEL>" {
  username = "<USERNAME>"
  password = "<PASSWORD>"
}
```
  Replace the following:
  - <BASIC_AUTH_LABEL>: The label for the component, such as default. The label you use must be unique across all otelcol.auth.basic components in the same configuration file.
  - <USERNAME>: The basic authentication username.
  - <PASSWORD>: The basic authentication password or API key.
2. Add the following line inside of the client block of your otelcol.exporter.otlp component:
  Alloy
```
auth = otelcol.auth.basic.<BASIC_AUTH_LABEL>.handler
```
  Replace the following:
  - <BASIC_AUTH_LABEL>: The label for the otelcol.auth.basic component.
If you have more than one server to export metrics to, create a new otelcol.exporter.otlp component for each additional server.

otelcol.exporter.otlp sends data using OTLP over gRPC (HTTP/2). To send to a server using HTTP/1.1, follow the preceding steps, but use the otelcol.exporter.otlphttp component instead.

The following example demonstrates configuring otelcol.exporter.otlp with authentication and a component that forwards data to it:

otelcol.exporter.otlp "default" {
  client {
    endpoint = "my-otlp-grpc-server:4317"
    auth     = otelcol.auth.basic.credentials.handler
  }
}

otelcol.auth.basic "credentials" {
  // Retrieve credentials using environment variables.

  username = env("BASIC_AUTH_USER")
  password = env("API_KEY")
}

otelcol.receiver.otlp "example" {
  grpc {
    endpoint = "127.0.0.1:4317"
  }

  http {
    endpoint = "127.0.0.1:4318"
  }

  output {
    metrics = [otelcol.exporter.otlp.default.input]
    logs    = [otelcol.exporter.otlp.default.input]
    traces  = [otelcol.exporter.otlp.default.input]
  }
}

For more information on writing OpenTelemetry data using the OpenTelemetry Protocol, refer to otelcol.exporter.otlp.

Configure batching

Production-ready Grafana Agent Flow configurations shouldn’t send OpenTelemetry data directly to an exporter for delivery. Instead, data is usually sent to one or more processor components that perform various transformations on the data.

Ensuring data is batched is a production-readiness step to improve data compression and reduce the number of outgoing network requests to external systems.

In this task, you will configure an otelcol.processor.batch component to batch data before sending it to the exporter.

Refer to the list of available Components for the full list of otelcol.processor components that you can use to process OpenTelemetry data. You can chain processors by having one processor send data to another processor.

To configure an otelcol.processor.batch component, complete the following steps:

Follow Configure an OpenTelemetry Protocol exporter to ensure received data can be written to an external system.
Add the following otelcol.processor.batch component into your configuration file:
Alloy
```
otelcol.processor.batch "<PROCESSOR_LABEL>" {
  output {
    metrics = [otelcol.exporter.otlp.<EXPORTER_LABEL>.input]
    logs    = [otelcol.exporter.otlp.<EXPORTER_LABEL>.input]
    traces  = [otelcol.exporter.otlp.>EXPORTER_LABEL>.input]
  }
}
```
Replace the following:
- <PROCESSOR_LABEL>: The label for the component, such as default. The label you use must be unique across all otelcol.processor.batch components in the same configuration file.
- <EXPORTER_LABEL>: The label for your existing otelcol.exporter.otlp component.
1. To disable one of the telemetry types, set the relevant type in the output block to the empty list, such as metrics = [].
2. To send batched data to another processor, replace the components in the output list with the processor components to use.

The following example demonstrates configuring a sequence of otelcol.processor components before being exported.

otelcol.processor.memory_limiter "default" {
  check_interval = "1s"
  limit          = "1GiB"

  output {
    metrics = [otelcol.processor.batch.default.input]
    logs    = [otelcol.processor.batch.default.input]
    traces  = [otelcol.processor.batch.default.input]
  }
}

otelcol.processor.batch "default" {
  output {
    metrics = [otelcol.exporter.otlp.default.input]
    logs    = [otelcol.exporter.otlp.default.input]
    traces  = [otelcol.exporter.otlp.default.input]
  }
}

otelcol.exporter.otlp "default" {
  client {
    endpoint = "my-otlp-grpc-server:4317"
  }
}

For more information on configuring OpenTelemetry data batching, refer to otelcol.processor.batch.

Configure an OpenTelemetry Protocol receiver

You can configure Grafana Agent Flow to receive OpenTelemetry metrics, logs, and traces. An OpenTelemetry receiver component is responsible for receiving OpenTelemetry data from an external system.

In this task, you will use the otelcol.receiver.otlp component to receive OpenTelemetry data over the network using the OpenTelemetry Protocol (OTLP). You can configure a receiver component to forward received data to other Grafana Agent Flow components.

Refer to the list of available Components for the full list of otelcol.receiver components that you can use to receive OpenTelemetry-compatible data.

To configure an otelcol.receiver.otlp component for receiving OTLP data, complete the following steps:

Follow Configure an OpenTelemetry Protocol exporter to ensure received data can be written to an external system.
Optional: Follow Configure batching to improve compression and reduce the total amount of network requests.
Add the following otelcol.receiver.otlp component to your configuration file.
Alloy
```
otelcol.receiver.otlp "<LABEL>" {
  output {
    metrics = [<COMPONENT_INPUT_LIST>]
    logs    = [<COMPONENT_INPUT_LIST>]
    traces  = [<COMPONENT_INPUT_LIST>]
  }
}
```
Replace the following:
- <LABEL>: The label for the component, such as default. The label you use must be unique across all otelcol.receiver.otlp components in the same configuration file.
- <COMPONENT_INPUT_LIST>: A comma-delimited list of component inputs to forward received data to. For example, to send data to an existing batch processor component, use otelcol.processor.batch.PROCESSOR_LABEL.input. To send data directly to an existing exporter component, use otelcol.exporter.otlp.EXPORTER_LABEL.input.
1. To allow applications to send OTLP data over gRPC on port 4317, add the following to your otelcol.receiver.otlp component.
  Alloy
```
grpc {
  endpoint = "<HOST>:4317"
}
```
  Replace the following:
  - <HOST>: A host to listen to traffic on. Use a narrowly scoped listen address whenever possible. To listen on all network interfaces, replace <HOST> with 0.0.0.0.
2. To allow applications to send OTLP data over HTTP/1.1 on port 4318, add the following to your otelcol.receiver.otlp component.
  Alloy
```
http {
  endpoint = "<HOST>:4318"
}
```
  Replace the following:
  - <HOST>: The host to listen to traffic on. Use a narrowly scoped listen address whenever possible. To listen on all network interfaces, replace <HOST> with 0.0.0.0.
3. To disable one of the telemetry types, set the relevant type in the output block to the empty list, such as metrics = [].

The following example demonstrates configuring otelcol.receiver.otlp and sending it to an exporter:

otelcol.receiver.otlp "example" {
  grpc {
    endpoint = "127.0.0.1:4317"
  }

  http {
    endpoint = "127.0.0.1:4318"
  }

  output {
    metrics = [otelcol.processor.batch.example.input]
    logs    = [otelcol.processor.batch.example.input]
    traces  = [otelcol.processor.batch.example.input]
  }
}

otelcol.processor.batch "example" {
  output {
    metrics = [otelcol.exporter.otlp.default.input]
    logs    = [otelcol.exporter.otlp.default.input]
    traces  = [otelcol.exporter.otlp.default.input]
  }
}

otelcol.exporter.otlp "default" {
  client {
    endpoint = "my-otlp-grpc-server:4317"
  }
}

For more information on receiving OpenTelemetry data using the OpenTelemetry Protocol, refer to otelcol.receiver.otlp.

OpenTelemetry to Grafana stack

Wed, 29 Jan 2025 11:00:55 +0000

OpenTelemetry to Grafana stack

You can configure Grafana Agent Flow to collect OpenTelemetry-compatible data and forward it to the Grafana stack.

This topic describes how to:

Configure Grafana Agent Flow to send your data to Loki.
Configure Grafana Agent Flow to send your data to Tempo.
Configure Grafana Agent Flow to send your data to Mimir or Prometheus Remote Write.

Components used in this topic

Before you begin

Ensure that you have basic familiarity with instrumenting applications with OpenTelemetry.
Have a set of OpenTelemetry applications ready to push telemetry data to Grafana Agent Flow.
Identify where Grafana Agent Flow will write received telemetry data.
Be familiar with the concept of Components in Grafana Agent Flow.
Complete the Collect open telemetry data task. You will pick up from where that guide ended.

The pipeline

You can start with the Grafana Agent Flow configuration you created in the Collect open telemetry data task.

otelcol.receiver.otlp "example" {
  grpc {
    endpoint = "127.0.0.1:4317"
  }

  http {
    endpoint = "127.0.0.1:4318"
  }

  output {
    metrics = [otelcol.processor.batch.example.input]
    logs    = [otelcol.processor.batch.example.input]
    traces  = [otelcol.processor.batch.example.input]
  }
}

otelcol.processor.batch "example" {
  output {
    metrics = [otelcol.exporter.otlp.default.input]
    logs    = [otelcol.exporter.otlp.default.input]
    traces  = [otelcol.exporter.otlp.default.input]
  }
}

otelcol.exporter.otlp "default" {
  client {
    endpoint = "my-otlp-grpc-server:4317"
  }
}

The pipeline currently looks like this:

Metrics, Logs, Traces: OTLP Receiver → batch processor → OTLP Exporter

You will implement the following pipelines to send your data to Loki, Tempo, and Mimir or Prometheus.

Metrics: OTel → batch processor → Mimir or Prometheus remote write
Logs: OTel → batch processor → Loki exporter
Traces: OTel → batch processor → OTel exporter

Grafana Loki

Grafana Loki is a horizontally scalable, highly available, multi-tenant log aggregation system inspired by Prometheus. Similar to Prometheus, to send from OTLP to Loki, you can do a passthrough from the otelcol.exporter.loki component to loki.write component.

otelcol.exporter.loki "default" {
	forward_to = [loki.write.default.receiver]
}

loki.write "default" {
	endpoint {
		url = "http://loki-endpoint:8080/loki/api/v1/push"
}

To use Loki with basic-auth, which is required with Grafana Cloud Loki, you must configure the loki.write component. You can get the Loki configuration from the Loki Details page in the Grafana Cloud Portal:

otelcol.exporter.loki "grafana_cloud_loki" {
	forward_to = [loki.write.grafana_cloud_loki.receiver]
}

loki.write "grafana_cloud_loki" {
	endpoint {
		url = "https://logs-prod-us-central1.grafana.net/loki/api/v1/push"

		basic_auth {
			username = 5252
			password = env("GRAFANA_CLOUD_API_KEY")
		}
	}
}

Grafana Tempo

Grafana Tempo is an open source, easy-to-use, scalable distributed tracing backend. Tempo can ingest OTLP directly, and you can use the OTLP exporter to send the traces to Tempo.

otelcol.exporter.otlp "default" {
  client {
    endpoint = "tempo-server:4317"
  }
}

To use Tempo with basic-auth, which is required with Grafana Cloud Tempo, you must use the otelcol.auth.basic component. You can get the Tempo configuration from the Tempo Details page in the Grafana Cloud Portal:

otelcol.exporter.otlp "grafana_cloud_tempo" {
	client {
		endpoint = "tempo-us-central1.grafana.net:443"
		auth     = otelcol.auth.basic.grafana_cloud_tempo.handler
	}
}

otelcol.auth.basic "grafana_cloud_tempo" {
	username = 4094
	password = env("GRAFANA_CLOUD_API_KEY")
}

Grafana Mimir or Prometheus Remote Write

Prometheus Remote Write is a popular metrics transmission protocol supported by most metrics systems, including Grafana Mimir and Grafana Cloud. To send from OTLP to Prometheus, you can do a passthrough from the otelcol.exporter.prometheus to the prometheus.remote_write component. The Prometheus remote write component in Grafana Agent Flow is a robust protocol implementation, including a Write Ahead Log (WAL) for resiliency.

otelcol.exporter.prometheus "default" {
	forward_to = [prometheus.remote_write.default.receiver]
}

prometheus.remote_write "default" {
	endpoint {
		url = "http://prometheus:9090/api/v1/write"
	}
}

To use Prometheus with basic-auth, which is required with Grafana Cloud Prometheus, you must configure the prometheus.remote_write component. You can get the Prometheus configuration from the Prometheus Details page in the Grafana Cloud Portal:

otelcol.exporter.prometheus "grafana_cloud_prometheus" {
        forward_to = [prometheus.remote_write.grafana_cloud_prometheus.receiver]
    }

prometheus.remote_write "grafana_cloud_prometheus" {
    endpoint {
        url = "https://prometheus-us-central1.grafana.net/api/prom/push"

        basic_auth {
            username = 12690
            password = env("GRAFANA_CLOUD_API_KEY")
        }
    }
}

Putting it all together

Instead of referencing otelcol.exporter.otlp.default.input in the output of otelcol.processor.batch, you need to reference the three exporters you set up. The final configuration becomes:

otelcol.receiver.otlp "example" {
  grpc {
    endpoint = "127.0.0.1:4317"
  }

  http {
    endpoint = "127.0.0.1:4318"
  }

  output {
    metrics = [otelcol.processor.batch.example.input]
    logs    = [otelcol.processor.batch.example.input]
    traces  = [otelcol.processor.batch.example.input]
  }
}

otelcol.processor.batch "example" {
  output {
    metrics = [otelcol.exporter.prometheus.grafana_cloud_prometheus.input]
    logs    = [otelcol.exporter.loki.grafana_cloud_loki.input]
    traces  = [otelcol.exporter.otlp.grafana_cloud_tempo.input]
  }
}

otelcol.exporter.otlp "grafana_cloud_tempo" {
	client {
		endpoint = "tempo-us-central1.grafana.net:443"
		auth     = otelcol.auth.basic.grafana_cloud_tempo.handler
	}
}

otelcol.auth.basic "grafana_cloud_tempo" {
	username = 4094
	password = env("GRAFANA_CLOUD_API_KEY")
}

otelcol.exporter.prometheus "grafana_cloud_prometheus" {
        forward_to = [prometheus.remote_write.grafana_cloud_prometheus.receiver]
    }

prometheus.remote_write "grafana_cloud_prometheus" {
    endpoint {
        url = "https://prometheus-us-central1.grafana.net/api/prom/push"

        basic_auth {
            username = 12690
            password = env("GRAFANA_CLOUD_API_KEY")
        }
    }
}

otelcol.exporter.loki "grafana_cloud_loki" {
	forward_to = [loki.write.grafana_cloud_loki.receiver]
}

loki.write "grafana_cloud_loki" {
	endpoint {
		url = "https://logs-prod-us-central1.grafana.net/loki/api/v1/push"

		basic_auth {
			username = 5252
			password = env("GRAFANA_CLOUD_API_KEY")
		}
	}
}

Running Grafana Agent Flow now will give you the following:

AGENT_MODE=flow ./grafana-agent run agent-config.river
AGENT_MODE=flow ./grafana-agent run agent-config.river
ts=2023-05-09T09:37:15.300959Z level=info msg="running usage stats reporter"
ts=2023-05-09T09:37:15.300958Z level=info msg="now listening for http traffic" addr=127.0.0.1:12345
ts=2023-05-09T09:37:15.301104Z level=info trace_id=6466516c9e1a556422df7a84c0ade6b0 msg="starting complete graph evaluation"
ts=2023-05-09T09:37:15.301307Z level=info trace_id=6466516c9e1a556422df7a84c0ade6b0 msg="finished node evaluation" node_id=loki.write.grafana_cloud_loki duration=188.209µs
ts=2023-05-09T09:37:15.301334Z level=info trace_id=6466516c9e1a556422df7a84c0ade6b0 msg="finished node evaluation" node_id=otelcol.exporter.loki.grafana_cloud_loki duration=18.791µs
ts=2023-05-09T09:37:15.303138Z component=prometheus.remote_write.grafana_cloud_prometheus level=info subcomponent=wal msg="replaying WAL, this may take a while" dir=data-agent/prometheus.remote_write.grafana_cloud_prometheus/wal
ts=2023-05-09T09:37:15.303257Z component=prometheus.remote_write.grafana_cloud_prometheus level=info subcomponent=wal msg="WAL segment loaded" segment=0 maxSegment=1
ts=2023-05-09T09:37:15.303302Z component=prometheus.remote_write.grafana_cloud_prometheus level=info subcomponent=wal msg="WAL segment loaded" segment=1 maxSegment=1
ts=2023-05-09T09:37:15.303507Z component=prometheus.remote_write.grafana_cloud_prometheus subcomponent=rw level=info remote_name=7f623a url=https://prometheus-us-central1.grafana.net/api/prom/push msg="Starting WAL watcher" queue=7f623a
ts=2023-05-09T09:37:15.303515Z component=prometheus.remote_write.grafana_cloud_prometheus subcomponent=rw level=info remote_name=7f623a url=https://prometheus-us-central1.grafana.net/api/prom/push msg="Starting scraped metadata watcher"
ts=2023-05-09T09:37:15.303522Z level=info trace_id=6466516c9e1a556422df7a84c0ade6b0 msg="finished node evaluation" node_id=prometheus.remote_write.grafana_cloud_prometheus duration=2.181958ms
ts=2023-05-09T09:37:15.303557Z level=info trace_id=6466516c9e1a556422df7a84c0ade6b0 msg="finished node evaluation" node_id=otelcol.exporter.prometheus.grafana_cloud_prometheus duration=30.083µs
ts=2023-05-09T09:37:15.303611Z component=prometheus.remote_write.grafana_cloud_prometheus subcomponent=rw level=info remote_name=7f623a url=https://prometheus-us-central1.grafana.net/api/prom/push msg="Replaying WAL" queue=7f623a
ts=2023-05-09T09:37:15.303618Z level=info trace_id=6466516c9e1a556422df7a84c0ade6b0 msg="finished node evaluation" node_id=otelcol.auth.basic.grafana_cloud_tempo duration=52.5µs
ts=2023-05-09T09:37:15.303694Z level=info trace_id=6466516c9e1a556422df7a84c0ade6b0 msg="finished node evaluation" node_id=otelcol.exporter.otlp.grafana_cloud_tempo duration=70.375µs
ts=2023-05-09T09:37:15.303782Z component=otelcol.processor.memory_limiter.default level=info msg="Memory limiter configured" limit_mib=150 spike_limit_mib=30 check_interval=1s
ts=2023-05-09T09:37:15.303802Z level=info trace_id=6466516c9e1a556422df7a84c0ade6b0 msg="finished node evaluation" node_id=otelcol.processor.memory_limiter.default duration=100.334µs
ts=2023-05-09T09:37:15.303853Z level=info trace_id=6466516c9e1a556422df7a84c0ade6b0 msg="finished node evaluation" node_id=otelcol.processor.batch.default duration=44.75µs
ts=2023-05-09T09:37:15.303948Z level=info trace_id=6466516c9e1a556422df7a84c0ade6b0 msg="finished node evaluation" node_id=otelcol.receiver.otlp.default duration=87.333µs
ts=2023-05-09T09:37:15.303968Z level=info trace_id=6466516c9e1a556422df7a84c0ade6b0 msg="finished node evaluation" node_id=tracing duration=10.792µs
ts=2023-05-09T09:37:15.303981Z level=info trace_id=6466516c9e1a556422df7a84c0ade6b0 msg="finished node evaluation" node_id=logging duration=9µs
ts=2023-05-09T09:37:15.303987Z level=info trace_id=6466516c9e1a556422df7a84c0ade6b0 msg="finished complete graph evaluation" duration=2.960333ms
ts=2023-05-09T09:37:15.304Z level=info msg="scheduling loaded components"
ts=2023-05-09T09:37:15.304109Z component=otelcol.receiver.otlp.default level=info msg="Starting GRPC server" endpoint=0.0.0.0:4317
ts=2023-05-09T09:37:15.304234Z component=otelcol.receiver.otlp.default level=info msg="Starting HTTP server" endpoint=0.0.0.0:4318

You can now check the pipeline graphically by visiting http://localhost:12345/graph

Configure Grafana Agent clustering in an existing installation

Wed, 29 Jan 2025 11:00:55 +0000

Configure Grafana Agent Flow clustering in an existing installation

You can configure Grafana Agent Flow to run with clustering so that individual Grafana Agents can work together for workload distribution and high availability.

This topic describes how to add clustering to an existing installation.

Configure Grafana Agent Flow clustering with Helm Chart

This section guides you through enabling clustering when Grafana Agent Flow is installed on Kubernetes using the Grafana Agent Helm chart.

Before you begin

Ensure that your values.yaml file has controller.type set to statefulset.

Steps

To configure clustering:

Amend your existing values.yaml file to add clustering.enabled=true inside the agent block.
YAML
```
agent:
  clustering:
    enabled: true
```
Upgrade your installation to use the new values.yaml file:
Bash
```
helm upgrade <RELEASE_NAME> -f values.yaml
```
Replace the following:
- <RELEASE_NAME>: The name of the installation you chose when you installed the Helm chart.
Use the Grafana Agent Flow UI to verify the cluster status:
1. Click Clustering in the navigation bar.
2. Ensure that all expected nodes appear in the resulting table.

Distribute Prometheus metrics scrape load

Wed, 29 Jan 2025 11:00:55 +0000

Distribute Prometheus metrics scrape load

A good predictor for the size of an Grafana Agent Flow deployment is the number of Prometheus targets each Grafana Agent scrapes. Clustering with target auto-distribution allows a fleet of Grafana Agents to work together to dynamically distribute their scrape load, providing high-availability.

Before you begin

Familiarize yourself with how to configure existing Grafana Agent Flow installations.
Configure Prometheus metrics collection.
Configure clustering.
Ensure that all of your clustered Grafana Agents have the same configuration file.

Steps

To distribute Prometheus metrics scrape load with clustering:

Add the following block to all prometheus.scrape components, which should use auto-distribution:
Alloy
```
clustering {
  enabled = true
}
```
Restart or reload Grafana Agents for them to use the new configuration.
Validate that auto-distribution is functioning:
1. Using the Grafana Agent UI on each Grafana Agent, navigate to the details page for one of the prometheus.scrape components you modified.
2. Compare the Debug Info sections between two different Grafana Agent to ensure that they’re not scraping the same sets of targets.

Debug issues with Grafana Agent Flow

Wed, 29 Jan 2025 11:00:55 +0000

Debug Grafana Agent Flow

Follow these steps to debug issues with Grafana Agent Flow:

Use the Grafana Agent Flow UI to debug issues.
If the Grafana Agent Flow UI doesn’t help with debugging an issue, logs can be examined instead.

Grafana Agent Flow UI

Grafana Agent Flow includes an embedded UI viewable from the Grafana Agent HTTP server, which defaults to listening at http://localhost:12345.

NOTE: For security reasons, installations of Grafana Agent Flow on non-containerized platforms default to listening on localhost. This default prevents other machines on the network from being able to view the UI.

To expose the UI to other machines on the network on non-containerized platforms, refer to the documentation for how you installed Grafana Agent Flow.

If you are running a custom installation of Grafana Agent Flow, refer to the documentation for [the grafana-agent run command][grafana-agent run] to learn how to change the HTTP listen address, and pass the appropriate flag when running Grafana Agent Flow.

Home page

The home page shows a table of components defined in the configuration file and their health.

Click View on a row in the table to navigate to the Component detail page for that component.

Click the Grafana Agent logo to navigate back to the home page.

Graph page

The Graph page shows a graph view of components defined in the configuration file and their health. Clicking a component in the graph navigates to the Component detail page for that component.

Component detail page

The component detail page shows the following information for each component:

The health of the component with a message explaining the health.
The current evaluated arguments for the component.
The current exports for the component.
The current debug info for the component (if the component has debug info).

Values marked as a secret are obfuscated and display as the text (secret).

Clustering page

The Clustering page shows the following information for each cluster node:

The node’s name.
The node’s advertised address.
The node’s current state (Viewer/Participant/Terminating).
The local node that serves the UI.

Debugging using the UI

To debug using the UI:

Ensure that no component is reported as unhealthy.
Ensure that the arguments and exports for misbehaving components appear correct.

Examining logs

Logs may also help debug issues with Grafana Agent Flow.

To reduce logging noise, many components hide debugging info behind debug-level log lines. It is recommended that you configure the logging block to show debug-level log lines when debugging issues with Grafana Agent Flow.

The location of Grafana Agent Flow logs is different based on how it’s deployed. Refer to the logging block page to see how to find logs for your system.

Debugging clustering issues

To debug issues when using clustering, check for the following symptoms.

Cluster not converging: The cluster peers aren’t converging on the same view of their peers’ status. This is most likely due to network connectivity issues between the cluster nodes. Use the Grafana Agent Flow UI of each running peer to understand which nodes aren’t being picked up correctly.
Cluster split brain: The cluster peers aren’t aware of one another, thinking they’re the only node present. Again, check for network connectivity issues. Check that the addresses or DNS names given to the node to join are correctly formatted and reachable.
Configuration drift: Clustering assumes that all nodes are running with the same configuration file at roughly the same time. Check the logs for issues with the reloaded configuration file as well as the graph page to verify changes have been applied.
Node name conflicts: Clustering assumes all nodes have unique names. Nodes with conflicting names are rejected and won’t join the cluster. Look at the clustering UI page for the list of current peers with their names, and check the logs for any reported name conflict events.
Node stuck in terminating state: The node attempted to gracefully shut down and set its state to Terminating, but it has not completely gone away. Check the clustering page to view the state of the peers and verify that the terminating Grafana Agent has been shut down.

Note
Some issues that appear to be clustering issues may be symptoms of other issues, for example, problems with scraping or service discovery can result in missing metrics for an agent that can be interpreted as a node not joining the cluster.