Migrate to Grafana Agent Flow on Grafana Labs

Migrate from Grafana Agent Operator to Grafana Agent Flow

Wed, 11 Sep 2024 17:43:35 +0000

Migrate from Grafana Agent Operator to Grafana Agent Flow

With the release of Grafana Agent Flow, Grafana Agent Operator is no longer the recommended way to deploy Grafana Agent in Kubernetes. Some of the Operator functionality has moved into Grafana Agent Flow itself, and the Helm Chart has replaced the remaining functionality.

The Monitor types (PodMonitor, ServiceMonitor, Probe, and PodLogs) are all supported natively by Grafana Agent Flow. You are no longer required to use the Operator to consume those CRDs for dynamic monitoring in your cluster.
The parts of the Operator that deploy the Grafana Agent itself (GrafanaAgent, MetricsInstance, and LogsInstance CRDs) are deprecated. Operator users should use the Grafana Agent [Helm Chart][] to deploy Grafana Agent directly to your clusters.

This guide provides some steps to get started with Grafana Agent Flow for users coming from Grafana Agent Operator.

Deploy Grafana Agent Flow with Helm

Create a values.yaml file, which contains options for deploying your Grafana Agent. You can start with the default values and customize as you see fit, or start with this snippet, which should be a good starting point for what the Operator does.
YAML
```
agent:
  mode: 'flow'
  configMap:
    create: true
  clustering:
    enabled: true
controller:
  type: 'statefulset'
  replicas: 2
crds:
  create: false
```
This configuration deploys Grafana Agent Flow as a StatefulSet using the built-in clustering functionality to allow distributing scrapes across all Grafana Agent Pods.

This is one of many deployment possible modes. For example, you may want to use a DaemonSet to collect host-level logs or metrics. See the Grafana Agent Flow deployment guide for more details about different topologies.
Create a Grafana Agent configuration file, agent.river.

In the next step, you add to this configuration as you convert MetricsInstances. You can add any additional configuration to this file as you need.

Install the Grafana Helm repository:

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

Create a Helm release. You can name the release anything you like. The following command installs a release called grafana-agent-metrics in the monitoring namespace.
shell
```
helm upgrade grafana-agent-metrics grafana/grafana-agent -i -n monitoring -f values.yaml --set-file agent.configMap.content=agent.river
```
This command uses the --set-file flag to pass the configuration file as a Helm value so that you can continue to edit it as a regular River file.

Convert `MetricsIntances` to Grafana Agent Flow components

A MetricsInstance resource primarily defines:

The remote endpoints Grafana Agent Flow should send metrics to.
The PodMonitor, ServiceMonitor, and Probe resources this Grafana Agent should discover.

You can use these functions in Grafana Agent Flow with the prometheus.remote_write, prometheus.operator.podmonitors, prometheus.operator.servicemonitors, and prometheus.operator.probes components respectively.

The following River sample is equivalent to the MetricsInstance from the operator guide.


// read the credentials secret for remote_write authorization
remote.kubernetes.secret "credentials" {
  namespace = "monitoring"
  name = "primary-credentials-metrics"
}

prometheus.remote_write "primary" {
    endpoint {
        url = "https://<PROMETHEUS_URL>/api/v1/push"
        basic_auth {
            username = nonsensitive(remote.kubernetes.secret.credentials.data["username"])
            password = remote.kubernetes.secret.credentials.data["password"]
        }
    }
}

prometheus.operator.podmonitors "primary" {
    forward_to = [prometheus.remote_write.primary.receiver]
    // leave out selector to find all podmonitors in the entire cluster
    selector {
        match_labels = {instance = "primary"}
    }
}

prometheus.operator.servicemonitors "primary" {
    forward_to = [prometheus.remote_write.primary.receiver]
    // leave out selector to find all servicemonitors in the entire cluster
    selector {
        match_labels = {instance = "primary"}
    }
}

Replace the following:

<PROMETHEUS_URL>: The endpoint you want to send metrics to.

This configuration discovers all PodMonitor, ServiceMonitor, and Probe resources in your cluster that match the label selector instance=primary. It then scrapes metrics from the targets and forward them to your remote write endpoint.

You may need to customize this configuration further if you use additional features in your MetricsInstance resources. Refer to the documentation for the relevant components for additional information:

Collecting Logs

Our current recommendation is to create an additional DaemonSet deployment of Grafana Agents to scrape logs.

We have components that can scrape pod logs directly from the Kubernetes API without needing a DaemonSet deployment. These are still considered experimental, but if you would like to try them, see the documentation for loki.source.kubernetes and loki.source.podlogs.

These values are close to what the Operator currently deploys for logs:

agent:
  mode: 'flow'
  configMap:
    create: true
  clustering:
    enabled: false
  controller:
    type: 'daemonset'
  mounts:
    # -- Mount /var/log from the host into the container for log collection.
    varlog: true

This command will install a release named grafana-agent-logs in the monitoring namespace:

helm upgrade grafana-agent-logs grafana/grafana-agent -i -n monitoring -f values-logs.yaml --set-file agent.configMap.content=agent-logs.river

This simple configuration will scrape logs for every pod on each node:

// read the credentials secret for remote_write authorization
remote.kubernetes.secret "credentials" {
  namespace = "monitoring"
  name      = "primary-credentials-logs"
}

discovery.kubernetes "pods" {
  role = "pod"
  // limit to pods on this node to reduce the amount you need to filter
  selectors {
    role  = "pod"
    field = "spec.nodeName=" + env("<HOSTNAME>")
  }
}

discovery.relabel "pod_logs" {
  targets = discovery.kubernetes.pods.targets
  rule {
    source_labels = ["__meta_kubernetes_namespace"]
    target_label  = "namespace"
  }
  rule {
    source_labels = ["__meta_kubernetes_pod_name"]
    target_label  = "pod"
  }
  rule {
    source_labels = ["__meta_kubernetes_pod_container_name"]
    target_label  = "container"
  }
  rule {
    source_labels = ["__meta_kubernetes_namespace", "__meta_kubernetes_pod_name"]
    separator     = "/"
    target_label  = "job"
  }
  rule {
    source_labels = ["__meta_kubernetes_pod_uid", "__meta_kubernetes_pod_container_name"]
    separator     = "/"
    action        = "replace"
    replacement   = "/var/log/pods/*$1/*.log"
    target_label  = "__path__"
  }
  rule {
    action = "replace"
    source_labels = ["__meta_kubernetes_pod_container_id"]
    regex = "^(\\w+):\\/\\/.+$"
    replacement = "$1"
    target_label = "tmp_container_runtime"
  }
}

local.file_match "pod_logs" {
  path_targets = discovery.relabel.pod_logs.output
}

loki.source.file "pod_logs" {
  targets    = local.file_match.pod_logs.targets
  forward_to = [loki.process.pod_logs.receiver]
}

// basic processing to parse the container format. You can add additional processing stages
// to match your application logs.
loki.process "pod_logs" {
  stage.match {
    selector = "{tmp_container_runtime=\"containerd\"}"
    // the cri processing stage extracts the following k/v pairs: log, stream, time, flags
    stage.cri {}
    // Set the extract flags and stream values as labels
    stage.labels {
      values = {
        flags   = "",
        stream  = "",
      }
    }
  }

  // if the label tmp_container_runtime from above is docker parse using docker
  stage.match {
    selector = "{tmp_container_runtime=\"docker\"}"
    // the docker processing stage extracts the following k/v pairs: log, stream, time
    stage.docker {}

    // Set the extract stream value as a label
    stage.labels {
      values = {
        stream  = "",
      }
    }
  }

  // drop the temporary container runtime label as it is no longer needed
  stage.label_drop {
    values = ["tmp_container_runtime"]
  }

  forward_to = [loki.write.loki.receiver]
}

loki.write "loki" {
  endpoint {
    url = "https://<LOKI_URL>/loki/api/v1/push"
    basic_auth {
      username = nonsensitive(remote.kubernetes.secret.credentials.data["username"])
      password = remote.kubernetes.secret.credentials.data["password"]
    }
}
}

Replace the following:

<LOKI_URL>: The endpoint of your Loki instance.

The logging subsystem is very powerful and has many options for processing logs. For further details, see the component documentation.

Integrations

The Integration CRD isn’t supported with Grafana Agent Flow. However, all static mode integrations have an equivalent component in the prometheus.exporter namespace. The [reference documentation][component documentation] should help convert those integrations to their Grafana Agent Flow equivalent.

Migrate from OpenTelemetry Collector to Grafana Agent Flow

Wed, 12 Mar 2025 11:45:31 +0100

Migrate from OpenTelemetry Collector to Grafana Agent Flow

The built-in Grafana Agent convert command can migrate your OpenTelemetry Collector configuration to a Grafana Agent Flow configuration.

This topic describes how to:

Convert an OpenTelemetry Collector configuration to a Grafana Agent Flow configuration.
Run an OpenTelemetry Collector configuration natively using Grafana Agent Flow.

Components used in this topic

Before you begin

You must have an existing OpenTelemetry Collector configuration.
You must have a set of OpenTelemetry Collector applications ready to push telemetry data to Grafana Agent Flow.
You must be familiar with the concept of Components in Grafana Agent Flow.

Convert an OpenTelemetry Collector configuration

To fully migrate your configuration fromOpenTelemetry Collector to Grafana Agent Flow, you must convert your OpenTelemetry Collector configuration into a Grafana Agent Flow configuration. This conversion will enable you to take full advantage of the many additional features available in Grafana Agent Flow.

In this task, you will use the convert CLI command to output a Grafana Agent Flow configuration from a OpenTelemetry Collector configuration.

Open a terminal window and run the following command.
static-binary

flow-binary
static-binary
AGENT_MODE=flow grafana-agent convert --source-format=otelcol --output=<OUTPUT_CONFIG_PATH> <INPUT_CONFIG_PATH>
flow-binary
grafana-agent-flow convert --source-format=otelcol --output=<OUTPUT_CONFIG_PATH> <INPUT_CONFIG_PATH>
Replace the following:
- <INPUT_CONFIG_PATH>: The full path to the OpenTelemetry Collector configuration.
- <OUTPUT_CONFIG_PATH>: The full path to output the Grafana Agent Flow configuration.
Run Grafana Agent Flow using the new Grafana Agent Flow configuration from <OUTPUT_CONFIG_PATH>:

Debugging

If the convert command can’t convert an OpenTelemetry Collector configuration, diagnostic information is sent to stderr.
You can bypass any non-critical issues and output the Grafana Agent Flow configuration using a best-effort conversion by including the --bypass-errors flag.

Caution
If you bypass the errors, the behavior of the converted configuration may not match the original OpenTelemetry Collector configuration. Make sure you fully test the converted configuration before using it in a production environment.
static-binary

flow-binary
static-binary
AGENT_MODE=flow grafana-agent convert --source-format=otelcol --bypass-errors --output=<OUTPUT_CONFIG_PATH> <INPUT_CONFIG_PATH>
flow-binary
grafana-agent-flow convert --source-format=otelcol --bypass-errors --output=<OUTPUT_CONFIG_PATH> <INPUT_CONFIG_PATH>
Replace the following:
- <INPUT_CONFIG_PATH>: The full path to the OpenTelemetry Collector configuration.
- <OUTPUT_CONFIG_PATH>: The full path to output the Grafana Agent Flow configuration.

You can also output a diagnostic report by including the --report flag.

static-binary

flow-binary

AGENT_MODE=flow grafana-agent convert --source-format=otelcol --report=<OUTPUT_REPORT_PATH> --output=<OUTPUT_CONFIG_PATH> <INPUT_CONFIG_PATH>

grafana-agent-flow convert --source-format=otelcol --report=<OUTPUT_REPORT_PATH> --output=<OUTPUT_CONFIG_PATH> <INPUT_CONFIG_PATH>

Replace the following:

<INPUT_CONFIG_PATH>: The full path to the OpenTelemetry Collector configuration.
<OUTPUT_CONFIG_PATH>: The full path to output the Grafana Agent Flow configuration.
<OUTPUT_REPORT_PATH>: The output path for the report.

Using the example OpenTelemetry Collector configuration below, the diagnostic report provides the following information:

(Info) Converted receiver/otlp into otelcol.receiver.otlp.default
(Info) Converted processor/memory_limiter into otelcol.processor.memory_limiter.default
(Info) Converted exporter/otlp into otelcol.exporter.otlp.default

A configuration file was generated successfully.

Run an OpenTelemetry Collector configuration

If you’re not ready to completely switch to a Grafana Agent Flow configuration, you can run Grafana Agent using your existing OpenTelemetry Collector configuration. The --config.format=otelcol flag tells Grafana Agent to convert your OpenTelemetry Collector configuration to a Grafana Agent Flow configuration and load it directly without saving the new configuration. This allows you to try Grafana Agent Flow without modifying your existing OpenTelemetry Collector configuration infrastructure.

In this task, you will use the run CLI command to run Grafana Agent Flow using an OpenTelemetry Collector configuration.

Run Grafana Agent Flow and include the command line flag --config.format=otelcol. Your configuration file must be a valid OpenTelemetry Collector configuration file rather than a Grafana Agent Flow configuration file.

Debugging

You can follow the convert CLI command debugging instructions to generate a diagnostic report.
Refer to the Grafana Agent Flow Debugging for more information about a running Grafana Agent Flow.
If your OpenTelemetry Collector configuration can’t be converted and loaded directly into Grafana Agent Flow, diagnostic information is sent to stderr. You can bypass any non-critical issues and start the Agent by including the --config.bypass-conversion-errors flag in addition to --config.format=otelcol.

Caution
If you bypass the errors, the behavior of the converted configuration may not match the original Prometheus configuration. Do not use this flag in a production environment.

Example

This example demonstrates converting an OpenTelemetry Collector configuration file to a Grafana Agent Flow configuration file.

The following OpenTelemetry Collector configuration file provides the input for the conversion.

receivers:
  otlp:
    protocols:
      grpc:
      http:

exporters:
  otlp:
    endpoint: database:4317

processors:
  memory_limiter:
    limit_percentage: 90
    check_interval: 1s


service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [memory_limiter]
      exporters: [otlp]
    logs:
      receivers: [otlp]
      processors: [memory_limiter]
      exporters: [otlp]
    traces:
      receivers: [otlp]
      processors: [memory_limiter]
      exporters: [otlp]

The convert command takes the YAML file as input and outputs a River file.

static-binary

flow-binary

AGENT_MODE=flow grafana-agent convert --source-format=otelcol --output=<OUTPUT_CONFIG_PATH> <INPUT_CONFIG_PATH>

grafana-agent-flow convert --source-format=otelcol --output=<OUTPUT_CONFIG_PATH> <INPUT_CONFIG_PATH>

Replace the following:

<INPUT_CONFIG_PATH>: The full path to the OpenTelemetry Collector configuration.
<OUTPUT_CONFIG_PATH>: The full path to output the Grafana Agent Flow configuration.

The new Grafana Agent Flow configuration file looks like this:

otelcol.receiver.otlp "default" {
	grpc { }

	http { }

	output {
		metrics = [otelcol.processor.memory_limiter.default.input]
		logs    = [otelcol.processor.memory_limiter.default.input]
		traces  = [otelcol.processor.memory_limiter.default.input]
	}
}

otelcol.processor.memory_limiter "default" {
	check_interval   = "1s"
	limit_percentage = 90

	output {
		metrics = [otelcol.exporter.otlp.default.input]
		logs    = [otelcol.exporter.otlp.default.input]
		traces  = [otelcol.exporter.otlp.default.input]
	}
}

otelcol.exporter.otlp "default" {
	client {
		endpoint = "database:4317"
	}
}

Limitations

Configuration conversion is done on a best-effort basis. Grafana Agent will issue warnings or errors where the conversion can’t be performed.

After the configuration is converted, review the Grafana Agent Flow configuration file created and verify that it’s correct before starting to use it in a production environment.

The following list is specific to the convert command and not Grafana Agent Flow:

Components are supported which directly embed upstream OpenTelemetry Collector features. You can get a general idea of which exist in Grafana Agent Flow for conversion by reviewing the otelcol.* components in the Component Reference. Any additional unsupported features are returned as errors during conversion.
Check if you are using any extra command line arguments with OpenTelemetry Collector that aren’t present in your configuration file.
Metamonitoring metrics exposed by Grafana Agent Flow usually match OpenTelemetry Collector metamonitoring metrics but will use a different name. Make sure that you use the new metric names, for example, in your alerts and dashboards queries.
The logs produced by Grafana Agent Flow differ from those produced by OpenTelemetry Collector.
Grafana Agent exposes the Grafana Agent Flow UI.

Migrate from Prometheus to Grafana Agent Flow

Wed, 12 Mar 2025 11:45:31 +0100

Migrate from Prometheus to Grafana Agent Flow

The built-in Grafana Agent convert command can migrate your Prometheus configuration to a Grafana Agent Flow configuration.

This topic describes how to:

Convert a Prometheus configuration to a Grafana Agent Flow configuration.
Run a Prometheus configuration natively using Grafana Agent Flow.

Components used in this topic

Before you begin

You must have an existing Prometheus configuration.
You must have a set of Prometheus applications ready to push telemetry data to Grafana Agent Flow.
You must be familiar with the concept of Components in Grafana Agent Flow.

Convert a Prometheus configuration

To fully migrate your configuration fromPrometheus to Grafana Agent Flow, you must convert your Prometheus configuration into a Grafana Agent Flow configuration. This conversion will enable you to take full advantage of the many additional features available in Grafana Agent Flow.

In this task, you will use the convert CLI command to output a Grafana Agent Flow configuration from a Prometheus configuration.

Open a terminal window and run the following command.
static-binary

flow-binary
static-binary
AGENT_MODE=flow grafana-agent convert --source-format=prometheus --output=<OUTPUT_CONFIG_PATH> <INPUT_CONFIG_PATH>
flow-binary
grafana-agent-flow convert --source-format=prometheus --output=<OUTPUT_CONFIG_PATH> <INPUT_CONFIG_PATH>
Replace the following:
- <INPUT_CONFIG_PATH>: The full path to the Prometheus configuration.
- <OUTPUT_CONFIG_PATH>: The full path to output the Grafana Agent Flow configuration.
Run Grafana Agent Flow using the new Grafana Agent Flow configuration from <OUTPUT_CONFIG_PATH>:

Debugging

If the convert command can’t convert a Prometheus configuration, diagnostic information is sent to stderr.
You can bypass any non-critical issues and output the Grafana Agent Flow configuration using a best-effort conversion by including the --bypass-errors flag.

Caution
If you bypass the errors, the behavior of the converted configuration may not match the original Prometheus configuration. Make sure you fully test the converted configuration before using it in a production environment.
static-binary

flow-binary
static-binary
AGENT_MODE=flow grafana-agent convert --source-format=prometheus --bypass-errors --output=<OUTPUT_CONFIG_PATH> <INPUT_CONFIG_PATH>
flow-binary
grafana-agent-flow convert --source-format=prometheus --bypass-errors --output=<OUTPUT_CONFIG_PATH> <INPUT_CONFIG_PATH>
Replace the following:
- <INPUT_CONFIG_PATH>: The full path to the Prometheus configuration.
- <OUTPUT_CONFIG_PATH>: The full path to output the Grafana Agent Flow configuration.

You can also output a diagnostic report by including the --report flag.

static-binary

flow-binary

AGENT_MODE=flow grafana-agent convert --source-format=prometheus --report=<OUTPUT_REPORT_PATH> --output=<OUTPUT_CONFIG_PATH> <INPUT_CONFIG_PATH>

grafana-agent-flow convert --source-format=prometheus --report=<OUTPUT_REPORT_PATH> --output=<OUTPUT_CONFIG_PATH> <INPUT_CONFIG_PATH>

Replace the following:

<INPUT_CONFIG_PATH>: The full path to the Prometheus configuration.
<OUTPUT_CONFIG_PATH>: The full path to output the Grafana Agent Flow configuration.
<OUTPUT_REPORT_PATH>: The output path for the report.

Using the example Prometheus configuration below, the diagnostic report provides the following information:

(Info) Converted scrape_configs job_name "prometheus" into...
  A prometheus.scrape.prometheus component
(Info) Converted 1 remote_write[s] "grafana-cloud" into...
  A prometheus.remote_write.default component

Run a Prometheus configuration

If you’re not ready to completely switch to a Grafana Agent Flow configuration, you can run Grafana Agent using your existing Prometheus configuration. The --config.format=prometheus flag tells Grafana Agent to convert your Prometheus configuration to a Grafana Agent Flow configuration and load it directly without saving the new configuration. This allows you to try Grafana Agent Flow without modifying your existing Prometheus configuration infrastructure.

In this task, you will use the run CLI command to run Grafana Agent Flow using a Prometheus configuration.

Run Grafana Agent Flow and include the command line flag --config.format=prometheus. Your configuration file must be a valid Prometheus configuration file rather than a Grafana Agent Flow configuration file.

Debugging

You can follow the convert CLI command debugging instructions to generate a diagnostic report.
Refer to the Grafana Agent Flow Debugging for more information about a running Grafana Agent Flow.
If your Prometheus configuration can’t be converted and loaded directly into Grafana Agent Flow, diagnostic information is sent to stderr. You can bypass any non-critical issues and start the Agent by including the --config.bypass-conversion-errors flag in addition to --config.format=prometheus.

Caution
If you bypass the errors, the behavior of the converted configuration may not match the original Prometheus configuration. Do not use this flag in a production environment.

Example

This example demonstrates converting a Prometheus configuration file to a Grafana Agent Flow configuration file.

The following Prometheus configuration file provides the input for the conversion.

global:
  scrape_timeout:    45s

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:12345"]

remote_write:
  - name: "grafana-cloud"
    url: "https://prometheus-us-central1.grafana.net/api/prom/push"
    basic_auth:
      username: <USERNAME>
      password: <PASSWORD>

The convert command takes the YAML file as input and outputs a River file.

static-binary

flow-binary

AGENT_MODE=flow grafana-agent convert --source-format=prometheus --output=<OUTPUT_CONFIG_PATH> <INPUT_CONFIG_PATH>

grafana-agent-flow convert --source-format=prometheus --output=<OUTPUT_CONFIG_PATH> <INPUT_CONFIG_PATH>

Replace the following:

<INPUT_CONFIG_PATH>: The full path to the Prometheus configuration.
<OUTPUT_CONFIG_PATH>: The full path to output the Grafana Agent Flow configuration.

The new Grafana Agent Flow configuration file looks like this:

prometheus.scrape "prometheus" {
  targets = [{
    __address__ = "localhost:12345",
  }]
  forward_to     = [prometheus.remote_write.default.receiver]
  job_name       = "prometheus"
  scrape_timeout = "45s"
}

prometheus.remote_write "default" {
  endpoint {
    name = "grafana-cloud"
    url  = "https://prometheus-us-central1.grafana.net/api/prom/push"

    basic_auth {
      username = "USERNAME"
      password = "PASSWORD"
    }

    queue_config {
      capacity             = 2500
      max_shards           = 200
      max_samples_per_send = 500
    }

    metadata_config {
      max_samples_per_send = 500
    }
  }
}

Limitations

Configuration conversion is done on a best-effort basis. Grafana Agent will issue warnings or errors where the conversion can’t be performed.

After the configuration is converted, review the Grafana Agent Flow configuration file created and verify that it’s correct before starting to use it in a production environment.

The following list is specific to the convert command and not Grafana Agent Flow:

The following configurations aren’t available for conversion to Grafana Agent Flow: rule_files, alerting, remote_read, storage, and tracing. Any additional unsupported features are returned as errors during conversion.
Check if you are using any extra command line arguments with Prometheus that aren’t present in your configuration file. For example, --web.listen-address.
Metamonitoring metrics exposed by Grafana Agent Flow usually match Prometheus metamonitoring metrics but will use a different name. Make sure that you use the new metric names, for example, in your alerts and dashboards queries.
The logs produced by Grafana Agent Flow differ from those produced by Prometheus.
Grafana Agent exposes the Grafana Agent Flow UI.

Migrate from Promtail to Grafana Agent Flow

Wed, 12 Mar 2025 11:45:31 +0100

Migrate from Promtail to Grafana Agent Flow

The built-in Grafana Agent convert command can migrate your Promtail configuration to a Grafana Agent Flow configuration.

This topic describes how to:

Convert a Promtail configuration to a Grafana Agent Flow configuration.
Run a Promtail configuration natively using Grafana Agent Flow.

Components used in this topic

Before you begin

You must have an existing Promtail configuration.
You must be familiar with the concept of Components in Grafana Agent Flow.

Convert a Promtail configuration

To fully migrate from Promtail to Grafana Agent Flow, you must convert your Promtail configuration into a Grafana Agent Flow configuration. This conversion will enable you to take full advantage of the many additional features available in Grafana Agent Flow.

In this task, you will use the convert CLI command to output a Grafana Agent Flow configuration from a Promtail configuration.

Open a terminal window and run the following command.
static-binary

flow-binary
static-binary
AGENT_MODE=flow grafana-agent convert --source-format=promtail --output=<OUTPUT_CONFIG_PATH> <INPUT_CONFIG_PATH>
flow-binary
grafana-agent-flow convert --source-format=promtail --output=<OUTPUT_CONFIG_PATH> <INPUT_CONFIG_PATH>
Replace the following:
- <INPUT_CONFIG_PATH>: The full path to the Promtail configuration.
- <OUTPUT_CONFIG_PATH>: The full path to output the Grafana Agent Flow configuration.
Run Grafana Agent Flow using the new configuration from <OUTPUT_CONFIG_PATH>:

Debugging

If the convert command can’t convert a Promtail configuration, diagnostic information is sent to stderr. You can bypass any non-critical issues and output the Grafana Agent Flow configuration using a best-effort conversion by including the --bypass-errors flag.

Caution
If you bypass the errors, the behavior of the converted configuration may not match the original Promtail configuration. Make sure you fully test the converted configuration before using it in a production environment.
static-binary

flow-binary
static-binary
AGENT_MODE=flow grafana-agent convert --source-format=promtail --bypass-errors --output=<OUTPUT_CONFIG_PATH> <INPUT_CONFIG_PATH>
flow-binary
grafana-agent-flow convert --source-format=promtail --bypass-errors --output=<OUTPUT_CONFIG_PATH> <INPUT_CONFIG_PATH>
Replace the following:
- <INPUT_CONFIG_PATH>: The full path to the Promtail configuration.
- <OUTPUT_CONFIG_PATH>: The full path to output the Grafana Agent Flow configuration.

You can also output a diagnostic report by including the --report flag.

static-binary

flow-binary

AGENT_MODE=flow grafana-agent convert --source-format=promtail --report=<OUTPUT_REPORT_PATH> --output=<OUTPUT_CONFIG_PATH> <INPUT_CONFIG_PATH>

grafana-agent-flow convert --source-format=promtail --report=<OUTPUT_REPORT_PATH> --output=<OUTPUT_CONFIG_PATH> <INPUT_CONFIG_PATH>

Replace the following:

<INPUT_CONFIG_PATH>: The full path to the Promtail configuration.
<OUTPUT_CONFIG_PATH>: The full path to output the Grafana Agent Flow configuration.
<OUTPUT_REPORT_PATH>: The output path for the report.

If you use the example Promtail configuration below, the diagnostic report provides the following information:

(Warning) If you have a tracing set up for Promtail, it cannot be migrated to Grafana Agent Flow automatically. Refer to the documentation on how to configure tracing in Grafana Agent Flow.
(Warning) The metrics from Grafana Agent Flow are different from the metrics emitted by Promtail. If you rely on Promtail's metrics, you must change your configuration, for example, your alerts and dashboards.

Run a Promtail configuration

If you’re not ready to completely switch to a Grafana Agent Flow configuration, you can run Grafana Agent using your existing Promtail configuration. The --config.format=promtail flag tells Grafana Agent to convert your Promtail configuration to Grafana Agent Flow and load it directly without saving the new configuration. This allows you to try Grafana Agent Flow without modifying your existing Promtail configuration infrastructure.

In this task, you will use the run CLI command to run Grafana Agent Flow using a Promtail configuration.

Run Grafana Agent Flow and include the command line flag --config.format=promtail. Your configuration file must be a valid Promtail configuration file rather than a Grafana Agent Flow configuration file.

Debugging

You can follow the convert CLI command debugging instructions to generate a diagnostic report.
Refer to the Grafana Agent Flow Debugging for more information about running Grafana Agent Flow.
If your Promtail configuration can’t be converted and loaded directly into Grafana Agent, diagnostic information is sent to stderr. You can bypass any non-critical issues and start Grafana Agent by including the --config.bypass-conversion-errors flag in addition to --config.format=promtail.

Caution
If you bypass the errors, the behavior of the converted configuration may not match the original Promtail configuration. Do not use this flag in a production environment.

Example

This example demonstrates converting a Promtail configuration file to a Grafana Agent Flow configuration file.

The following Promtail configuration file provides the input for the conversion.

clients:
  - url: http://localhost/loki/api/v1/push
scrape_configs:
  - job_name: example
    static_configs:
      - targets:
          - localhost
        labels:
          __path__: /var/log/*.log

The convert command takes the YAML file as input and outputs a River file.

static-binary

flow-binary

AGENT_MODE=flow grafana-agent convert --source-format=promtail --output=<OUTPUT_CONFIG_PATH> <INPUT_CONFIG_PATH>

grafana-agent-flow convert --source-format=promtail --output=<OUTPUT_CONFIG_PATH> <INPUT_CONFIG_PATH>

Replace the following:

<INPUT_CONFIG_PATH>: The full path to the Promtail configuration.
<OUTPUT_CONFIG_PATH>: The full path to output the Grafana Agent Flow configuration.

The new Grafana Agent Flow configuration file looks like this:

local.file_match "example" {
	path_targets = [{
		__address__ = "localhost",
		__path__    = "/var/log/*.log",
	}]
}

loki.source.file "example" {
	targets    = local.file_match.example.targets
	forward_to = [loki.write.default.receiver]
}

loki.write "default" {
	endpoint {
		url = "http://localhost/loki/api/v1/push"
	}
	external_labels = {}
}

Limitations

Configuration conversion is done on a best-effort basis. Grafana Agent will issue warnings or errors where the conversion can’t be performed.

After the configuration is converted, review the Grafana Agent Flow configuration file created and verify that it’s correct before starting to use it in a production environment.

The following list is specific to the convert command and not Grafana Agent Flow:

Check if you are using any extra command line arguments with Promtail that aren’t present in your configuration file. For example, -max-line-size.
Check if you are setting any environment variables, whether expanded in the configuration file itself or consumed directly by Promtail, such as JAEGER_AGENT_HOST.
In Grafana Agent Flow, the positions file is saved at a different location. Refer to the loki.source.file documentation for more details. Check if you have any existing setup, for example, a Kubernetes Persistent Volume, that you must update to use the new positions file path.
Metamonitoring metrics exposed by Grafana Agent Flow usually match Promtail metamonitoring metrics but will use a different name. Make sure that you use the new metric names, for example, in your alerts and dashboards queries.
The logs produced by Grafana Agent Flow will differ from those produced by Promtail.
Grafana Agent Flow exposes the Grafana Agent Flow UI, which differs from Promtail’s Web UI.

Migrate Grafana Agent Static to Grafana Agent Flow

Wed, 11 Sep 2024 17:43:35 +0000

Migrate from Grafana Agent Static to Grafana Agent Flow

The built-in Grafana Agent convert command can migrate your Static configuration to a Grafana Agent Flow configuration.

This topic describes how to:

Convert a Grafana Agent Static configuration to a Grafana Agent Flow configuration.
Run a Grafana Agent Static configuration natively using Grafana Agent Flow.

Components used in this topic

Before you begin

You must have an existing Grafana Agent Static configuration.
You must be familiar with the Components concept in Grafana Agent Flow.

Convert a Grafana Agent Static configuration

To fully migrate Grafana Agent Static to Grafana Agent Flow, you must convert your Static configuration into a Grafana Agent Flow configuration. This conversion will enable you to take full advantage of the many additional features available in Grafana Agent Flow.

In this task, you will use the convert CLI command to output a Grafana Agent Flow configuration from a Static configuration.

Open a terminal window and run the following command.
static-binary

flow-binary
static-binary
AGENT_MODE=flow grafana-agent convert --source-format=static --output=<OUTPUT_CONFIG_PATH> <INPUT_CONFIG_PATH>
flow-binary
grafana-agent-flow convert --source-format=static --output=<OUTPUT_CONFIG_PATH> <INPUT_CONFIG_PATH>
Replace the following:
- <INPUT_CONFIG_PATH>: The full path to the Static configuration.
- _<OUTPUT_CONFIG_PATH>_: The full path to output the Grafana Agent Flow configuration.
Run Grafana Agent Flow using the new Grafana Agent Flow configuration from <OUTPUT_CONFIG_PATH>:

Debugging

If the convert command can’t convert a Static configuration, diagnostic information is sent to stderr. You can use the --bypass-errors flag to bypass any non-critical issues and output the Grafana Agent Flow configuration using a best-effort conversion.

Caution
If you bypass the errors, the behavior of the converted configuration may not match the original Grafana Agent Static configuration. Make sure you fully test the converted configuration before using it in a production environment.
static-binary

flow-binary
static-binary
AGENT_MODE=flow grafana-agent convert --source-format=static --bypass-errors --output=<OUTPUT_CONFIG_PATH> <INPUT_CONFIG_PATH>
flow-binary
grafana-agent-flow convert --source-format=static --bypass-errors --output=<OUTPUT_CONFIG_PATH> <INPUT_CONFIG_PATH>
Replace the following:
- <INPUT_CONFIG_PATH>: The full path to the Static configuration.
- <OUTPUT_CONFIG_PATH>: The full path to output the Grafana Agent Flow configuration.
You can use the --report flag to output a diagnostic report.
static-binary

flow-binary
static-binary
AGENT_MODE=flow grafana-agent convert --source-format=static --report=<OUTPUT_REPORT_PATH> --output=<OUTPUT_CONFIG_PATH> <INPUT_CONFIG_PATH>
flow-binary
grafana-agent-flow convert --source-format=static --report=<OUTPUT_REPORT_PATH> --output=<OUTPUT_CONFIG_PATH> <INPUT_CONFIG_PATH> ```
Replace the following:
- <INPUT_CONFIG_PATH>: The full path to the Static configuration.
- <OUTPUT_CONFIG_PATH>: The full path to output the Grafana Agent Flow configuration.
- <OUTPUT_REPORT_PATH>: The output path for the report.
Using the example Grafana Agent Static configuration below, the diagnostic report provides the following information.
plaintext
```
(Warning) Please review your agent command line flags and ensure they are set in your Grafana Agent Flow configuration file where necessary.
```

Run a Static mode configuration

If you’re not ready to completely switch to a Grafana Agent Flow configuration, you can run Grafana Agent using your existing Grafana Agent Static configuration. The --config.format=static flag tells Grafana Agent to convert your Static configuration to Grafana Agent Flow and load it directly without saving the new configuration. This allows you to try Grafana Agent Flow without modifying your existing Grafana Agent Static configuration infrastructure.

In this task, you will use the run CLI command to run Grafana Agent Flow using a Static configuration.

Run Grafana Agent Flow and include the command line flag --config.format=static. Your configuration file must be a valid Static configuration file.

Debugging

You can follow the convert CLI command debugging instructions to generate a diagnostic report.
Refer to the Grafana Agent Flow debugging UI for more information about running Grafana Agent Flow.
If your Static configuration can’t be converted and loaded directly into Grafana Agent Flow, diagnostic information is sent to stderr. You can use the --config.bypass-conversion-errors flag with --config.format=static to bypass any non-critical issues and start Grafana Agent Flow.

Caution
If you bypass the errors, the behavior of the converted configuration may not match the original Grafana Agent Static configuration. Do not use this flag in a production environment.

Example

This example demonstrates converting a Static configuration file to a Grafana Agent Flow configuration file.

The following Static configuration file provides the input for the conversion.

server:
  log_level: info

metrics:
  global:
    scrape_interval: 15s
    remote_write:
      - url: https://prometheus-us-central1.grafana.net/api/prom/push
        basic_auth:
          username: USERNAME
          password: PASSWORD
  configs:
    - name: test
      host_filter: false
      scrape_configs:
        - job_name: local-agent
          static_configs:
            - targets: ['127.0.0.1:12345']
              labels:
                cluster: 'localhost'

logs:
  global:
    file_watch_config:
      min_poll_frequency: 1s
      max_poll_frequency: 5s
  positions_directory: /var/lib/agent/data-agent
  configs:
    - name: varlogs
      scrape_configs:
        - job_name: varlogs
          static_configs:
            - targets:
              - localhost
              labels:
                job: varlogs
                host: mylocalhost
                __path__: /var/log/*.log
          pipeline_stages:
            - match:
                selector: '{filename="/var/log/*.log"}'
                stages:
                - drop:
                    expression: '^[^0-9]{4}'
                - regex:
                    expression: '^(?P<timestamp>\d{4}/\d{2}/\d{2} \d{2}:\d{2}:\d{2}) \[(?P<level>[[:alpha:]]+)\] (?:\d+)\#(?:\d+): \*(?:\d+) (?P<message>.+)$'
                - pack:
                    labels:
                      - level
      clients:
        - url: https://USER_ID:API_KEY@logs-prod3.grafana.net/loki/api/v1/push

The convert command takes the YAML file as input and outputs a River file.

static-binary

flow-binary

AGENT_MODE=flow grafana-agent convert --source-format=static --output=<OUTPUT_CONFIG_PATH> <INPUT_CONFIG_PATH>

grafana-agent-flow convert --source-format=static --output=<OUTPUT_CONFIG_PATH> <INPUT_CONFIG_PATH>

Replace the following:

<INPUT_CONFIG_PATH>: The full path to the Static configuration.
<OUTPUT_CONFIG_PATH>: The full path to output the Grafana Agent Flow configuration.

The new Grafana Agent Flow configuration file looks like this:

prometheus.scrape "metrics_test_local_agent" {
	targets = [{
		__address__ = "127.0.0.1:12345",
		cluster     = "localhost",
	}]
	forward_to      = [prometheus.remote_write.metrics_test.receiver]
	job_name        = "local-agent"
	scrape_interval = "15s"
}

prometheus.remote_write "metrics_test" {
	endpoint {
		name = "test-3a2a1b"
		url  = "https://prometheus-us-central1.grafana.net/api/prom/push"

		basic_auth {
			username = "<USERNAME>"
			password = "<PASSWORD>"
		}

		queue_config { }

		metadata_config { }
	}
}

local.file_match "logs_varlogs_varlogs" {
	path_targets = [{
		__address__ = "localhost",
		__path__    = "/var/log/*.log",
		host        = "mylocalhost",
		job         = "varlogs",
	}]
}

loki.process "logs_varlogs_varlogs" {
	forward_to = [loki.write.logs_varlogs.receiver]

	stage.match {
		selector = "{filename=\"/var/log/*.log\"}"

		stage.drop {
			expression = "^[^0-9]{4}"
		}

		stage.regex {
			expression = "^(?P<timestamp>\\d{4}/\\d{2}/\\d{2} \\d{2}:\\d{2}:\\d{2}) \\[(?P<level>[[:alpha:]]+)\\] (?:\\d+)\\#(?:\\d+): \\*(?:\\d+) (?P<message>.+)$"
		}

		stage.pack {
			labels           = ["level"]
			ingest_timestamp = false
		}
	}
}

loki.source.file "logs_varlogs_varlogs" {
	targets    = local.file_match.logs_varlogs_varlogs.targets
	forward_to = [loki.process.logs_varlogs_varlogs.receiver]

	file_watch {
		min_poll_frequency = "1s"
		max_poll_frequency = "5s"
	}
}

loki.write "logs_varlogs" {
	endpoint {
		url = "https://USER_ID:API_KEY@logs-prod3.grafana.net/loki/api/v1/push"
	}
	external_labels = {}
}

Integrations Next

You can convert [integrations next][] configurations by adding the extra-args flag for convert or config.extra-args for run.

static-binary

flow-binary

AGENT_MODE=flow grafana-agent convert --source-format=static --extra-args="-enable-features=integrations-next" --output=<OUTPUT_CONFIG_PATH> <INPUT_CONFIG_PATH>

grafana-agent-flow convert --source-format=static --extra-args="-enable-features=integrations-next" --output=<OUTPUT_CONFIG_PATH> <INPUT_CONFIG_PATH>

Replace the following:

<INPUT_CONFIG_PATH>: The full path to the Static configuration.
<OUTPUT_CONFIG_PATH>: The full path to output the Grafana Agent Flow configuration.

Environment Vars

You can use the -config.expand-env command line flag to interpret environment variables in your Grafana Agent Static configuration. You can pass these flags to convert with --extra-args="-config.expand-env" or to run with --config.extra-args="-config.expand-env".

It’s possible to combine integrations-next with expand-env. For convert, you can use --extra-args="-enable-features=integrations-next -config.expand-env"

Limitations

Configuration conversion is done on a best-effort basis. Grafana Agent will issue warnings or errors where the conversion can’t be performed.

After the configuration is converted, review the Grafana Agent Flow configuration file and verify that it’s correct before starting to use it in a production environment.

The following list is specific to the convert command and not Grafana Agent Flow:

The Agent Management configuration options can’t be automatically converted to Grafana Agent Flow. Any additional unsupported features are returned as errors during conversion.
There is no gRPC server to configure for Grafana Agent Flow, as any non-default configuration will show as unsupported during the conversion.
Check if you are using any extra command line arguments with Static that aren’t present in your configuration file. For example, -server.http.address.
Check if you are using any environment variables in your Static configuration. These will be evaluated during conversion and you may want to replace them with the Grafana Agent Flow Standard library env function after conversion.
Review additional Prometheus Limitations for limitations specific to your Metrics configuration.
Review additional Promtail Limitations for limitations specific to your Logs configuration.
The logs produced by Grafana Agent Flow mode will differ from those produced by Static.
Grafana Agent exposes the Grafana Agent Flow UI.

Migrate to Grafana Agent Flow on Grafana Labs

Migrate from Grafana Agent Operator to Grafana Agent Flow

Migrate from Grafana Agent Operator to Grafana Agent Flow

Deploy Grafana Agent Flow with Helm

Convert MetricsIntances to Grafana Agent Flow components

Collecting Logs

Integrations

Migrate from OpenTelemetry Collector to Grafana Agent Flow

Migrate from OpenTelemetry Collector to Grafana Agent Flow

Components used in this topic

Before you begin

Convert an OpenTelemetry Collector configuration

Debugging

Run an OpenTelemetry Collector configuration

Debugging

Example

Limitations

Migrate from Prometheus to Grafana Agent Flow

Migrate from Prometheus to Grafana Agent Flow

Components used in this topic

Before you begin

Convert a Prometheus configuration

Debugging

Run a Prometheus configuration

Debugging

Example

Limitations

Migrate from Promtail to Grafana Agent Flow

Migrate from Promtail to Grafana Agent Flow

Components used in this topic

Before you begin

Convert a Promtail configuration

Debugging

Run a Promtail configuration

Debugging

Example

Limitations

Migrate Grafana Agent Static to Grafana Agent Flow

Migrate from Grafana Agent Static to Grafana Agent Flow

Components used in this topic

Before you begin

Convert a Grafana Agent Static configuration

Debugging

Run a Static mode configuration

Debugging

Example

Integrations Next

Environment Vars

Limitations

Convert `MetricsIntances` to Grafana Agent Flow components