Grafana Agent v0.37: Feature parity between Static and Flow mode, easy migration configs, and more

Grafana Agent v0.37 is here!

This new release brings a lot of exciting new features and marks the pinnacle of a year-long effort to achieve feature parity between Grafana Agent Flow mode and Grafana Agent Static mode. We also extended our config converter to ease the migration from Static to Flow mode and we added the possibility to split your Flow configuration into multiple files.

Please make note of some breaking changes in this release.

What is Grafana Agent Flow?

Grafana Agent Flow is a component-based revision of Grafana Agent with a focus on ease-of-use, debuggability, and ability to adapt to the needs of power users. It contains a custom configuration language called River to dynamically configure and connect components. The data then flows through the components forming pipelines. Flow will soon be the future of Grafana Agent, replacing Static as the default mode.

For more information, have a look at our Introducing Grafana Agent Flow blog post. We’ve also prepared a number of Grafana Agent Flow tutorials as well as extensive Grafana Agent Flow documentation. There is also a config language reference for River to help you get started.

Grafana Agent Static mode and Grafana Agent Flow mode: feature parity

In recent releases, we added a lot of components to Flow mode, aiming for feature parity with Static mode. We are happy to announce that with this release, you can now use Flow components for all Static mode use cases. This change comes with plenty of new components:

• discovery.consulagent discovers scrape targets from Consul Agent
• discovery.kuma discovers scrape targets from the Kuma control plane
• discovery.linode discovers scrape targets from the Linode API
• discovery.marathon discovers scrape targets from Marathon servers
• discovery.ionos discovers scrape targets from the IONOS Cloud API
• discovery.triton discovers scrape targets from Triton Container Monitor
• discovery.nerve discovers scrape targets from AirBnB’s Nerve
• discovery.serverset discovers Serversets stored in Zookeeper
• discovery.scaleway discovers scrape targets from Scaleway virtual instances and bare metal machines
• discovery.dockerswarm discovers scrape targets from Docker Swarm
• faro.receiver accepts Grafana Faro-formatted telemetry data over the network and forwards it to other components
• otelcol.connector.servicegraph creates service graph metrics from spans. It is the flow mode equivalent to static mode’s service_graphs processor
• otelcol.connector.spanlogs creates logs from spans. It is the flow mode equivalent to static mode’s automatic_logging processor
• otelcol.processor.k8sattributes adds Kubernetes metadata as resource attributes to spans, logs, and metrics
• otelcol.processor.probabilistic_sampler samples logs and traces based on configuration options
• otelcol.processor.transform transforms OTLP telemetry data using the OpenTelemetry Transformation Language (OTTL). It is most commonly used for transformations on attributes
• remote.kubernetes.configmap loads a configmap’s data for use in other components
• remote.kubernetes.secret loads a secret’s data for use in other components
• prometheus.exporter.agent exposes the agent’s internal metrics
• prometheus.exporter.azure collects metrics from Azure
• prometheus.exporter.cadvisor exposes cAdvisor metrics
• prometheus.exporter.vsphere exposes vmware vsphere metrics

Migrate from Static to Flow mode with one command

In the previous releases, we launched a config converter to assist users in transitioning from Prometheus and Promtail to Flow. With this release, we’ve enhanced the converter, now allowing you to easily transform your Static config into a Flow config.

We wrote user guides to help you through the migration process to Grafana Agent Flow:

• Migrate from Prometheus
• Migrate from Promtail
• Migrate from Static mode

Let’s take a look at an example where we start with a static.yaml config and generate an equivalent flow.river config:

bash Copy

grafana-agent convert --source-format=static --output=flow.river static.yaml

Given this existing static configuration:

yaml Copy

metrics:
  global:
    scrape_interval: 15s
    remote_write:
      - url: https://prometheus-us-central1.grafana.net/api/prom/push
        basic_auth:
          username: USERNAME
          password: PASSWORD
  configs:
    - name: test
      host_filter: false
      scrape_configs:
        - job_name: local-agent
          static_configs:
            - targets: ['127.0.0.1:12345']
              labels:
                cluster: 'localhost'

logs:
  positions_directory: /var/lib/agent/data-agent
  configs:
    - name: varlogs
      scrape_configs:
        - job_name: varlogs
          static_configs:
            - targets:
              - localhost
              labels:
                job: varlogs
                host: mylocalhost
                __path__: /var/log/*.log

integrations:
  mongodb_exporter:
    enabled: true
    mongodb_uri: mongodb://mongodb-a:27017
    relabel_configs:
    - source_labels: [__address__]
      target_label: mongodb_cluster
      replacement: 'prod-cluster'

The grafana-agent convert command would result in this Flow mode config:

river Copy

prometheus.scrape "metrics_test_local_agent" {
	targets = [{
		__address__ = "127.0.0.1:12345",
		cluster     = "localhost",
	}]
	forward_to      = [prometheus.remote_write.metrics_test.receiver]
	job_name        = "local-agent"
	scrape_interval = "15s"
}

prometheus.remote_write "metrics_test" {
	endpoint {
		name = "test-3a2a1b"
		url  = "https://prometheus-us-central1.grafana.net/api/prom/push"

		basic_auth {
			username = "USERNAME"
			password = "PASSWORD"
		}

		queue_config { }

		metadata_config { }
	}
}

local.file_match "logs_varlogs_varlogs" {
	path_targets = [{
		__address__ = "localhost",
		__path__    = "/var/log/*.log",
		host        = "mylocalhost",
		job         = "varlogs",
	}]
}

loki.source.file "logs_varlogs_varlogs" {
	targets    = local.file_match.logs_varlogs_varlogs.targets
	forward_to = []
}

prometheus.exporter.mongodb "integrations_mongodb_exporter" {
	mongodb_uri    = "mongodb://mongodb-a:27017"
	direct_connect = true
}

discovery.relabel "integrations_mongodb_exporter" {
	targets = prometheus.exporter.mongodb.integrations_mongodb_exporter.targets

	rule {
		source_labels = ["__address__"]
		target_label  = "mongodb_cluster"
		replacement   = "prod-cluster"
	}
}

prometheus.scrape "integrations_mongodb_exporter" {
	targets         = discovery.relabel.integrations_mongodb_exporter.output
	forward_to      = [prometheus.remote_write.integrations.receiver]
	job_name        = "integrations/mongodb_exporter"
	scrape_interval = "15s"
}

prometheus.remote_write "integrations" {
	endpoint {
		name = "test-3a2a1b"
		url  = "https://prometheus-us-central1.grafana.net/api/prom/push"

		basic_auth {
			username = "USERNAME"
			password = "PASSWORD"
		}

		queue_config { }

		metadata_config { }
	}
}

If you’re not ready to switch over to Flow config, you can also run the Static config without having to save it as a Flow config. This allows you to try Flow mode without having to modify your existing Static configuration infrastructure.

Configuration conversion is done on a best effort basis. The Agent will issue warnings or errors where the conversion cannot be performed. Once the configuration is converted, we recommend that you review the Flow mode configuration file and verify that it is correct before you start using it in a production environment.

Some options are not yet available for conversion to Flow mode: Integrations next, Traces, and Agent Management.

We’re eager to hear about your experience with our converter. Feel free to share your feedback on our Grafana Labs Community Slack.

Run Grafana Agent in Flow mode with multiple config files

Maintaining lengthy config files can be challenging. Scrolling through them is tedious and handling Git conflicts is painful. In Grafana Agent v0.33, we introduced modules as a way to create reusable pipelines.

To provide an additional degree of separation, in this release, we added the capability to split your Flow config into several files. Now, you can use the run command with a directory path as its argument. The Agent will combine all *.river files from the specified directory into a single unit (subdirectories are not recursively searched for further merging).

Using a directory path to run an agent is just as straightforward as using a single river file path:

bash Copy

grafana-agent run path/to/directory

Many thanks to community member Hainenber for contributing this new feature!

Learn more about Grafana Agent

To learn more about other features in the latest release, please refer to our Grafana Agent documentation or check out our Grafana Agent upgrade guide.

As always, we’d love to hear from you, so feel free to drop by our Grafana Labs Community Slack or check out the Grafana Agent repo directly. We look forward to your comments and feedback!

If you’re not already using Grafana Cloud — the easiest way to get started with observability — sign up now for a free 14-day trial of Grafana Cloud Pro, with unlimited metrics, logs, traces, and users, long-term retention, and access to all Enterprise plugins.