The Grafana Agent command-line interface on Grafana Labs

The convert command

Mon, 24 Feb 2025 10:20:37 +0100

The convert command

The convert command converts a supported configuration format to Grafana Agent Flow River format.

Caution
This command has no backward compatibility guarantees and may change or be removed between releases.

Usage

Usage:

AGENT_MODE=flow grafana-agent convert [<FLAG> ...] <FILE_NAME>
grafana-agent-flow convert [<FLAG> ...] <FILE_NAME>

Replace the following:
- <FLAG>: One or more flags that define the input and output of the command.
- <FILE_NAME>: The Grafana Agent configuration file.

If the FILE_NAME argument isn’t provided or if the FILE_NAME argument is equal to -, convert converts the contents of standard input. Otherwise, convert reads and converts the file from disk specified by the argument.

There are several different flags available for the convert command. You can use the --output flag to write the contents of the converted configuration to a specified path. You can use the --report flag to generate a diagnostic report. The --bypass-errors flag allows you to bypass any errors generated during the file conversion.

The command fails if the source configuration has syntactically incorrect configuration or can’t be converted to Grafana Agent Flow River format.

The following flags are supported:

--output, -o: The filepath and filename where the output is written.
--report, -r: The filepath and filename where the report is written.
--source-format, -f: Required. The format of the source file. Supported formats: otelcol, prometheus, promtail, static.
--bypass-errors, -b: Enable bypassing errors when converting.
--extra-args, e: Extra arguments from the original format used by the converter.

Defaults

Grafana Agent Flow defaults are managed as follows:

If a provided source configuration value matches a Grafana Agent Flow default value, the property is left off the output.
If a non-provided source configuration value default matches a Grafana Agent Flow default value, the property is left off the output.
If a non-provided source configuration value default doesn’t match a Grafana Agent Flow default value, the default value is included in the output.

Errors

Errors are defined as non-critical issues identified during the conversion where an output can still be generated. These can be bypassed using the --bypass-errors flag.

OpenTelemetry Collector

You can use the --source-format=otelcol to convert the source configuration from an OpenTelemetry Collector to a Grafana Agent Flow configuration.

Many OpenTelemetry Collector components are supported. Review the otelcol.* component information in the Component Reference for more information about otelcol components that you can convert. If a source configuration has unsupported features, you will receive errors when you convert it to a flow configuration. The converter raises warnings for configuration options that may require your attention.

Refer to Migrate from OpenTelemetry Collector to Grafana Agent Flow for a detailed migration guide.

Prometheus

Using the --source-format=prometheus will convert the source configuration from Prometheus v2.45 to Grafana Agent Flow configuration.

This includes Prometheus features such as scrape_config, relabel_config, metric_relabel_configs, remote_write, and many supported *_sd_configs. Unsupported features in a source configuration result in errors.

Refer to Migrate from Prometheus to Grafana Agent Flow for a detailed migration guide.

Promtail

Using the --source-format=promtail will convert the source configuration from Promtail v2.8.x to Grafana Agent Flow configuration.

Nearly all Promtail features are supported and can be converted to Grafana Agent Flow configuration.

If you have unsupported features in a source configuration, you will receive errors when you convert to a flow configuration. The converter will also raise warnings for configuration options that may require your attention.

Refer to Migrate from Promtail to Grafana Agent Flow for a detailed migration guide.

Static

Using the --source-format=static will convert the source configuration from a Grafana Agent Static configuration to a Grafana Agent Flow configuration.

Include --extra-args for passing additional command line flags from the original format. For example, --extra-args="-enable-features=integrations-next" will convert a Grafana Agent Static integrations-next configuration to a Grafana Agent Flow configuration. You can also expand environment variables with --extra-args="-config.expand-env". You can combine multiple command line flags with a space between each flag, for example --extra-args="-enable-features=integrations-next -config.expand-env".

If you have unsupported features in a Static mode source configuration, you will receive errors when you convert to a Flow mode configuration. The converter will also raise warnings for configuration options that may require your attention.

Refer to Migrate from Grafana Agent Static to Grafana Agent Flow for a detailed migration guide.

The fmt command

Wed, 11 Sep 2024 17:43:35 +0000

The fmt command

The fmt command formats a given Grafana Agent Flow configuration file.

Usage

Usage:

AGENT_MODE=flow grafana-agent fmt [FLAG ...] FILE_NAME
grafana-agent-flow fmt [FLAG ...] FILE_NAME

Replace the following:
- FLAG: One or more flags that define the input and output of the command.
- FILE_NAME: The Grafana Agent Flow configuration file.

If the FILE_NAME argument is not provided or if the FILE_NAME argument is equal to -, fmt formats the contents of standard input. Otherwise, fmt reads and formats the file from disk specified by the argument.

The --write flag can be specified to replace the contents of the original file on disk with the formatted results. --write can only be provided when fmt is not reading from standard input.

The command fails if the file being formatted has syntactically incorrect River configuration, but does not validate whether Flow components are configured properly.

The following flags are supported:

--write, -w: Write the formatted file back to disk when not reading from standard input.

The run command

Wed, 12 Mar 2025 11:45:31 +0100

The run command

The run command runs Grafana Agent Flow in the foreground until an interrupt is received.

Usage

Usage:

AGENT_MODE=flow grafana-agent run [FLAG ...] PATH_NAME
grafana-agent-flow run [FLAG ...] PATH_NAME

Replace the following:
- FLAG: One or more flags that define the input and output of the command.
- PATH_NAME: Required. The Grafana Agent Flow configuration file/directory path.

If the PATH_NAME argument is not provided, or if the configuration path can’t be loaded or contains errors during the initial load, the run command will immediately exit and show an error message.

If you give the PATH_NAME argument a directory path, Grafana Agent Flow will find *.river files (ignoring nested directories) and load them as a single configuration source. However, component names must be unique across all River files, and configuration blocks must not be repeated.

Grafana Agent Flow will continue to run if subsequent reloads of the configuration file fail, potentially marking components as unhealthy depending on the nature of the failure. When this happens, Grafana Agent Flow will continue functioning in the last valid state.

run launches an HTTP server that exposes metrics about itself and its components. The HTTP server is also exposes a UI at / for debugging running components.

The following flags are supported:

--server.http.enable-pprof: Enable /debug/pprof profiling endpoints. (default true)
--server.http.memory-addr: Address to listen for in-memory HTTP traffic on (default agent.internal:12345).
--server.http.listen-addr: Address to listen for HTTP traffic on (default 127.0.0.1:12345).
--server.http.ui-path-prefix: Base path where the UI is exposed (default /).
--storage.path: Base directory where components can store data (default data-agent/).
--disable-reporting: Disable data collection (default false).
--cluster.enabled: Start Grafana Agent Flow in clustered mode (default false).
--cluster.node-name: The name to use for this node (defaults to the environment’s hostname).
--cluster.join-addresses: Comma-separated list of addresses to join the cluster at (default ""). Mutually exclusive with --cluster.discover-peers.
--cluster.discover-peers: List of key-value tuples for discovering peers (default ""). Mutually exclusive with --cluster.join-addresses.
--cluster.rejoin-interval: How often to rejoin the list of peers (default "60s").
--cluster.advertise-address: Address to advertise to other cluster nodes (default "").
--cluster.advertise-interfaces: List of interfaces used to infer an address to advertise. Set to all to use all available network interfaces on the system. (default "eth0,en0").
--cluster.max-join-peers: Number of peers to join from the discovered set (default 5).
--cluster.name: Name to prevent nodes without this identifier from joining the cluster (default "").
--config.format: The format of the source file. Supported formats: flow, otelcol, prometheus, promtail, static (default "flow").
--config.bypass-conversion-errors: Enable bypassing errors when converting (default false).
--config.extra-args: Extra arguments from the original format used by the converter.

Update the configuration file

The configuration file can be reloaded from disk by either:

Sending an HTTP POST request to the /-/reload endpoint.
Sending a SIGHUP signal to the Grafana Agent Flow process.

When this happens, the component controller synchronizes the set of running components with the latest set of components specified in the configuration file. Components that are no longer defined in the configuration file after reloading are shut down, and components that have been added to the configuration file since the previous reload are created.

All components managed by the component controller are reevaluated after reloading.

Clustering

The --cluster.enabled command-line argument starts Grafana Agent in clustering mode. The rest of the --cluster.* command-line flags can be used to configure how nodes discover and connect to one another.

Each cluster member’s name must be unique within the cluster. Nodes which try to join with a conflicting name are rejected and will fall back to bootstrapping a new cluster of their own.

Peers communicate over HTTP/2 on the built-in HTTP server. Each node must be configured to accept connections on --server.http.listen-addr and the address defined or inferred in --cluster.advertise-address.

If the --cluster.advertise-address flag isn’t explicitly set, Grafana Agent Flow tries to infer a suitable one from --cluster.advertise-interfaces. If --cluster.advertise-interfaces isn’t explicitly set, Grafana Agent Flow will infer one from the eth0 and en0 local network interfaces. Grafana Agent Flow will fail to start if it can’t determine the advertised address. Since Windows doesn’t use the interface names eth0 or en0, Windows users must explicitly pass at least one valid network interface for --cluster.advertise-interfaces or a value for --cluster.advertise-address.

The comma-separated list of addresses provided in --cluster.join-addresses can either be IP addresses with an optional port, or DNS records to lookup. The ports on the list of addresses default to the port used for the HTTP listener if not explicitly provided. We recommend that you align the port numbers on as many nodes as possible to simplify the deployment process.

The --cluster.discover-peers command-line flag expects a list of tuples in the form of provider=XXX key=val key=val .... Clustering uses the go-discover package to discover peers and fetch their IP addresses, based on the chosen provider and the filtering key-values it supports. Clustering supports the default set of providers available in go-discover and registers the k8s provider on top.

If either the key or the value in a tuple pair contains a space, a backslash, or double quotes, then it must be quoted with double quotes. Within this quoted string, the backslash can be used to escape double quotes or the backslash itself.

The --cluster.rejoin-interval flag defines how often each node should rediscover peers based on the contents of the --cluster.join-addresses and --cluster.discover-peers flags and try to rejoin them. This operation is useful for addressing split-brain issues if the initial bootstrap is unsuccessful and for making clustering easier to manage in dynamic environments. To disable this behavior, set the --cluster.rejoin-interval flag to "0s".

Discovering peers using the --cluster.join-addresses and --cluster.discover-peers flags only happens on startup; after that, cluster nodes depend on gossiping messages with each other to converge on the cluster’s state.

The first node that is used to bootstrap a new cluster (also known as the “seed node”) can either omit the flags that specify peers to join or can try to connect to itself.

To join or rejoin a cluster, Grafana Agent Flow will try to connect to a certain number of peers limited by the --cluster.max-join-peers flag. This flag can be useful for clusters of significant sizes because connecting to a high number of peers can be an expensive operation. To disable this behavior, set the --cluster.max-join-peers flag to 0. If the value of --cluster.max-join-peers is higher than the number of peers discovered, Grafana Agent Flow will connect to all of them.

The --cluster.name flag can be used to prevent clusters from accidentally merging. When --cluster.name is provided, nodes will only join peers who share the same cluster name value. By default, the cluster name is empty, and any node that doesn’t set the flag can join. Attempting to join a cluster with a wrong --cluster.name will result in a “failed to join memberlist” error.

Clustering states

Clustered Grafana Agents are in one of three states:

Viewer: Grafana Agent Flow has a read-only view of the cluster and isn’t participating in workload distribution.
Participant: Grafana Agent Flow is participating in workload distribution for components that have clustering enabled.
Terminating: Grafana Agent Flow is shutting down and will no longer assign new work to itself.

Each Grafana Agent initially joins the cluster in the viewer state and then transitions to the participant state after the process startup completes. Each Grafana Agent then transitions to the terminating state when shutting down.

The current state of a clustered Grafana Agent is shown on the clustering page in the UI.

Configuration conversion (beta)

When you use the --config.format command-line argument with a value other than flow, Grafana Agent converts the configuration file from the source format to River and immediately starts running with the new configuration. This conversion uses the converter API described in the grafana-agent-flow convert docs.

If you include the --config.bypass-conversion-errors command-line argument, Grafana Agent Flow will ignore any errors from the converter. Use this argument with caution because the resulting conversion may not be equivalent to the original configuration.

Include --config.extra-args to pass additional command line flags from the original format to the converter. Refer to grafana-agent-flow convert for more details on how extra-args work.

The tools command

Wed, 11 Sep 2024 17:43:35 +0000

The tools command

The tools command contains command line tooling grouped by Flow component.

Caution
Utilities in this command have no backward compatibility guarantees and may change or be removed between releases.

Subcommands

prometheus.remote_write sample-stats

Usage:

AGENT_MODE=flow grafana-agent tools prometheus.remote_write sample-stats [FLAG ...] WAL_DIRECTORY
grafana-agent-flow tools prometheus.remote_write sample-stats [FLAG ...] WAL_DIRECTORY

The sample-stats command reads the Write-Ahead Log (WAL) specified by WAL_DIRECTORY and collects information on metric samples within it.

For each metric discovered, sample-stats emits:

The timestamp of the oldest sample received for that metric.
The timestamp of the newest sample received for that metric.
The total number of samples discovered for that metric.

By default, sample-stats will return information for every metric in the WAL. You can pass the --selector flag to filter the reported metrics to a smaller set.

The following flag is supported:

--selector: A PromQL label selector to filter data by. (default {})

prometheus.remote_write target-stats

Usage:

AGENT_MODE=flow grafana-agent tools prometheus.remote_write target-stats --job JOB --instance INSTANCE WAL_DIRECTORY
grafana-agent-flow tools prometheus.remote_write target-stats --job JOB --instance INSTANCE WAL_DIRECTORY

The target-stats command reads the Write-Ahead Log (WAL) specified by WAL_DIRECTORY and collects metric cardinality information for a specific target.

For the target specified by the --job and --instance flags, unique metric names for that target are printed along with the number of series with that metric name.

The following flags are supported:

--job: The job label of the target.
--instance: The instance label of the target.

The --job and --instance labels are required.

prometheus.remote_write wal-stats

Usage:

AGENT_MODE=flow grafana-agent tools prometheus.remote_write wal-stats WAL_DIRECTORY
grafana-agent-flow tools prometheus.remote_write wal-stats WAL_DIRECTORY

The wal-stats command reads the Write-Ahead Log (WAL) specified by WAL_DIRECTORY and collects general information about it.

The following information is reported:

The timestamp of the oldest sample in the WAL.
The timestamp of the newest sample in the WAL.
The total number of unique series defined in the WAL.
The total number of samples in the WAL.
The number of hash collisions detected, if any.
The total number of invalid records in the WAL, if any.
The most recent WAL checkpoint segment number.
The oldest segment number in the WAL.
The newest segment number in the WAL.

Additionally, wal-stats reports per-target information, where a target is defined as a unique combination of the job and instance label values. For each target, wal-stats reports the number of series and the number of metric samples associated with that target.

The wal-stats command does not support any flags.