Configure Grafana Agent on Grafana Labs

Create a config file

Sat, 11 Apr 2026 09:32:03 +0000

Create a configuration file

The Grafana Agent supports configuring multiple independent “subsystems.” Each subsystem helps you collect data for a specific type of telemetry.

The Metrics subsystem allows you collect metrics to send to Prometheus.
The Logs subsystem allows you to collect logs to send to Grafana Loki.
The Traces subsystem allows you to collect spans to send to Grafana Tempo.
The Integrations subsystem allows you to collect metrics for common applications, such as MySQL.

Integrations are recommended for first-time users of observability platforms, especially newcomers to Prometheus. Users with more experience with Prometheus or users that already have an existing Prometheus config file can configure the Prometheus subsystem manually.

Integrations

Integrations are individual features that collect metrics for you. For example, the agent integration collects metrics from that running instance of the Grafana Agent. The node_exporter integration will collect metrics from the Linux machine that the Grafana Agent is running on.

metrics:
  wal_directory: /tmp/wal
  global:
    remote_write:
      - url: http://localhost:9009/api/prom/push

integrations:
  agent:
    enabled: true

In this example, we first must configure the wal_directory which is used to store metrics in a Write-Ahead Log. This is required, but ensures that samples will be resent in case of failure (e.g., network issues, machine reboot). We also configure remote_write, which is where all metrics should be sent by default.

Then, the individual integrations are configured. In this example, just the agent integration is enabled. Finally, prometheus_remote_write is configured with a location to send metrics. You will have to replace this URL with the appropriate URL for your remote_write system (such as a Grafana Cloud Hosted Prometheus instance).

When the Agent is run with this file, it will collect metrics from itself and send those metrics to the default remote_write endpoint. All metrics from integrations will have an instance label matching the hostname of the machine the Grafana Agent is running on. This label helps to uniquely identify the source of metrics if you are running multiple Grafana Agents across multiple machines.

Full configuration options can be found in the configuration reference.

Prometheus config/migrating from Prometheus

The Prometheus subsystem config is useful for those migrating from Prometheus and those who want to scrape metrics from something that currently does not have an associated integration.

To migrate from an existing Prometheus config, use this Agent config as a template and copy and paste subsections from your existing Prometheus config into it:

metrics:
  global:
  # PASTE PROMETHEUS global SECTION HERE
  configs:
    - name: agent
      scrape_configs:
        # PASTE scrape_configs SECTION HERE
      remote_write:
        # PASTE remote_write SECTION HERE

For example, this configuration file configures the Grafana Agent to scrape itself without using the integration:

server:
  log_level: info

metrics:
  global:
    scrape_interval: 1m
  configs:
    - name: agent
      scrape_configs:
        - job_name: agent
          static_configs:
            - targets: ['127.0.0.1:12345']
      remote_write:
        - url: http://localhost:9009/api/prom/push

Like with integrations, full configuration options can be found in the configuration.

Loki Config/Migrating from Promtail

The Loki Config allows for collecting logs to send to a Loki API. Users that are familiar with Promtail will notice that the Loki config for the Agent matches their existing Promtail config with the following exceptions:

The deprecated field client is not present
The server field is not present

To migrate from an existing Promtail config, make sure you are using clients instead of client and remove the server block if present. Then paste your Promtail config into the Agent config file inside of a logs section:

logs:
  configs:
  - name: default
    # PASTE YOUR PROMTAIL CONFIG INSIDE OF HERE

Full config example

Here is an example full config file, using integrations, Prometheus, Loki, and Tempo:

server:
  log_level: info

metrics:
  global:
    scrape_interval: 1m
    remote_write:
      - url: http://localhost:9009/api/prom/push
  configs:
    - name: default
      scrape_configs:
        - job_name: agent
          static_configs:
            - targets: ['127.0.0.1:12345']

logs:
  configs:
  - name: default
    positions:
      filename: /tmp/positions.yaml
    scrape_configs:
      - job_name: varlogs
        static_configs:
          - targets: [localhost]
            labels:
              job: varlogs
              __path__: /var/log/*log
    clients:
      - url: http://localhost:3100/loki/api/v1/push

traces:
  configs:
  - name: default
    receivers:
      jaeger:
        protocols:
          grpc: # listens on the default jaeger grpc port: 14250
    remote_write:
      - endpoint: localhost:55680
        insecure: true  # only add this if TLS is not required
    batch:
      timeout: 5s
      send_batch_size: 100

integrations:
  node_exporter:
    enabled: true

Command-line flags

Sat, 11 Apr 2026 09:32:03 +0000

Command-line flags

Command-line flags are used to configure settings of Grafana Agent which cannot be updated at runtime.

All flags may be prefixed with either one hypen or two (i.e., both -config.file and --config.file are valid).

Note: There may be flags returned by -help which are not listed here; this document only lists flags that do not have an equivalent in the YAML file.

Basic

-version: Print out version information
-help: Print out help

Experimental feature flags

Grafana Agent has some experimental features that require being enabled through an -enable-features flag. This flag takes a comma-delimited list of feature names to enable.

Valid feature names are:

remote-configs: Enable retrieving config files over HTTP/HTTPS
integrations-next: Enable revamp of the integrations subsystem
dynamic-config: Enable support for dynamic configuration
extra-scrape-metrics: When enabled, additional time series are exposed for each metrics instance scrape. See Extra scrape metrics.

Report information usage

By default, Grafana Agent sends anonymous, but uniquely-identifiable usage information from your running Grafana Agent instance to Grafana Labs. These statistics are sent to stats.grafana.org.

Statistics help us better understand how Grafana Agent is used. This helps us prioritize features and documentation.

The usage information includes the following details:

A randomly generated and an anonymous unique ID (UUID).
Timestamp of when the UID was first generated.
Timestamp of when the report was created (by default, every 4h).
Version of running Grafana Agent.
Operating system Grafana Agent is running on.
System architecture Grafana Agent is running on.
List of enabled feature flags.
List of enabled integrations.

This list may change over time. All newly reported data will also be documented in the CHANGELOG.

If you would like to disable the reporting, Grafana Agent provides the flag -disable-reporting to stop the reporting.

Support bundles

Grafana Agent allows the exporting of ‘support bundles’ on the /-/support endpoint. Support bundles are zip files containing commonly-used information that provide a baseline for debugging issues with the Agent.

Support bundles contain all information in plain text, so that they can be inspected before sharing to verify that no sensitive information has leaked.

Support bundles contain the following data:

agent-config.yaml contains the current agent configuration (when the -config.enable-read-api flag is passed).
agent-logs.txt contains the agent logs during the bundle generation.
agent-metadata.yaml contains the agent’s build version, operating system, architecture, uptime, plus a string payload defining which extra agent features have been enabled via command-line flags.
agent-metrics-instances.json and agent-metrics-targets.json contain the active metric subsystem instances, and the discovered scraped targets for each one.
agent-logs-instances.json contains the active logs subsystem instances.
agent-metrics.txt contains a snapshot of the agent’s internal metrics.
The pprof/ directory contains Go runtime profiling data (CPU, heap, goroutine, mutex, block profiles) as exported by the pprof package.

To disable the endpoint that exports these support bundles, you can pass in the -disable-support-bundle command-line flag.

Configuration file

-config.file: Path to the configuration file to load. May be an HTTP(s) URL when the remote-configs feature is enabled.
-config.file.type: Type of file which -config.file refers to (default yaml). Valid values are yaml and dynamic.
-config.expand-env: Expand environment variables in the loaded configuration file
-config.enable-read-api: Enables the /-/config and /agent/api/v1/configs/{name} API endpoints to print YAML configuration

Remote Configuration

These flags require the remote-configs feature to be enabled:

-config.url.basic-auth-user: Basic Authentication username to use when fetching the remote configuration file -config.url.basic-auth-password-file: File containing a Basic Authentication password to use when fetching the remote configuration file

Dynamic Configuration

The dynamic-config and integrations-next features must be enabled when -config.file.type is set to dynamic.

Server

-server.register-instrumentation: Expose the /metrics and /debug/pprof/ instrumentation handlers over HTTP (default true)
-server.graceful-shutdown-timeout: Timeout for a graceful server shutdown
-server.log.source-ips.enabled: Whether to log IP addresses of incoming requests
-server.log.source-ips.header: Header field to extract incoming IP requests from (defaults to Forwarded, X-Real-IP, X-Forwarded-For)
-server.log.source-ips.regex: Regex to extract the IP out of the read header, using the first capture group as the IP address
-server.http.network: HTTP server listen network (default tcp)
-server.http.address: HTTP server listen:port (default 127.0.0.1:12345)
-server.http.enable-tls: Enable TLS for the HTTP server
-server.http.conn-limit: Maximum number of simultaneous HTTP connections
-server.http.idle-timeout: HTTP server idle timeout
-server.http.read-timeout: HTTP server read timeout
-server.http.write-timeout: HTTP server write timeout
-server.http.in-memory-addr: Internal address used for the agent to make in-memory HTTP connections to itself. (default agent.internal:12345) The port number specified here is virtual and does not open a real network port.
-server.grpc.network gRPC server listen network (default grpc)
-server.grpc.address: gRPC server listen host:port (default 127.0.0.1:12346)
-server.grpc.enable-tls: Enable TLS for the gRPC server
-server.grpc.conn-limit: Maximum number of simultaneous gRPC connections
-server.grpc.keepalive.max-connection-age Maximum age for any gRPC connection for a graceful shutdown
-server.grpc.keepalive.max-connection-age-grace Grace period to forceibly close connections after a graceful shutdown starts
-server.grpc.keepalive.max-connection-idle Time to wait before closing idle gRPC connections
-server.grpc.keepalive.min-time-between-pings Maximum frequency that clients may send pings at
-server.grpc.keepalive.ping-without-stream-allowed Allow clients to send pings without having a gRPC stream
-server.grpc.keepalive.time Frequency to send keepalive pings from the server
-server.grpc.keepalive.timeout How long to wait for a keepalive pong before closing the connection
-server.grpc.max-concurrent-streams Maximum number of concurrent gRPC streams (0 = unlimited)
-server.grpc.max-recv-msg-size-bytes Maximum size in bytes for received gRPC messages
-server.grpc.max-send-msg-size-bytes Maximum size in bytes for send gRPC messages
-server.grpc.in-memory-addr: Internal address used for the agent to make in-memory gRPC connections to itself. (default agent.internal:12346). The port number specified here is virtual and does not open a real network port.

TLS Support

TLS support can be enabled with -server.http.tls-enabled and -server.grpc.tls-enabled for the HTTP and gRPC servers respectively.

server.http_tls_config and integrations.http_tls_config must be set in the YAML configuration when the -server.http.tls-enabled flag is used.

server.grpc_tls_config must be set in the YAML configuration when the -server.grpc.tls-enabled flag is used.

Metrics

-metrics.wal-directory: Directory to store the metrics Write-Ahead Log in

server_config

Sat, 11 Apr 2026 09:32:03 +0000

server_config

The server_config block configures the Agent’s behavior as an HTTP server, gRPC server, and the log level for the whole process.

The Agent exposes an HTTP server for scraping its own metrics and gRPC for the scraping service mode.

# Log only messages with the given severity or above. Supported values [debug,
# info, warn, error]. This level affects logging for all Agent-level logs, not
# just the HTTP and gRPC server.
#
# Note that some integrations use their own loggers which ignore this
# setting.
[log_level: <string> | default = "info"]

# Log messages with the given format. Supported values [logfmt, json].
# This affects logging for all Agent-levle logs, not just the HTTP and gRPC
# server.
#
# Note that some integrations use their own loggers which ignore this
# setting.
[log_format: <string> | default = "logfmt"]

# TLS configuration for the HTTP server. Reuqired when the
# -server.http.tls-enabled flag is provided, ignored otherwise.
[http_tls_config: <server_tls_config>]

# TLS configuration for the gRPC server. Required when the
# -server.grpc.tls-enabled flag is provided, ignored otherwise.
[grpc_tls_config: <server_tls_config>]

server_tls_config

The server_tls_config configures TLS.

# File path to the server certificate
[cert_file: <string>]

# File path to the server key
[key_file: <string>]

# Tells the server what is acceptable from the client, this drives the options in client_tls_config
[client_auth_type: <string>]

# File path to the signing CA certificate, needed if CA is not trusted
[client_ca_file: <string>]

# Windows certificate filter allows selecting client CA and server certificate from the Windows Certificate store
[windows_certificate_filter: <windows_certificate_filter_config>]

windows_certificate_filter_config

The windows_certificate_filter_config configures the use of the Windows Certificate store. Setting cert_file, key_file, and client_ca_file are invalid settings when using the windows_certificate_filter.

# Client configuration, optional. If nothing specific will use the default client ca root
[client: <windows_client_config>]
  
# Name of the store to look for the Client Certificate ex My, CA
server: <windows_server_config>

windows_client_config

# Array of issuer common names to check against
issuer_common_names:
  [- <string> ... ]

# Regular expression to match Subject name
[subject_regex: <string>]

# Client Template ID to match in ASN1 format ex "1.2.3"
[template_id: <string>]

windows_server_config

# Name of the system store to look for the Server Certificate ex LocalMachine, CurrentUser
system_store: <string>

# Name of the store to look for the Server Certificate ex My, CA
store: <string>

# Array of issuer common names to check against
issuer_common_names:
[- <string> ... ]


# Server Template ID to match in ASN1 format ex "1.2.3"
[template_id: <string>]

# How often to refresh the server certificate ex 5m, 1h
[refresh_interval: <duration>]

metrics_config

Sat, 11 Apr 2026 09:32:03 +0000

metrics_config

The metrics_config block is used to define a collection of metrics instances. Each instance defines a collection of Prometheus-compatible scrape_configs and remote_write rules. Most users will only need to define one instance.

# Configures the optional scraping service to cluster agents.
[scraping_service: <scraping_service_config>]

# Configures the gRPC client used for agents to connect to other
# clustered agents.
[scraping_service_client: <scraping_service_client_config>]

# Configure values for all Prometheus instances.
[global: <global_config>]

# Configure the directory used by instances to store their WAL.
#
# The Grafana Agent assumes that all folders within wal_directory are managed by
# the agent itself. This means if you are using a PVC, you must point
# wal_directory to a subdirectory of the PVC mount.
[wal_directory: <string> | default = ""]

# Configures how long ago an abandoned (not associated with an instance) WAL
# may be written to before being eligible to be deleted
[wal_cleanup_age: <duration> | default = "12h"]

# Configures how often checks for abandoned WALs to be deleted are performed.
# A value of 0 disables periodic cleanup of abandoned WALs
[wal_cleanup_period: <duration> | default = "30m"]

# Allows to disable HTTP Keep-Alives when scraping; the Agent will only use
# outgoing each connection for a single request.
[http_disable_keepalives: <boolean> | default = false]

# Allows to configure the maximum amount of time an idle Keep-Alive connection
# can remain idle before closing itself. Zero means no limit.
# The setting is ignored when `http_disable_keepalives` is enabled.
[http_idle_conn_timeout: <duration> | default = "5m"]

# The list of Prometheus instances to launch with the agent.
configs:
  [- <metrics_instance_config>]

# If an instance crashes abnormally, how long should we wait before trying
# to restart it. 0s disables the backoff period and restarts the agent
# immediately.
[instance_restart_backoff: <duration> | default = "5s"]

# How to spawn instances based on instance configs. Supported values: shared,
# distinct.
[instance_mode: <string> | default = "shared"]

scraping_service_config

The scraping_service block configures the scraping service, an operational mode where configurations are stored centrally in a KV store and a cluster of agents distribute discovery and scrape load between nodes.

# Whether to enable scraping service mode. When enabled, local configs
# cannot be used.
[enabled: <boolean> | default = false]

# Note these next 3 configuration options are confusing. Due to backwards compatibility the naming
# is less than ideal.

# How often should the agent manually refresh the configuration. Useful for if KV change
# events are not sent by an agent.
[reshard_interval: <duration> | default = "1m"]

# The timeout for configuration refreshes. This can occur on cluster events or
# on the reshard interval. A timeout of 0 indicates no timeout.
[reshard_timeout: <duration> | default = "30s"]

# The timeout for a cluster reshard events. A timeout of 0 indicates no timeout.
[cluster_reshard_event_timeout: <duration> | default = "30s"]

# Configuration for the KV store to store configurations.
kvstore: <kvstore_config>

# When set, allows configs pushed to the KV store to specify configuration
# fields that can read secrets from files.
#
# This is disabled by default. When enabled, a malicious user can craft an
# instance config that reads arbitrary files on the machine the Agent runs
# on and sends its contents to a specically crafted remote_write endpoint.
#
# If enabled, ensure that no untrusted users have access to the Agent API.
[dangerous_allow_reading_files: <boolean>]

# Configuration for how agents will cluster together.
lifecycler: <lifecycler_config>

kvstore_config

The kvstore_config block configures the KV store used as storage for configurations in the scraping service mode.

# Which underlying KV store to use. Can be either consul or etcd
[store: <string> | default = ""]

# Key prefix to store all configurations with. Must end in /.
[prefix: <string> | default = "configurations/"]

# Configuration for a Consul client. Only applies if store
# is "consul"
consul:
  # The hostname and port of Consul.
  [host: <string> | duration = "localhost:8500"]

  # The ACL Token used to interact with Consul.
  [acltoken: <string>]

  # The HTTP timeout when communicating with Consul
  [httpclienttimeout: <duration> | default = 20s]

  # Whether or not consistent reads to Consul are enabled.
  [consistentreads: <boolean> | default = true]

# Configuration for an ETCD v3 client. Only applies if
# store is "etcd"
etcd:
  # The ETCD endpoints to connect to.
  endpoints:
    - <string>

  # The Dial timeout for the ETCD connection.
  [dial_tmeout: <duration> | default = 10s]

  # The maximum number of retries to do for failed ops to ETCD.
  [max_retries: <int> | default = 10]

lifecycler_config

The lifecycler_config block configures the lifecycler; the component that Agents use to cluster together.

# Configures the distributed hash ring storage.
ring:
  # KV store for getting and sending distributed hash ring updates.
  kvstore: <kvstore_config>

  # Specifies when other agents in the clsuter should be considered
  # unhealthy if they haven't sent a heartbeat within this duration.
  [heartbeat_timeout: <duration> | default = "1m"]

# Number of tokens to generate for the distributed hash ring.
[num_tokens: <int> | default = 128]

# How often agents should send a heartbeat to the distributed hash
# ring.
[heartbeat_period: <duration> | default = "5s"]

# How long to wait for tokens from other agents after generating
# a new set to resolve collisions. Useful only when using a gossip
# KV store.
[observe_period: <duration> | default = "0s"]

# Period to wait before joining the ring. 0s means to join immediately.
[join_after: <duration> | default = "0s"]

# Minimum duration to wait before marking the agent as ready to receive
# traffic. Used to work around race conditions for multiple agents exiting
# the distributed hash ring at the same time.
[min_ready_duration: <duration> | default = "1m"]

# Network interfaces to resolve addresses defined by other agents
# registered in distributed hash ring.
[interface_names: <string array> | default = ["eth0", "en0"]]

# Duration to sleep before exiting. Ensures that metrics get scraped
# before the process quits.
[final_sleep: <duration> | default = "30s"]

# File path to store tokens. If empty, tokens will not be stored during
# shutdown and will not be restored at startup.
[tokens_file_path: <string> | default = ""]

# Availability zone of the host the agent is running on. Default is an
# empty string which disables zone awareness for writes.
[availability_zone: <string> | default = ""]

scraping_service_client_config

The scraping_service_client_config block configures how clustered Agents will generate gRPC clients to connect to each other.

grpc_client_config:
  # Maximum size in bytes the gRPC client will accept from the connected server.
  [max_recv_msg_size: <int> | default = 104857600]

  # Maximum size in bytes the gRPC client will sent to the connected server.
  [max_send_msg_size: <int> | default = 16777216]

  # Whether messages should be gzipped.
  [use_gzip_compression: <boolean> | default = false]

  # The rate limit for gRPC clients; 0 means no rate limit.
  [rate_limit: <float64> | default = 0]

  # gRPC burst allowed for rate limits.
  [rate_limit_burst: <int> | default = 0]

  # Controls if when a rate limit is hit whether the client should
  # retry the request.
  [backoff_on_ratelimits: <boolean> | default = false]

  # Configures the retry backoff when backoff_on_ratelimits is
  # true.
  backoff_config:
    # The minimum delay when backing off.
    [min_period: <duration> | default = "100ms"]

    # The maximum delay when backing off.
    [max_period: <duration> | default = "10s"]

    # The number of times to backoff and retry before failing.
    [max_retries: <int> | default = 10]

global_config

The global_config block configures global values for all launched Prometheus instances.

# How frequently should Prometheus instances scrape.
[scrape_interval: duration | default = "1m"]

# How long to wait before timing out a scrape from a target.
[scrape_timeout: duration | default = "10s"]

# A list of static labels to add for all metrics.
external_labels:
  { <string>: <string> }

# Default set of remote_write endpoints. If an instance doesn't define any
# remote_writes, it will use this list.
remote_write:
  - [<remote_write>]

Note: For more information on remote_write, refer to the Prometheus documentation

metrics_instance_config

The metrics_instance_config block configures an individual metrics instance, which acts as its own mini Prometheus-compatible agent, though without support for the TSDB.

# Name of the instance. Must be present. Will be added as a label to agent
# metrics.
name: string

# Whether this agent instance should only scrape from targets running on the
# same machine as the agent process.
[host_filter: <boolean> | default = false]

# Relabel configs to apply against discovered targets. The relabeling is
# temporary and just used for filtering targets.
host_filter_relabel_configs:
  [ - <relabel_config> ... ]

# How frequently the WAL truncation process should run. Every iteration of
# the truncation will checkpoint old series and remove old samples. If data
# has not been sent within this window, some of it may be lost.
#
# The size of the WAL will increase with less frequent truncations. Making
# truncations more frequent reduces the size of the WAL but increases the
# chances of data loss when remote_write is failing for longer than the
# specified frequency.
[wal_truncate_frequency: <duration> | default = "60m"]

# The minimum amount of time that series and samples should exist in the WAL
# before being considered for deletion. The consumed disk space of the WAL will
# increase by making this value larger.
#
# Setting this value to 0s is valid, but may delete series before all
# remote_write shards have been able to write all data, and may cause errors on
# slower machines.
[min_wal_time: <duration> | default = "5m"]

# The maximum amount of time that series and samples may exist within the WAL
# before being considered for deletion. Series that have not received writes
# since this period will be removed, and all samples older than this period will
# be removed.
#
# This value is useful in long-running network outages, preventing the WAL from
# growing forever.
#
# Must be larger than min_wal_time.
[max_wal_time: <duration> | default = "4h"]

# Deadline for flushing data when a Prometheus instance shuts down
# before giving up and letting the shutdown proceed.
[remote_flush_deadline: <duration> | default = "1m"]

# When true, writes staleness markers to all active series to
# remote_write.
[write_stale_on_shutdown: <boolean> | default = false]

# A list of scrape configuration rules.
scrape_configs:
  - [<scrape_config>]

# A list of remote_write targets.
remote_write:
  - [<remote_write>]

Note: More information on the following types can be found on the Prometheus website:

relabel_config

scrape_config

remote_write

logs_config

Sat, 11 Apr 2026 09:32:03 +0000

logs_config

The logs_config block configures how the Agent collects logs and sends them to a Loki push API endpoint. logs_config is identical to how Promtail is configured, except deprecated fields have been removed and the server_config is not supported.

Refer to the Promtail documentation for the supported values for these fields.

# Directory to store Loki Promtail positions files in. Positions files are
# required to read logs, and are used to store the last read offset of log
# sources. The positions files will be stored in
# <positions_directory>/<logs_instance_config.name>.yml.
#
# Optional only if every config has a positions.filename manually provided.
#
# This directory will be automatically created if it doesn't exist.
[positions_directory: <string>]

# Configure values for all Loki Promtail instances.
[global: <global_config>]

# Loki Promtail instances to run for log collection.
configs:
  - [<logs_instance_config>]

global_config

The global_config block configures global values for all launched Loki Promtail instances.

clients:
  - [<promtail.client_config>]

Note: More information on the following types can be found on the documentation for Promtail:

promtail.client_config

logs_instance_config

The logs_instance_config block is an individual instance of Promtail with its own set of scrape rules and where to forward logs. It is identical to how Promtail is configured, except deprecated fields have been removed and the server_config block is not supported.

# Name of this config. Required, and must be unique across all Loki configs.
# The name of the config will be the value of a logs_config label for all
# Loki Promtail metrics.
name: <string>

clients:
  - [<promtail.client_config>]

# Optional configuration for where to store the positions files. If
# positions.filename is left empty, the file will be stored in
# <logs_config.positions_directory>/<logs_instance_config.name>.yml.
#
# The directory of the positions file will automatically be created on start up
# if it doesn't already exist..
[positions: <promtail.position_config>]

scrape_configs:
  - [<promtail.scrape_config>]

[target_config: <promtail.target_config>]

Note: More information on the following types can be found on the documentation for Promtail:

promtail.client_config

promtail.scrape_config

promtail.target_config

Note: Backticks in values are not supported.

Note: Because of how YAML treats backslashes in double-quoted strings, all backslashes in a regex expression must be escaped when using double quotes. But because of double processing, in Grafana Agent config file you must use quadruple backslash (\\\\) construction to add backslashes into regular expressions, here is example for name=(\w+)\s regex:

  selector: '{app="my-app"} |~ "name=(\\\\w+)\\\\s"'

Using single or double backslash construction produces the error:

failed to make file target manager: invalid match stage config: invalid selector syntax for match stage: parse error at line 1, col 40: literal not terminated

Using backticks produces the error:

invalid match stage config: invalid selector syntax for match stage: parse error at line 1, col 51: syntax error: unexpected IDENTIFIER, expecting STRING"

traces_config

Sat, 11 Apr 2026 09:32:03 +0000

traces_config

The traces_config block configures a set of Tempo instances, each of which configures its own tracing pipeline. Having multiple configs allows you to configure multiple distinct pipelines, each of which collects spans and sends them to a different location.

Note that if using multiple configs, you must manually set port numbers for each receiver, otherwise they will all try to use the same port and fail to start.

configs:
 [ - <traces_instance_config> ... ]

traces_instance_config

# Name configures the name of this Tempo instance. Names must be non-empty and
# unique across all Tempo instances. The value of the name here will appear in
# logs and as a label on metrics.
name: <string>

# This field allows for the general manipulation of tags on spans that pass
# through this agent. A common use may be to add an environment or cluster
# variable.
[ attributes: <attributes.config> ]

# This field allows to configure grouping spans into batches. Batching helps
# better compress the data and reduce the number of outgoing connections
# required transmit the data.
[ batch: <batch.config> ]

remote_write:
  # host:port to send traces to
  # Here must be the port of gRPC receiver, not the Tempo default port.
  # Example for cloud instances:  `tempo-us-central1.grafana.net:443`
  # For local / on-premises instances: `localhost:55680` or `tempo.example.com:14250`
  # Note: for non-encrypted connections you must also set `insecure: true`
  - endpoint: <string>

    # Custom HTTP headers to be sent along with each remote write request.
    # Be aware that 'authorization' header will be overwritten in presence
    # of basic_auth.
    headers:
      [ <string>: <string> ... ]

    # Controls whether compression is enabled.
    [ compression: <string> | default = "gzip" | supported = "none", "gzip"]

    # Controls what protocol to use when exporting traces.
    # Only "grpc" is supported in Grafana Cloud.
    [ protocol: <string> | default = "grpc" | supported = "grpc", "http" ]

    # Controls what format to use when exporting traces, in combination with protocol.
    # protocol/format supported combinations are grpc/otlp, http/otlp and grpc/jaeger
    # Only grpc/otlp is supported in Grafana Cloud.
    [ format: <string> | default = "otlp" | supported = "otlp", "jaeger" ]

    # Controls whether or not TLS is required.  See https://godoc.org/google.golang.org/grpc#WithInsecure
    [ insecure: <boolean> | default = false ]

    # Deprecated in favor of tls_config
    # If both `insecure_skip_verify` and `tls_config.insecure_skip_verify` are used,
    # the latter take precedence.
    [ insecure_skip_verify: <bool> | default = false ]

    # Configures opentelemetry exporters to use the OpenTelemetry auth extension `oauth2clientauthextension`.
    # Can not be used in combination with `basic_auth`.
    # See https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/extension/oauth2clientauthextension/README.md
    oauth2:
      # Configures the TLS settings specific to the oauth2 client
      # The client identifier issued to the oauth client
      [ client_id: <string> ]
      # The secret string associated with the oauth client
      [ client_secret: <string> ]
      # The resource server's token endpoint URL
      [ token_url: <string> ]
      # Optional, requested permissions associated with the oauth client
      [ scopes: [<string>] ]
      # Optional, specifies the timeout fetching tokens from the token_url. Default: no timeout
      [ timeout: <duration> ]
      tls:
        # Disable validation of the server certificate.
        [ insecure: <bool> | default = false ]
        # Path to the CA cert. For a client this verifies the server certificate. If empty uses system root CA.
        [ ca_file: <string> ]
        # Path to the TLS cert to use for TLS required connections
        [ cert_file: <string> ]
        # Path to the TLS key to use for TLS required connections
        [ key_file: <string> ]

    # Controls TLS settings of the exporter's client. See https://github.com/open-telemetry/opentelemetry-collector/blob/v0.21.0/config/configtls/README.md
    # This should be used only if `insecure` is set to false
    tls_config:
      # Path to the CA cert. For a client this verifies the server certificate. If empty uses system root CA.
      [ ca_file: <string> ]
      # Path to the TLS cert to use for TLS required connections
      [ cert_file: <string> ]
      # Path to the TLS key to use for TLS required connections
      [ key_file: <string> ]
      # Disable validation of the server certificate.
      [ insecure_skip_verify: <bool> | default = false ]

    # Sets the `Authorization` header on every trace push with the
    # configured username and password.
    # password and password_file are mutually exclusive.
    basic_auth:
      [ username: <string> ]
      [ password: <secret> ]
      [ password_file: <string> ]

    [ sending_queue: <otlpexporter.sending_queue> ]
    [ retry_on_failure: <otlpexporter.retry_on_failure> ]

# This processor writes a well formatted log line to a logs instance for each span, root, or process
# that passes through the Agent. This allows for automatically building a mechanism for trace
# discovery and building metrics from traces using Loki. It should be considered experimental.
automatic_logging:
  # Indicates where the stream of log lines should go. Either supports writing
  # to a logs instance defined in this same config or to stdout.
  [ backend: <string> | default = "stdout" | supported "stdout", "logs_instance" ]
  # Indicates the logs instance to write logs to.
  # Required if backend is set to logs_instance.
  [ logs_instance_name: <string> ]
  # Log one line per span. Warning! possibly very high volume
  [ spans: <boolean> ]
  # Log one line for every root span of a trace.
  [ roots: <boolean> ]
  # Log one line for every process
  [ processes: <boolean> ]
  # Additional span attributes to log
  [ span_attributes: <string array> ]
  # Additional process attributes to log
  [ process_attributes: <string array> ]
  # Timeout on writing logs to Loki when backend is "logs_instance."
  [ timeout: <duration> | default = 1ms ]
  # Configures a set of key values that will be logged as labels
  # They need to be span or process attributes logged in the log line
  #
  # This feature only applies when `backend = logs_instance`
  #
  # Loki only accepts alphanumeric and "_" as valid characters for labels.
  # Labels are sanitized by replacing invalid characters with underscores.
  [ labels: <string array> ]
  overrides:
    [ logs_instance_tag: <string> | default = "traces" ]
    [ service_key: <string> | default = "svc" ]
    [ span_name_key: <string> | default = "span" ]
    [ status_key: <string> | default = "status" ]
    [ duration_key: <string> | default = "dur" ]
    [ trace_id_key: <string> | default = "tid" ]

# Receiver configurations are mapped directly into the OpenTelemetry receivers
# block. At least one receiver is required.
# The Agent uses OpenTelemetry v0.36.0. Refer to the corresponding receiver's config.
#
# Supported receivers: otlp, jaeger, kafka, opencensus and zipkin.
receivers: <receivers>

# A list of prometheus scrape configs.  Targets discovered through these scrape
# configs have their __address__ matched against the ip on incoming spans. If a
# match is found then relabeling rules are applied.
scrape_configs:
  [ - <scrape_config> ... ]
# Defines what method is used when adding k/v to spans.
# Options are `update`, `insert` and `upsert`.
# `update` only modifies an existing k/v and `insert` only appends if the k/v
# is not present. `upsert` does both.
[ prom_sd_operation_type: <string> | default = "upsert" ]
# Configures what methods to use to do association between spans and pods.
# PromSD processor matches the IP address of the metadata labels from the k8s API
# with the IP address obtained from the specified pod association method.
# If a match is found then the span is labeled.
#
# Options are `ip`, `net.host.ip`, `k8s.pod.ip`, `hostname` and `connection`.
#   - `ip`, `net.host.ip` and `k8s.pod.ip`, `hostname` match spans tags.
#   - `connection` inspects the context from the incoming requests (gRPC and HTTP).
#
# Tracing instrumentation is commonly the responsible for tagging spans
# with IP address to the labels mentioned above.
# If running on kubernetes, `k8s.pod.ip` can be automatically attached via the
# downward API. For example, if you're using OTel instrumentation libraries, set
# OTEL_RESOURCE_ATTRIBUTES=k8s.pod.ip=$(POD_IP) to inject spans with the sender
# pod's IP.
#
# By default, all methods are enabled, and evaluated in the order specified above.
# Order of evaluation is honored when multiple methods are enabled.
prom_sd_pod_associations:
  [ - <string> ... ]

# spanmetrics supports aggregating Request, Error and Duration (R.E.D) metrics
# from span data.
#
# spanmetrics generates two metrics from spans and uses remote_write or
# OpenTelemetry Prometheus exporters to serve the metrics locally.
#
# In order to use the remote_write exporter, you have to configure a Prometheus
# instance in the Agent and pass its name to the `metrics_instance` field.
#
# If you want to use the OpenTelemetry Prometheus exporter, you have to
# configure handler_endpoint and then scrape that endpoint.
#
# The first generated metric is `calls`, a counter to compute requests.
# The second generated metric is `latency`, a histogram to compute the
# operation's duration.
#
# If you want to rename the generated metrics, you can configure the `namespace`
# option of prometheus exporter.
#
# This is an experimental feature of Opentelemetry-Collector and the behavior
# may change in the future.
spanmetrics:
  # latency_histogram_buckets and dimensions are the same as the configs in
  # spanmetricsprocessor.
  [ latency_histogram_buckets: <spanmetricsprocessor.latency_histogram_buckets> ]
  [ dimensions: <spanmetricsprocessor.dimensions> ]
  # const_labels are labels that will always get applied to the exported
  # metrics.
  const_labels:
    [ <string>: <string> ... ]
  # Metrics are namespaced to `traces_spanmetrics` by default.
  # They can be further namespaced, i.e. `{namespace}_traces_spanmetrics`
  [ namespace: <string> ]
  # metrics_instance is the metrics instance used to remote write metrics.
  [ metrics_instance: <string> ]
  # handler_endpoint defines the endpoint where the OTel prometheus exporter will be exposed.
  [ handler_endpoint: <string> ]

# tail_sampling supports tail-based sampling of traces in the agent.
#
# Policies can be defined that determine what traces are sampled and sent to the
# backends and what traces are dropped.
#
# In order to make a correct sampling decision it's important that the agent has
# a complete trace. This is achieved by waiting a given time for all the spans
# before evaluating the trace.
#
# Tail sampling also supports multi agent deployments, allowing to group all
# spans of a trace in the same agent by load balancing the spans by trace ID
# between the instances.
# * To make use of this feature, check load_balancing below *
tail_sampling:
  # policies define the rules by which traces will be sampled. Multiple policies
  # can be added to the same pipeline.
  policies:
    [ - <tailsamplingprocessor.policies> ... ]

  # Time that to wait before making a decision for a trace.
  # Longer wait times reduce the probability of sampling an incomplete trace at
  # the cost of higher memory usage.
  [ decision_wait: <duration> | default = 5s ]

  # Optional, number of traces kept in memory
  [ num_traces: <int> | default = 50000 ]

  # Optional, expected number of new traces (helps in allocating data structures)
  [ expected_new_traces_per_sec: <int> | default = 0 ]

# load_balancing configures load balancing of spans across multi agent deployments.
# It ensures that all spans of a trace are sampled in the same instance.
# It works by exporting spans based on their traceID via consistent hashing.
#
# Enabling this feature is required for tail_sampling to correctly work when
# different agent instances can receive spans for the same trace.
#
# Load balancing works by layering two pipelines and consistently exporting
# spans belonging to a trace to the same agent instance.
# Agent instances need to be able to communicate with each other via gRPC.
#
# Load balancing significantly increases CPU usage. This is because spans are
# exported an additional time between agents.
load_balancing:
  # resolver configures the resolution strategy for the involved backends
  # It can be static, with a fixed list of hostnames, or DNS, with a hostname
  # (and port) that will resolve to all IP addresses.
  resolver:
    static:
      hostnames:
        [ - <string> ... ]
    dns:
      hostname: <string>
      [ port: <int> ]

  # receiver_port is the port the instance will use to receive load balanced traces
  receiver_port: [ <int> | default = 4318 ]

  # Load balancing is done via an otlp exporter.
  # The remaining configuration is common with the remote_write block.
  exporter:
    # Controls whether compression is enabled.
    [ compression: <string> | default = "gzip" | supported = "none", "gzip"]

    # Controls whether or not TLS is required.
    [ insecure: <boolean> | default = false ]

    # Disable validation of the server certificate. Only used when insecure is set
    # to false.
    [ insecure_skip_verify: <bool> | default = false ]

    # Sets the `Authorization` header on every trace push with the
    # configured username and password.
    # password and password_file are mutually exclusive.
    basic_auth:
      [ username: <string> ]
      [ password: <secret> ]
      [ password_file: <string> ]

# service_graphs configures processing of traces for building service graphs in
# the form of prometheus metrics. The generated metrics represent edges between
# nodes in the graph. Nodes are represented by `client` and `server` labels.
#
#  e.g. tempo_service_graph_request_total{client="app", server="db"} 20
#
# Service graphs works by inspecting spans and looking for the tag `span.kind`.
# If it finds the span kind to be client or server, it stores the request in a
# local in-memory store.
#
# That request waits until its corresponding client or server pair span is
# processed or until the maximum waiting time has passed.
# When either of those conditions is reached, the request is processed and
# removed from the local store. If the request is complete by that time, it'll
# be recorded as an edge in the graph.
#
# Service graphs supports multi-agent deployments, allowing to group all spans
# of a trace in the same agent by load balancing the spans by trace ID between
# the instances.
# * To make use of this feature, check load_balancing above *
service_graphs:
  [ enabled: <bool> | default = false ]

  # configures the time the processor will wait since a span is consumed until
  # it's considered expired if its paired has not been processed.
  #
  # increasing the waiting time will increase the percentage of paired spans.
  # retaining unpaired spans for longer will make reaching max_items more likely.
  [ wait: <duration> | default = 10s ]

  # configures the max amount of edges that will be stored in memory.
  #
  # spans that arrive to the processor that do not pair with an already
  # processed span are dropped.
  #
  # a higher max number of items increases the max throughput of processed spans
  # with a higher memory consumption.
  [ max_items: <integer> | default = 10_000 ]

  # configures the number of workers that will process completed edges concurrently.
  # as edges are completed, they get queued to be collected as metrics for the graph.
  [ workers: <integer> | default = 10 ]

  # configures what status codes are considered as successful (e.g. HTTP 404).
  #
  # by default, a request is considered failed in the following cases:
  #   1. HTTP status is not 2XX
  #   1. gRPC status code is not OK
  #   1. span status is Error
  success_codes:
    # http status codes not to be considered as failure
    http:
      [ - <int> ... ]
    # grpc status codes not to be considered as failure
    grpc:
      [ - <int> ... ]

Note: More information on the following types can be found on the documentation for their respective projects:

dynamic_config

Sat, 11 Apr 2026 09:32:03 +0000

Dynamic Configuration - Experimental

This is experimental and subject to change at anytime, feedback is much appreciated. This is a feature that MAY NOT make it production.

Dynamic Configuration is the combination of two things:

Loading from multiple files
Using templates and datasources

Both of these make heavy use of the excellent gomplate. The goal is that as the configuration grows that it can be split it up into smaller segments to allow better readability and handling. The configurations cannot be patched in any order and instead are allowed at several levels.

The second goal is to allow the use of templating, functions for gomplate doc go into detail on what functions are available.

Configuration

Dynamic configuration files can be used by passing -config.file.type=dynamic -enable-features=dynamic-config,integrations-next. When these flags are passed, the file referred to -config.file will be loaded as a dynamic configuration file.

Dynamic configuration files are YAML which conform the following schema:

# Sources to pull template values
datasources:
  [- <sources_config>]

# Locations to use searching for templates, the system does NOT look into subdirectories. Follows gomplate schema
# from [gomplate datasources](https://docs.gomplate.ca/datasources/). File and S3/GCP templates are currently supported
template_paths:
  [ - string ]

# Filters allow you to override the default naming convention

agent_filter:            string # defaults to agent-*.yml
server_filter:           string # defaults to server-*.yml
metrics_filter:          string # defaults to metrics-*.yml
metrics_instance_filter: string # defaults to metrics_instances-*.yml
integrations_filter:     string # defaults to integrations-*.yml
logs_filter:             string # defaults to logs-*.yml
traces_filter:           string # defaults to traces-*.yml

sources_config

# Name of the source to use when templating
name: string

# Path to datasource using schema from [gomplate datasources](https://docs.gomplate.ca/datasources/)
url: string

Templates

Note when adding a template you MUST NOT add the type as the top level yaml field. For instance if using traces:

Incorrect

traces:
  configs:
  - name: default
    automatic_logging:
      backend: loki
      loki_name: default
      spans: true

Correct

configs:
- name: default
  automatic_logging:
    backend: loki
    loki_name: default
    spans: true

Configurations are loaded in the order as they are listed below.

Agent

Agent template is the standard agent configuration file in its entirety. The default filter is agent-*.yml. Only one file is supported. This is processed first then any subsequent configurations found REPLACE the values here, it is not additive.

Reference

Server

The default filter is server-*.yml, only ONE server file is supported.

Reference

Metrics

The default filter is metrics-*.yml, only ONE metrics file is supported.

Reference

Metric Instances

The default filter is metrics_instances-*.yml. Any metric instances are appended to the instances defined in Metrics above. Any number of metric instance files are supporter.

Reference in the metrics instance

Integrations

The default filter is integrations-*.yml, these support more than one file, and multiple integrations can be defined in a file. Do not assume any order of loading for integrations. For any integration that is a singleton, loading multiple of those will result in an error.

Reference

Traces

The default filter is traces-*.yml. This supports ONE file.

Reference

Logs

The default filter is logs-*.yml. This supports ONE file.

Reference

integrations_config

Sat, 11 Apr 2026 09:32:03 +0000

integrations_config

The integrations_config block configures how the Agent runs integrations that scrape and send metrics without needing to run specific Prometheus exporters or manually write scrape_configs:

# Controls the Agent integration
agent:
  # Enables the Agent integration, allowing the Agent to automatically
  # collect and send metrics about itself.
  [enabled: <boolean> | default = false]

  # Sets an explicit value for the instance label when the integration is
  # self-scraped. Overrides inferred values.
  #
  # The default value for this integration is inferred from the agent hostname
  # and HTTP listen port, delimited by a colon.
  [instance: <string>]

  # Automatically collect metrics from this integration. If disabled,
  # the agent integration will be run but not scraped and thus not
  # remote_written. Metrics for the integration will be exposed at
  # /integrations/agent/metrics and can be scraped by an external process.
  [scrape_integration: <boolean> | default = <integrations_config.scrape_integrations>]

  # How often should the metrics be collected? Defaults to
  # prometheus.global.scrape_interval.
  [scrape_interval: <duration> | default = <global_config.scrape_interval>]

  # The timeout before considering the scrape a failure. Defaults to
  # prometheus.global.scrape_timeout.
  [scrape_timeout: <duration> | default = <global_config.scrape_timeout>]

  # How frequent to truncate the WAL for this integration.
  [wal_truncate_frequency: <duration> | default = "60m"]

  # Allows for relabeling labels on the target.
  relabel_configs:
    [- <relabel_config> ... ]

  # Relabel metrics coming from the integration, allowing to drop series
  # from the integration that you don't care about.
  metric_relabel_configs:
    [ - <relabel_config> ... ]

# Client TLS Configuration
# Client Cert/Key Values need to be defined if the server is requesting a certificate
#  (Client Auth Type = RequireAndVerifyClientCert || RequireAnyClientCert).
http_tls_config: <tls_config>

# Controls the apache_http integration
apache_http: <apache_http_config>

# Controls the node_exporter integration
node_exporter: <node_exporter_config>

# Controls the process_exporter integration
process_exporter: <process_exporter_config>

# Controls the mysqld_exporter integration
mysqld_exporter: <mysqld_exporter_config>

# Controls the redis_exporter integration
redis_exporter: <redis_exporter_config>

# Controls the dnsmasq_exporter integration
dnsmasq_exporter: <dnsmasq_exporter_config>

# Controls the elasticsearch_exporter integration
elasticsearch_exporter: <elasticsearch_exporter_config>

# Controls the memcached_exporter integration
memcached_exporter: <memcached_exporter_config>

# Controls the postgres_exporter integration
postgres_exporter: <postgres_exporter_config>

# Controls the snmp_exporter integration
snmp_exporter: <snmp_exporter_config>

# Controls the snowflake integration
snowflake: <snowflake_config>

# Controls the statsd_exporter integration
statsd_exporter: <statsd_exporter_config>

# Controls the consul_exporter integration
consul_exporter: <consul_exporter_config>

# Controls the windows_exporter integration
windows_exporter: <windows_exporter_config>

# Controls the kafka_exporter integration
kafka_exporter: <kafka_exporter_config>

# Controls the mongodb_exporter integration
mongodb_exporter: <mongodb_exporter_config>

# Controls the github_exporter integration
github_exporter: <github_exporter_config>

# Controls the blackbox_exporter integration
blackbox: <blackbox_config>

# Automatically collect metrics from enabled integrations. If disabled,
# integrations will be run but not scraped and thus not remote_written. Metrics
# for integrations will be exposed at /integrations/<integration_key>/metrics
# and can be scraped by an external process.
[scrape_integrations: <boolean> | default = true]

# Extra labels to add to all samples coming from integrations.
labels:
  { <string>: <string> }

# The period to wait before restarting an integration that exits with an
# error.
[integration_restart_backoff: <duration> | default = "5s"]

# A list of remote_write targets. Defaults to global_config.remote_write.
# If provided, overrides the global defaults.
prometheus_remote_write:
  - [<remote_write>]

Scraping Service Mode

Sat, 11 Apr 2026 09:32:03 +0000

Scraping Service Mode (Beta)

Scraping Service Mode is a third operational mode of the Grafana Agent that allows for clustering a set of Agent processes and distributing scrape load across them.

Determining what to scrape is done by writing instance configuration files to an API, which then stores the configuration files in a KV store backend. All agents in the cluster must use the same KV store so they see the same set of config files.

Each process of the Grafana Agent can be running multiple independent “instances” at once, where an “instance” refers to the combination of:

Service discovery for all scrape_configs within that loaded config
Scrapes metrics from all discovered targets
Stores data in its own Write-Ahead Log specific to the loaded config
Remote Writes scraped metrics to the configured remote_write destinations specified within the loaded config.

The “instance configuration file,” then, is the configuration file that specifies the set of scrape_configs and remote_write endpoints. For example, a small instance configuration file looks like:

scrape_configs:
  - job_name: self-scrape
    static_configs:
      - targets: ['localhost:9090']
        labels:
          process: 'agent'
remote_write:
  - url: http://cortex:9009/api/prom/push

The full set of supported options for an instance configuration file is available in the metrics-config.md file.

Having multiple instance configuration files is necessary for sharding; each config file is distributed to a particular agent on the cluster based on the hash of its contents.

When Scraping Service Mode is enabled, Agents disallow specifying instance configurations locally in the configuration file; using the KV store is required. agentctl can be used to manually sync instance configuration files to the Agent’s API server.

Distributed hash ring

Scraping Service Mode uses a Distributed Hash Ring (commonly just called “the ring”) to cluster agents and to shard configurations within that ring. Each Agent joins the ring with a random distinct set of tokens that are used for sharding. The default number of generated tokens is 128.

The Distributed Hash Ring is also stored in a KV store. Since a KV store is also needed for storing configuration files, it is encouraged to re-use the same KV store for the ring.

When sharding, the Agent currently uses the name of a config file stored in the KV store for load distribution. Config names are guaranteed to be unique keys. The hash of the name is used as the lookup key in the ring and determines which agent (based on token) should be responsible for that config. “Price is Right” rules are used for the Agent lookup; the Agent owning the token with the closest value to the key without going over is responsible for the config.

All Agents are simultaneously watching the KV store for changes to the set of configuration files. When a config file is added or updated in the configuration store, each Agent will run the config name hash through their copy of the Hash Ring to determine if they are responsible for that config.

When an Agent receives a new config that it is responsible for, it launches a new instance from the instance config. If a config is deleted from the KV store, this will be detected by the owning Agent and it will stop the metric collection process for that config file.

When an Agent receives an event for an updated configuration file that they used to be the owner of but are no longer the owner, the associated instance for that configuration file is stopped for that Agent. This can happen when the cluster size changes.

Scraping Service Mode currently does not support replication; only one agent at a time will be responsible for scraping a certain config.

Resharding

When a new Agent joins or leaves the cluster, the set of tokens in the ring may cause configurations to hash to a new Agent. The process of responding to this action is called “resharding.”

Resharding is run:

When an Agent joins the ring
When an Agent leaves the ring
When the KV store sends a notification indicating a config has changed.
On a specified interval in case KV change events have not fired.

The resharding process involves each Agent retrieving the full set of configurations stored in the KV store and determining if:

The config owned by the current resharding Agent has changed and needs to be reloaded.
The config is no longer owned by the current resharding Agent and the associated instance should be stopped.
The config has been deleted and the associated instance should be stopped.

Best practices

Because distribution is determined by the number of config files and not how many targets exist per config file, the best amount of distribution is achieved when each config file has the lowest amount of targets possible. The best distribution will be achieved if each config file stored in the KV store is limited to one static config with only one target.

A better distribution mechanism that distributes based on discovered targets is planned for the future.

Example

Here’s an example agent.yaml config file that uses the same etcd server for both configuration storage and the distributed hash ring storage:

server:
  log_level: debug

metrics:
  global:
    scrape_interval: 1m
  scraping_service:
    enabled: true
    kvstore:
      store: etcd
      etcd:
        endpoints:
          - etcd:2379
    lifecycler:
      ring:
        replication_factor: 1
        kvstore:
          store: etcd
          etcd:
            endpoints:
              - etcd:2379

Note that there are no instance configs present in this example; instance configs must be passed to the API for the Agent to start scraping metrics.

agentctl

agentctl is a tool included with this repository that helps users interact with the new Config Management API. The agentctl config-sync subcommand uses local YAML files as a source of truth and syncs their contents with the API. Entries in the API not in the synced directory will be deleted.

agentctl is distributed in binary form with each release and as a Docker container with the grafana/agentctl image. Tanka configurations that utilize grafana/agentctl and sync a set of configurations to the API are planned for the future.