Configure static mode on Grafana Labs

Create a configuration file

Wed, 11 Sep 2024 17:43:35 +0000

Create a configuration file

The Grafana Agent supports configuring multiple independent “subsystems.” Each subsystem helps you collect data for a specific type of telemetry.

The Metrics subsystem allows you collect metrics to send to Prometheus.
The Logs subsystem allows you to collect logs to send to Grafana Loki.
The Traces subsystem allows you to collect spans to send to Grafana Tempo.
The Integrations subsystem allows you to collect metrics for common applications, such as MySQL.

Integrations are recommended for first-time users of observability platforms, especially newcomers to Prometheus. Users with more experience with Prometheus or users that already have an existing Prometheus config file can configure the Prometheus subsystem manually.

Integrations

Integrations are individual features that collect metrics for you. For example, the agent integration collects metrics from that running instance of the Grafana Agent. The node_exporter integration will collect metrics from the Linux machine that the Grafana Agent is running on.

metrics:
  wal_directory: /tmp/wal
  global:
    remote_write:
      - url: http://localhost:9009/api/prom/push

integrations:
  agent:
    enabled: true

In this example, we first must configure the wal_directory which is used to store metrics in a Write-Ahead Log (WAL). The WAL is required and ensures that samples will be redelivered in case of failure (e.g., network issues, machine reboot). We also configure remote_write, which is where all metrics should be sent by default.

Then, the individual integrations are configured. In this example, just the agent integration is enabled. Finally, prometheus_remote_write is configured with a location to send metrics. You will have to replace this URL with the appropriate URL for your remote_write system (such as a Grafana Cloud Hosted Prometheus instance).

When the Agent is run with this file, it will collect metrics from itself and send those metrics to the default remote_write endpoint. All metrics from integrations will have an instance label matching the hostname of the machine the Grafana Agent is running on. This label helps to uniquely identify the source of metrics if you are running multiple Grafana Agents across multiple machines.

Full configuration options can be found in the configuration reference.

Prometheus config/migrating from Prometheus

The Prometheus subsystem config is useful for those migrating from Prometheus and those who want to scrape metrics from something that currently does not have an associated integration.

To migrate from an existing Prometheus config, use this Agent config as a template and copy and paste subsections from your existing Prometheus config into it:

metrics:
  global:
  # PASTE PROMETHEUS global SECTION HERE
  configs:
    - name: agent
      scrape_configs:
        # PASTE scrape_configs SECTION HERE
      remote_write:
        # PASTE remote_write SECTION HERE

For example, this configuration file configures the Grafana Agent to scrape itself without using the integration:

server:
  log_level: info

metrics:
  global:
    scrape_interval: 1m
  configs:
    - name: agent
      scrape_configs:
        - job_name: agent
          static_configs:
            - targets: ['127.0.0.1:12345']
      remote_write:
        - url: http://localhost:9009/api/prom/push

Like with integrations, full configuration options can be found in the configuration.

Loki Config/Migrating from Promtail

The Loki Config allows for collecting logs to send to a Loki API. Users that are familiar with Promtail will notice that the Loki config for the Agent matches their existing Promtail config with the following exceptions:

The deprecated field client is not present
The server field is not present

To migrate from an existing Promtail config, make sure you are using clients instead of client and remove the server block if present. Then paste your Promtail config into the Agent config file inside of a logs section:

logs:
  configs:
  - name: default
    # PASTE YOUR PROMTAIL CONFIG INSIDE OF HERE

Full config example

Here is an example full config file, using integrations, Prometheus, Loki, and Tempo:

server:
  log_level: info

metrics:
  global:
    scrape_interval: 1m
    remote_write:
      - url: http://localhost:9009/api/prom/push
  configs:
    - name: default
      scrape_configs:
        - job_name: agent
          static_configs:
            - targets: ['127.0.0.1:12345']

logs:
  configs:
  - name: default
    positions:
      filename: /tmp/positions.yaml
    scrape_configs:
      - job_name: varlogs
        static_configs:
          - targets: [localhost]
            labels:
              job: varlogs
              __path__: /var/log/*log
    clients:
      - url: http://localhost:3100/loki/api/v1/push

traces:
  configs:
  - name: default
    receivers:
      jaeger:
        protocols:
          grpc: # listens on the default jaeger grpc port: 14250
    remote_write:
      - endpoint: localhost:55680
        insecure: true  # only add this if TLS is not required
    batch:
      timeout: 5s
      send_batch_size: 100

integrations:
  node_exporter:
    enabled: true

Command-line flags

Wed, 11 Sep 2024 17:43:35 +0000

Command-line flags

Command-line flags are used to configure settings of Grafana Agent which cannot be updated at runtime.

All flags may be prefixed with either one hyphen or two (i.e., both -config.file and --config.file are valid).

Note: There may be flags returned by -help which are not listed here; this document only lists flags that do not have an equivalent in the YAML file.

Basic

-version: Print out version information
-help: Print out help

Experimental feature flags

Grafana Agent has some experimental features that require being enabled through an -enable-features flag. This flag takes a comma-delimited list of feature names to enable.

Valid feature names are:

remote-configs: Enable retrieving config files over HTTP/HTTPS
integrations-next: Enable revamp of the integrations subsystem
extra-scrape-metrics: When enabled, additional time series are exposed for each metrics instance scrape. See Extra scrape metrics.
agent-management: Enable support for agent management.

Report information usage

By default, Grafana Agent sends anonymous, but uniquely-identifiable usage information from your running Grafana Agent instance to Grafana Labs. These statistics are sent to stats.grafana.org.

Statistics help us better understand how Grafana Agent is used. This helps us prioritize features and documentation.

The usage information includes the following details:

A randomly generated and an anonymous unique ID (UUID).
Timestamp of when the UID was first generated.
Timestamp of when the report was created (by default, every 4h).
Version of running Grafana Agent.
Operating system Grafana Agent is running on.
System architecture Grafana Agent is running on.
List of enabled feature flags.
List of enabled integrations.

This list may change over time. All newly reported data will also be documented in the CHANGELOG.

If you would like to disable the reporting, Grafana Agent provides the flag -disable-reporting to stop the reporting.

Support bundles

Grafana Agent allows the exporting of ‘support bundles’ on the /-/support endpoint. Support bundles are zip files containing commonly-used information that provide a baseline for debugging issues with the Agent.

Support bundles contain all information in plain text, so that they can be inspected before sharing to verify that no sensitive information has leaked.

Support bundles contain the following data:

agent-config.yaml contains the current agent configuration (when the -config.enable-read-api flag is passed).
agent-logs.txt contains the agent logs during the bundle generation.
agent-metadata.yaml contains the agent’s build version, operating system, architecture, uptime, plus a string payload defining which extra agent features have been enabled via command-line flags.
agent-metrics-instances.json and agent-metrics-targets.json contain the active metric subsystem instances, and the discovered scraped targets for each one.
agent-logs-instances.json and agent-logs-targets.json contains the active logs subsystem instances and the discovered log targets for each one.
agent-metrics.txt contains a snapshot of the agent’s internal metrics.
The pprof/ directory contains Go runtime profiling data (CPU, heap, goroutine, mutex, block profiles) as exported by the pprof package.

To disable the endpoint that exports these support bundles, you can pass in the -disable-support-bundle command-line flag.

Configuration file

-config.file: Path to the configuration file to load. May be an HTTP(s) URL when the remote-configs feature is enabled.
-config.file.type: Type of file which -config.file refers to (default yaml). Valid values are yaml and dynamic.
-config.expand-env: Expand environment variables in the loaded configuration file
-config.enable-read-api: Enables the /-/config and /agent/api/v1/configs/{name} API endpoints to print YAML configuration

Remote Configuration

These flags require the remote-configs feature to be enabled:

-config.url.basic-auth-user: Basic Authentication username to use when fetching the remote configuration file -config.url.basic-auth-password-file: File containing a Basic Authentication password to use when fetching the remote configuration file

Server

-server.register-instrumentation: Expose the /metrics and /debug/pprof/ instrumentation handlers over HTTP (default true)
-server.graceful-shutdown-timeout: Timeout for a graceful server shutdown
-server.log.source-ips.enabled: Whether to log IP addresses of incoming requests
-server.log.source-ips.header: Header field to extract incoming IP requests from (defaults to Forwarded, X-Real-IP, X-Forwarded-For)
-server.log.source-ips.regex: Regex to extract the IP out of the read header, using the first capture group as the IP address
-server.http.network: HTTP server listen network (default tcp)
-server.http.address: HTTP server listen:port (default 127.0.0.1:12345)
-server.http.enable-tls: Enable TLS for the HTTP server
-server.http.conn-limit: Maximum number of simultaneous HTTP connections
-server.http.idle-timeout: HTTP server idle timeout
-server.http.read-timeout: HTTP server read timeout
-server.http.write-timeout: HTTP server write timeout
-server.http.in-memory-addr: Internal address used for the agent to make in-memory HTTP connections to itself. (default agent.internal:12345) The port number specified here is virtual and does not open a real network port.
-server.grpc.network gRPC server listen network (default grpc)
-server.grpc.address: gRPC server listen host:port (default 127.0.0.1:12346)
-server.grpc.enable-tls: Enable TLS for the gRPC server
-server.grpc.conn-limit: Maximum number of simultaneous gRPC connections
-server.grpc.keepalive.max-connection-age Maximum age for any gRPC connection for a graceful shutdown
-server.grpc.keepalive.max-connection-age-grace Grace period to forcibly close connections after a graceful shutdown starts
-server.grpc.keepalive.max-connection-idle Time to wait before closing idle gRPC connections
-server.grpc.keepalive.min-time-between-pings Maximum frequency that clients may send pings at
-server.grpc.keepalive.ping-without-stream-allowed Allow clients to send pings without having a gRPC stream
-server.grpc.keepalive.time Frequency to send keepalive pings from the server
-server.grpc.keepalive.timeout How long to wait for a keepalive pong before closing the connection
-server.grpc.max-concurrent-streams Maximum number of concurrent gRPC streams (0 = unlimited)
-server.grpc.max-recv-msg-size-bytes Maximum size in bytes for received gRPC messages
-server.grpc.max-send-msg-size-bytes Maximum size in bytes for send gRPC messages
-server.grpc.in-memory-addr: Internal address used for the agent to make in-memory gRPC connections to itself. (default agent.internal:12346). The port number specified here is virtual and does not open a real network port.

TLS Support

TLS support can be enabled with -server.http.tls-enabled and -server.grpc.tls-enabled for the HTTP and gRPC servers respectively.

server.http_tls_config and integrations.http_tls_config must be set in the YAML configuration when the -server.http.tls-enabled flag is used.

server.grpc_tls_config must be set in the YAML configuration when the -server.grpc.tls-enabled flag is used.

Metrics

-metrics.wal-directory: Directory to store the metrics Write-Ahead Log in

server_config

Wed, 11 Sep 2024 17:43:35 +0000

server_config

The server_config block configures the Agent’s behavior as an HTTP server, gRPC server, and the log level for the whole process.

The Agent exposes an HTTP server for scraping its own metrics and gRPC for the scraping service mode.

# Log only messages with the given severity or above. Supported values [debug,
# info, warn, error]. This level affects logging for all Agent-level logs, not
# just the HTTP and gRPC server.
#
# Note that some integrations use their own loggers which ignore this
# setting.
[log_level: <string> | default = "info"]

# Log messages with the given format. Supported values [logfmt, json].
# This affects logging for all Agent-levle logs, not just the HTTP and gRPC
# server.
#
# Note that some integrations use their own loggers which ignore this
# setting.
[log_format: <string> | default = "logfmt"]

# TLS configuration for the HTTP server. Required when the
# -server.http.tls-enabled flag is provided, ignored otherwise.
[http_tls_config: <server_tls_config>]

# TLS configuration for the gRPC server. Required when the
# -server.grpc.tls-enabled flag is provided, ignored otherwise.
[grpc_tls_config: <server_tls_config>]

server_tls_config

The server_tls_config configures TLS.

# File path to the server certificate
[cert_file: <string>]

# File path to the server key
[key_file: <string>]

# Tells the server what is acceptable from the client, this drives the options in client_tls_config
[client_auth_type: <string>]

# File path to the signing CA certificate, needed if CA is not trusted
[client_ca_file: <string>]

# Windows certificate filter allows selecting client CA and server certificate from the Windows Certificate store
[windows_certificate_filter: <windows_certificate_filter_config>]

windows_certificate_filter_config

The windows_certificate_filter_config configures the use of the Windows Certificate store. Setting cert_file, key_file, and client_ca_file are invalid settings when using the windows_certificate_filter.

# Client configuration, optional. If nothing specific will use the default client ca root
[client: <windows_client_config>]

# Name of the store to look for the Client Certificate ex My, CA
server: <windows_server_config>

windows_client_config

# Array of issuer common names to check against
issuer_common_names:
  [- <string> ... ]

# Regular expression to match Subject name
[subject_regex: <string>]

# Client Template ID to match in ASN1 format ex "1.2.3"
[template_id: <string>]

windows_server_config

# Name of the system store to look for the Server Certificate ex LocalMachine, CurrentUser
system_store: <string>

# Name of the store to look for the Server Certificate ex My, CA
store: <string>

# Array of issuer common names to check against
issuer_common_names:
[- <string> ... ]


# Server Template ID to match in ASN1 format ex "1.2.3"
[template_id: <string>]

# How often to refresh the server certificate ex 5m, 1h
[refresh_interval: <duration>]

metrics_config

Wed, 11 Sep 2024 17:43:35 +0000

metrics_config

The metrics_config block is used to define a collection of metrics instances. Each instance defines a collection of Prometheus-compatible scrape_configs and remote_write rules. Most users will only need to define one instance.

# Configures the optional scraping service to cluster agents.
[scraping_service: <scraping_service_config>]

# Configures the gRPC client used for agents to connect to other
# clustered agents.
[scraping_service_client: <scraping_service_client_config>]

# Configure values for all Prometheus instances.
[global: <global_config>]

# Configure the directory used by instances to store their WAL.
#
# The Grafana Agent assumes that all folders within wal_directory are managed by
# the agent itself. This means if you are using a PVC, you must point
# wal_directory to a subdirectory of the PVC mount.
[wal_directory: <string> | default = "data-agent/"]

# Configures how long ago an abandoned (not associated with an instance) WAL
# may be written to before being eligible to be deleted
[wal_cleanup_age: <duration> | default = "12h"]

# Configures how often checks for abandoned WALs to be deleted are performed.
# A value of 0 disables periodic cleanup of abandoned WALs
[wal_cleanup_period: <duration> | default = "30m"]

# Allows to disable HTTP Keep-Alives when scraping; the Agent will only use
# outgoing each connection for a single request.
[http_disable_keepalives: <boolean> | default = false]

# Allows to configure the maximum amount of time an idle Keep-Alive connection
# can remain idle before closing itself. Zero means no limit.
# The setting is ignored when `http_disable_keepalives` is enabled.
[http_idle_conn_timeout: <duration> | default = "5m"]

# The list of Prometheus instances to launch with the agent.
configs:
  [- <metrics_instance_config>]

# If an instance crashes abnormally, how long should we wait before trying
# to restart it. 0s disables the backoff period and restarts the agent
# immediately.
[instance_restart_backoff: <duration> | default = "5s"]

# How to spawn instances based on instance configs. Supported values: shared,
# distinct.
[instance_mode: <string> | default = "shared"]

scraping_service_config

The scraping_service block configures the scraping service, an operational mode where configurations are stored centrally in a KV store and a cluster of agents distributes discovery and scrape load between nodes.

# Whether to enable scraping service mode. When enabled, local configs
# cannot be used.
[enabled: <boolean> | default = false]

# Note these next 3 configuration options are confusing. Due to backwards compatibility the naming
# is less than ideal.

# How often should the agent manually refresh the configuration. Useful for if KV change
# events are not sent by an agent.
[reshard_interval: <duration> | default = "1m"]

# The timeout for configuration refreshes. This can occur on cluster events or
# on the reshard interval. A timeout of 0 indicates no timeout.
[reshard_timeout: <duration> | default = "30s"]

# The timeout for a cluster reshard events. A timeout of 0 indicates no timeout.
[cluster_reshard_event_timeout: <duration> | default = "30s"]

# Configuration for the KV store to store configurations.
kvstore: <kvstore_config>

# When set, allows configs pushed to the KV store to specify configuration
# fields that can read secrets from files.
#
# This is disabled by default. When enabled, a malicious user can craft an
# instance config that reads arbitrary files on the machine the Agent runs
# on and sends its contents to a specically crafted remote_write endpoint.
#
# If enabled, ensure that no untrusted users have access to the Agent API.
[dangerous_allow_reading_files: <boolean>]

# Configuration for how agents will cluster together.
lifecycler: <lifecycler_config>

kvstore_config

The kvstore_config block configures the KV store used as storage for configurations in the scraping service mode.

# Which underlying KV store to use. Can be either consul or etcd
[store: <string> | default = ""]

# Key prefix to store all configurations with. Must end in /.
[prefix: <string> | default = "configurations/"]

# Configuration for a Consul client. Only applies if store
# is "consul"
consul:
  # The hostname and port of Consul.
  [host: <string> | duration = "localhost:8500"]

  # The ACL Token used to interact with Consul.
  [acltoken: <string>]

  # The HTTP timeout when communicating with Consul
  [httpclienttimeout: <duration> | default = 20s]

  # Whether or not consistent reads to Consul are enabled.
  [consistentreads: <boolean> | default = true]

# Configuration for an ETCD v3 client. Only applies if
# store is "etcd"
etcd:
  # The ETCD endpoints to connect to.
  endpoints:
    - <string>

  # The Dial timeout for the ETCD connection.
  [dial_tmeout: <duration> | default = 10s]

  # The maximum number of retries to do for failed ops to ETCD.
  [max_retries: <int> | default = 10]

lifecycler_config

The lifecycler_config block configures the lifecycler; the component that Agents use to cluster together.

# Configures the distributed hash ring storage.
ring:
  # KV store for getting and sending distributed hash ring updates.
  kvstore: <kvstore_config>

  # Specifies when other agents in the clsuter should be considered
  # unhealthy if they haven't sent a heartbeat within this duration.
  [heartbeat_timeout: <duration> | default = "1m"]

# Number of tokens to generate for the distributed hash ring.
[num_tokens: <int> | default = 128]

# How often agents should send a heartbeat to the distributed hash
# ring.
[heartbeat_period: <duration> | default = "5s"]

# How long to wait for tokens from other agents after generating
# a new set to resolve collisions. Useful only when using a gossip
# KV store.
[observe_period: <duration> | default = "0s"]

# Period to wait before joining the ring. 0s means to join immediately.
[join_after: <duration> | default = "0s"]

# Minimum duration to wait before marking the agent as ready to receive
# traffic. Used to work around race conditions for multiple agents exiting
# the distributed hash ring at the same time.
[min_ready_duration: <duration> | default = "1m"]

# Network interfaces to resolve addresses defined by other agents
# registered in distributed hash ring.
[interface_names: <string array> | default = ["eth0", "en0"]]

# Duration to sleep before exiting. Ensures that metrics get scraped
# before the process quits.
[final_sleep: <duration> | default = "30s"]

# File path to store tokens. If empty, tokens will not be stored during
# shutdown and will not be restored at startup.
[tokens_file_path: <string> | default = ""]

# Availability zone of the host the agent is running on. Default is an
# empty string which disables zone awareness for writes.
[availability_zone: <string> | default = ""]

scraping_service_client_config

The scraping_service_client_config block configures how clustered Agents will generate gRPC clients to connect to each other.

grpc_client_config:
  # Maximum size in bytes the gRPC client will accept from the connected server.
  [max_recv_msg_size: <int> | default = 104857600]

  # Maximum size in bytes the gRPC client will sent to the connected server.
  [max_send_msg_size: <int> | default = 16777216]

  # Whether messages should be gzipped.
  [use_gzip_compression: <boolean> | default = false]

  # The rate limit for gRPC clients; 0 means no rate limit.
  [rate_limit: <float64> | default = 0]

  # gRPC burst allowed for rate limits.
  [rate_limit_burst: <int> | default = 0]

  # Controls if when a rate limit is hit whether the client should
  # retry the request.
  [backoff_on_ratelimits: <boolean> | default = false]

  # Configures the retry backoff when backoff_on_ratelimits is
  # true.
  backoff_config:
    # The minimum delay when backing off.
    [min_period: <duration> | default = "100ms"]

    # The maximum delay when backing off.
    [max_period: <duration> | default = "10s"]

    # The number of times to backoff and retry before failing.
    [max_retries: <int> | default = 10]

global_config

The global_config block configures global values for all launched Prometheus instances.

# How frequently should Prometheus instances scrape.
[scrape_interval: duration | default = "1m"]

# How long to wait before timing out a scrape from a target.
[scrape_timeout: duration | default = "10s"]

# A list of static labels to add for all metrics.
external_labels:
  { <string>: <string> }

# Default set of remote_write endpoints. If an instance doesn't define any
# remote_writes, it will use this list.
remote_write:
  - [<remote_write>]

Note: For more information on remote_write, refer to the Prometheus documentation.

The following default values set by Grafana Agent Static Mode are different than the default set by Prometheus:

remote_write: send_exemplars default value is true

remote_write: queue_config: retry_on_http_429 default value is true

metrics_instance_config

The metrics_instance_config block configures an individual metrics instance, which acts as its own mini Prometheus-compatible agent, though without support for the TSDB.

# Name of the instance. Must be present. Will be added as a label to agent
# metrics.
name: string

# Whether this agent instance should only scrape from targets running on the
# same machine as the agent process.
[host_filter: <boolean> | default = false]

# Relabel configs to apply against discovered targets. The relabeling is
# temporary and just used for filtering targets.
host_filter_relabel_configs:
  [ - <relabel_config> ... ]

# How frequently the WAL truncation process should run. Every iteration of
# the truncation will checkpoint old series and remove old samples. If data
# has not been sent within this window, some of it may be lost.
#
# The size of the WAL will increase with less frequent truncations. Making
# truncations more frequent reduces the size of the WAL but increases the
# chances of data loss when remote_write is failing for longer than the
# specified frequency.
[wal_truncate_frequency: <duration> | default = "60m"]

# The minimum amount of time that series and samples should exist in the WAL
# before being considered for deletion. The consumed disk space of the WAL will
# increase by making this value larger.
#
# Setting this value to 0s is valid, but may delete series before all
# remote_write shards have been able to write all data, and may cause errors on
# slower machines.
[min_wal_time: <duration> | default = "5m"]

# The maximum amount of time that series and samples may exist within the WAL
# before being considered for deletion. Series that have not received writes
# since this period will be removed, and all samples older than this period will
# be removed.
#
# This value is useful in long-running network outages, preventing the WAL from
# growing forever.
#
# Must be larger than min_wal_time.
[max_wal_time: <duration> | default = "4h"]

# Deadline for flushing data when a Prometheus instance shuts down
# before giving up and letting the shutdown proceed.
[remote_flush_deadline: <duration> | default = "1m"]

# When true, writes staleness markers to all active series to
# remote_write.
[write_stale_on_shutdown: <boolean> | default = false]

# A list of scrape configuration rules.
scrape_configs:
  - [<scrape_config>]

# A list of remote_write targets.
remote_write:
  - [<remote_write>]

Note: More information on the following types can be found on the Prometheus website:

relabel_config

scrape_config

remote_write

Data retention

The prometheus.remote_write component uses a Write Ahead Log (WAL) to prevent data loss during network outages. The component buffers the received metrics in a WAL for each configured endpoint. The queue shards can use the WAL after the network outage is resolved and flush the buffered metrics to the endpoints.

The WAL records metrics in 128 MB files called segments. To avoid having a WAL that grows on-disk indefinitely, the component truncates its segments on a set interval.

On each truncation, the WAL deletes references to series that are no longer present and also checkpoints roughly the oldest two thirds of the segments (rounded down to the nearest integer) written to it since the last truncation period. A checkpoint means that the WAL only keeps track of the unique identifier for each existing metrics series, and can no longer use the samples for remote writing. If that data has not yet been pushed to the remote endpoint, it is lost.

This behavior dictates the data retention for the prometheus.remote_write component. It also means that it’s impossible to directly correlate data retention directly to the data age itself, as the truncation logic works on segments, not the samples themselves. This makes data retention less predictable when the component receives a non-consistent rate of data.

The WAL block in Flow mode, or the metrics config in Static mode contain some configurable parameters that can be used to control the tradeoff between memory usage, disk usage, and data retention.

The truncate_frequency or wal_truncate_frequency parameter configures the interval at which truncations happen. A lower value leads to reduced memory usage, but also provides less resiliency to long outages.

When a WAL clean-up starts, the most recently successfully sent timestamp is used to determine how much data is safe to remove from the WAL. The min_keepalive_time or min_wal_time controls the minimum age of samples considered for removal. No samples more recent than min_keepalive_time are removed. The max_keepalive_time or max_wal_time controls the maximum age of samples that can be kept in the WAL. Samples older than max_keepalive_time are forcibly removed.

Extended `remote_write` outages

When the remote write endpoint is unreachable over a period of time, the most recent successfully sent timestamp is not updated. The min_keepalive_time and max_keepalive_time arguments control the age range of data kept in the WAL.

If the remote write outage is longer than the max_keepalive_time parameter, then the WAL is truncated, and the oldest data is lost.

Intermittent `remote_write` outages

If the remote write endpoint is intermittently reachable, the most recent successfully sent timestamp is updated whenever the connection is successful. A successful connection updates the series’ comparison with min_keepalive_time and triggers a truncation on the next truncate_frequency interval which checkpoints two thirds of the segments (rounded down to the nearest integer) written since the previous truncation.

Falling behind

If the queue shards cannot flush data quickly enough to keep up-to-date with the most recent data buffered in the WAL, we say that the component is ‘falling behind’. It’s not unusual for the component to temporarily fall behind 2 or 3 scrape intervals. If the component falls behind more than one third of the data written since the last truncate interval, it is possible for the truncate loop to checkpoint data before being pushed to the remote_write endpoint.

WAL corruption

WAL corruption can occur when Grafana Agent unexpectedly stops while the latest WAL segments are still being written to disk. For example, the host computer has a general disk failure and crashes before you can stop Grafana Agent and other running services. When you restart Grafana Agent, it verifies the WAL, removing any corrupt segments it finds. Sometimes, this repair is unsuccessful, and you must manually delete the corrupted WAL to continue.

If the WAL becomes corrupted, Grafana Agent writes error messages such as err="failed to find segment for index" to the log file.

Note
Deleting a WAL segment or a WAL file permanently deletes the stored WAL data.

To delete the corrupted WAL:

Stop Grafana Agent.
Find and delete the contents of the wal directory.

By default the wal directory is a subdirectory of the data-agent directory located in the Grafana Agent working directory. The WAL data directory may be different than the default depending on the wal_directory setting in your Static configuration file or the path specified by the Flow command line flag --storage-path.
Note
There is one wal directory per:
- Metrics instance running in Static mode
- prometheus.remote_write component running in Flow mode
Start Grafana Agent and verify that the WAL is working correctly.

logs_config

Thu, 21 Nov 2024 17:35:09 +0000

logs_config

The logs_config block configures how the Agent collects logs and sends them to a Loki push API endpoint. logs_config is identical to how Promtail is configured, except deprecated fields have been removed and the server_config is not supported.

Refer to the Promtail documentation for the supported values for these fields.

# Directory to store Loki Promtail positions files in. Positions files are
# required to read logs, and are used to store the last read offset of log
# sources. The positions files will be stored in
# <positions_directory>/<logs_instance_config.name>.yml.
#
# Optional only if every config has a positions.filename manually provided.
#
# This directory will be automatically created if it doesn't exist.
[positions_directory: <string>]

# Configure values for all Loki Promtail instances.
[global: <global_config>]

# Loki Promtail instances to run for log collection.
configs:
  - [<logs_instance_config>]

global_config

The global_config block configures global values for all launched Loki Promtail instances.

clients:
  - [<promtail.client_config>]
# Configure how frequently log files from disk get polled for changes.
[file_watch_config: <file_watch_config>]

Note: More information on the following types can be found on the documentation for Promtail:

promtail.client_config

file_watch_config

The file_watch_config block configures how often to poll log files from disk for changes:

# Minimum frequency to poll for files. Any time file changes are detected, the
# poll frequency gets reset to this duration.
  [min_poll_frequency: <duration> | default = "250ms"]
  # Maximum frequency to poll for files. Any time no file changes are detected,
  # the poll frequency doubles in value up to the maximum duration specified by
  # this value.
  #
  # The default is set to the same as min_poll_frequency.
  [max_poll_frequency: <duration> | default = "250ms"]

logs_instance_config

The logs_instance_config block is an individual instance of Promtail with its own set of scrape rules and where to forward logs. It is identical to how Promtail is configured, except deprecated fields have been removed and the server_config block is not supported.

# Name of this config. Required, and must be unique across all Loki configs.
# The name of the config will be the value of a logs_config label for all
# Loki Promtail metrics.
name: <string>

clients:
  - [<promtail.client_config>]

# Optional configuration for where to store the positions files. If
# positions.filename is left empty, the file will be stored in
# <logs_config.positions_directory>/<logs_instance_config.name>.yml.
#
# The directory of the positions file will automatically be created on start up
# if it doesn't already exist..
[positions: <promtail.position_config>]

scrape_configs:
  - [<promtail.scrape_config>]

[target_config: <promtail.target_config>]

[limits_config: <promtail.limits_config>]

Note: More information on the following types can be found on the documentation for Promtail:

promtail.client_config

promtail.scrape_config

promtail.target_config

promtail.limits_config

Note: Backticks in values are not supported.

Note: Because of how YAML treats backslashes in double-quoted strings, all backslashes in a regex expression must be escaped when using double quotes. But because of double processing, in Grafana Agent config file you must use quadruple backslash (\\\\) construction to add backslashes into regular expressions, here is example for name=(\w+)\s regex:

  selector: '{app="my-app"} |~ "name=(\\\\w+)\\\\s"'

Using single or double backslash construction produces the error:

failed to make file target manager: invalid match stage config: invalid selector syntax for match stage: parse error at line 1, col 40: literal not terminated

Using backticks produces the error:

invalid match stage config: invalid selector syntax for match stage: parse error at line 1, col 51: syntax error: unexpected IDENTIFIER, expecting STRING"

traces_config

Wed, 11 Sep 2024 17:43:35 +0000

traces_config

The traces_config block configures a set of Tempo instances, each of which configures its own tracing pipeline. Having multiple configs allows you to configure multiple distinct pipelines, each of which collects spans and sends them to a different location.

Note
If you are using multiple configs, you must manually set port numbers for each receiver, otherwise they will all try to use the same port and fail to start.

configs:
 [ - <traces_instance_config> ... ]

traces_instance_config

# Name configures the name of this Tempo instance. Names must be non-empty and
# unique across all Tempo instances. The value of the name here will appear in
# logs and as a label on metrics.
name: <string>

# This field allows for the general manipulation of tags on spans that pass
# through this agent. A common use may be to add an environment or cluster
# variable.
[ attributes: <attributes.config> ]

# This field allows to configure grouping spans into batches. Batching helps
# better compress the data and reduce the number of outgoing connections
# required transmit the data.
[ batch: <batch.config> ]

remote_write:
  # host:port to send traces to.
  # Here must be the port of gRPC receiver, not the Tempo default port.
  # Example for cloud instances: `tempo-us-central1.grafana.net:443`
  # For local / on-premises instances: `localhost:55680` or `tempo.example.com:14250`
  # Note: for non-encrypted connections you must also set `insecure: true`
  - endpoint: <string>

    # Custom HTTP headers to be sent along with each remote write request.
    # Be aware that 'authorization' header will be overwritten in presence
    # of basic_auth.
    headers:
      [ <string>: <string> ... ]

    # Controls whether compression is enabled.
    [ compression: <string> | default = "gzip" | supported = "none", "gzip"]

    # Controls what protocol to use when exporting traces.
    # Only "grpc" is supported in Grafana Cloud.
    [ protocol: <string> | default = "grpc" | supported = "grpc", "http" ]

    # Controls what format to use when exporting traces, in combination with protocol.
    # protocol/format supported combinations are grpc/otlp and http/otlp.
    # Only grpc/otlp is supported in Grafana Cloud.
    [ format: <string> | default = "otlp" | supported = "otlp" ]

    # Controls whether or not TLS is required. See https://godoc.org/google.golang.org/grpc#WithInsecure
    [ insecure: <boolean> | default = false ]

    # Deprecated in favor of tls_config
    # If both `insecure_skip_verify` and `tls_config.insecure_skip_verify` are used,
    # the latter take precedence.
    [ insecure_skip_verify: <bool> | default = false ]

    # Configures opentelemetry exporters to use the OpenTelemetry auth extension `oauth2clientauthextension`.
    # Can not be used in combination with `basic_auth`.
    # See https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/v0.96.0/extension/oauth2clientauthextension/README.md
    oauth2:
      # Configures the TLS settings specific to the oauth2 client
      # The client identifier issued to the oauth client
      [ client_id: <string> ]
      # The secret string associated with the oauth client
      [ client_secret: <string> ]
      # Additional parameters for requests to the token endpoint
      [ endpoint_params: <string> ]
      # The resource server's token endpoint URL
      [ token_url: <string> ]
      # Optional, requested permissions associated with the oauth client
      [ scopes: [<string>] ]
      # Optional, specifies the timeout fetching tokens from the token_url. Default: no timeout
      [ timeout: <duration> ]
      # TLS client configuration for the underneath client to authorization server.
      # https://github.com/open-telemetry/opentelemetry-collector/blob/v0.96.0/config/configtls/README.md
      tls:
        # Disable validation of the server certificate.
        [ insecure: <bool> | default = false ]
        # InsecureSkipVerify will enable TLS but not verify the certificate.
        [ insecure_skip_verify: <bool> | default = false ]
        # ServerName requested by client for virtual hosting.
        # This sets the ServerName in the TLSConfig. Please refer to
        # https://godoc.org/crypto/tls#Config for more information.
        [ server_name_override: <string> ]
        # Path to the CA cert. For a client this verifies the server certificate. If empty uses system root CA.
        [ ca_file: <string> ]
        # In memory PEM encoded cert.
        [ ca_pem: <string> ]
        # Path to the TLS cert to use for TLS required connections
        [ cert_file: <string> ]
        # In memory PEM encoded TLS cert to use for TLS required connections.
        [ cert_pem: <string> ]
        # Path to the TLS key to use for TLS required connections
        [ key_file: <string> ]
        # In memory PEM encoded TLS key to use for TLS required connections.
        [ key_pem: <string> ]
        # Minimum acceptable TLS version.
        [ min_version: <string> | default = "1.2" ]
        # Maximum acceptable TLS version.
        # If not set, it is handled by crypto/tls - currently it is "1.3".
        [ max_version: <string> | default = "" ]
        # The duration after which the certificate will be reloaded.
        # If not set, it will never be reloaded.
        [ reload_interval: <duration> ]
        # If true, load system CA certificates pool in addition to the certificates
        # configured in this struct.
        [ include_system_ca_certs_pool: <duration> ]
        # A list of TLS cipher suites that the TLS transport can use.
        # If left blank, a safe default list is used.
        # See https://go.dev/src/crypto/tls/cipher_suites.go for a list of supported cipher suites.
        [ cipher_suites: <duration> ]

    # Controls TLS settings of the exporter's client:
    # https://prometheus.io/docs/prometheus/2.45/configuration/configuration/#tls_config
    # This should be used only if `insecure` is set to false
    tls_config:
      # Path to the CA cert. For a client this verifies the server certificate. If empty uses system root CA.
      [ ca_file: <string> ]
      # Path to the TLS cert to use for TLS required connections
      [ cert_file: <string> ]
      # Path to the TLS key to use for TLS required connections
      [ key_file: <string> ]
      # Disable validation of the server certificate.
      [ insecure_skip_verify: <bool> | default = false ]

    # Sets the `Authorization` header on every trace push with the
    # configured username and password.
    # password and password_file are mutually exclusive.
    basic_auth:
      [ username: <string> ]
      [ password: <secret> ]
      [ password_file: <string> ]

    [ sending_queue: <otlpexporter.sending_queue> ]
    [ retry_on_failure: <otlpexporter.retry_on_failure> ]

# This processor writes a well formatted log line to a logs instance for each span, root, or process
# that passes through the Agent. This allows for automatically building a mechanism for trace
# discovery and building metrics from traces using Loki. It should be considered experimental.
automatic_logging:
  # Indicates where the stream of log lines should go. Either supports writing
  # to a logs instance defined in this same config or to stdout.
  [ backend: <string> | default = "stdout" | supported = "stdout", "logs_instance" ]
  # Indicates the logs instance to write logs to.
  # Required if backend is set to logs_instance.
  [ logs_instance_name: <string> ]
  # Log one line per span. Warning! possibly very high volume
  [ spans: <boolean> ]
  # Log one line for every root span of a trace.
  [ roots: <boolean> ]
  # Log one line for every process
  [ processes: <boolean> ]
  # Additional span attributes to log
  [ span_attributes: <string array> ]
  # Additional process attributes to log
  [ process_attributes: <string array> ]
  # Timeout on writing logs to Loki when backend is "logs_instance."
  [ timeout: <duration> | default = 1ms ]
  # Configures a set of key values that will be logged as labels
  # They need to be span or process attributes logged in the log line
  #
  # This feature only applies when `backend = logs_instance`
  #
  # Loki only accepts alphanumeric and "_" as valid characters for labels.
  # Labels are sanitized by replacing invalid characters with underscores.
  [ labels: <string array> ]
  overrides:
    [ logs_instance_tag: <string> | default = "traces" ]
    [ service_key: <string> | default = "svc" ]
    [ span_name_key: <string> | default = "span" ]
    [ status_key: <string> | default = "status" ]
    [ duration_key: <string> | default = "dur" ]
    [ trace_id_key: <string> | default = "tid" ]

# Receiver configurations are mapped directly into the OpenTelemetry receivers
# block. At least one receiver is required.
# The Agent uses OpenTelemetry v0.96.0. Refer to the corresponding receiver's config.
#
# Supported receivers: otlp, jaeger, kafka, opencensus and zipkin.
receivers: <receivers>

# A list of prometheus scrape configs.  Targets discovered through these scrape
# configs have their __address__ matched against the ip on incoming spans. If a
# match is found then relabeling rules are applied.
scrape_configs:
  [ - <scrape_config> ... ]
# Defines what method is used when adding k/v to spans.
# Options are `update`, `insert` and `upsert`.
# `update` only modifies an existing k/v and `insert` only appends if the k/v
# is not present. `upsert` does both.
[ prom_sd_operation_type: <string> | default = "upsert" ]
# Configures what methods to use to do association between spans and pods.
# PromSD processor matches the IP address of the metadata labels from the k8s API
# with the IP address obtained from the specified pod association method.
# If a match is found then the span is labeled.
#
# Options are `ip`, `net.host.ip`, `k8s.pod.ip`, `hostname` and `connection`.
#   - `ip`, `net.host.ip` and `k8s.pod.ip`, `hostname` match spans tags.
#   - `connection` inspects the context from the incoming requests (gRPC and HTTP).
#
# Tracing instrumentation is commonly the responsible for tagging spans
# with IP address to the labels mentioned above.
# If running on kubernetes, `k8s.pod.ip` can be automatically attached via the
# downward API. For example, if you're using OTel instrumentation libraries, set
# OTEL_RESOURCE_ATTRIBUTES=k8s.pod.ip=$(POD_IP) to inject spans with the sender
# pod's IP.
#
# By default, all methods are enabled, and evaluated in the order specified above.
# Order of evaluation is honored when multiple methods are enabled.
prom_sd_pod_associations:
  [ - <string> ... ]

# spanmetrics supports aggregating Request, Error and Duration (R.E.D) metrics
# from span data.
#
# spanmetrics generates two metrics from spans and uses remote_write or
# OpenTelemetry Prometheus exporters to serve the metrics locally.
#
# In order to use the remote_write exporter, you have to configure a Prometheus
# instance in the Agent and pass its name to the `metrics_instance` field.
#
# If you want to use the OpenTelemetry Prometheus exporter, you have to
# configure handler_endpoint and then scrape that endpoint.
#
# The first generated metric is `calls`, a counter to compute requests.
# The second generated metric is `latency`, a histogram to compute the
# operation's duration.
#
# If you want to rename the generated metrics, you can configure the `namespace`
# option of prometheus exporter.
#
# This is an experimental feature of Opentelemetry-Collector and the behavior
# may change in the future.
spanmetrics:
  # latency_histogram_buckets and dimensions are the same as the configs in
  # spanmetricsprocessor.
  [ latency_histogram_buckets: <spanmetricsprocessor.latency_histogram_buckets> ]
  [ dimensions: <spanmetricsprocessor.dimensions> ]
  # const_labels are labels that will always get applied to the exported
  # metrics.
  const_labels:
    [ <string>: <string> ... ]
  # Metrics are namespaced to `traces_spanmetrics` by default.
  # They can be further namespaced, i.e. `{namespace}_traces_spanmetrics`
  [ namespace: <string> ]
  # metrics_instance is the metrics instance used to remote write metrics.
  [ metrics_instance: <string> ]
  # handler_endpoint defines the endpoint where the OTel prometheus exporter will be exposed.
  [ handler_endpoint: <string> ]
  # dimensions_cache_size defines the size of cache for storing Dimensions.
  [ dimensions_cache_size: <int> | default = 1000 ]
  # aggregation_temporality configures whether to reset the metrics after flushing.
  # It can be either AGGREGATION_TEMPORALITY_CUMULATIVE or AGGREGATION_TEMPORALITY_DELTA.
  [ aggregation_temporality: <string> | default = "AGGREGATION_TEMPORALITY_CUMULATIVE" ]
  # metrics_flush_interval configures how often to flush generated metrics.
  [ metrics_flush_interval: <duration> | default = 15s ]

# tail_sampling supports tail-based sampling of traces in the agent.
#
# Policies can be defined that determine what traces are sampled and sent to the
# backends and what traces are dropped.
#
# In order to make a correct sampling decision it's important that the agent has
# a complete trace. This is achieved by waiting a given time for all the spans
# before evaluating the trace.
#
# Tail sampling also supports multi agent deployments, allowing to group all
# spans of a trace in the same agent by load balancing the spans by trace ID
# between the instances.
# * To make use of this feature, check load_balancing below *
tail_sampling:
  # policies define the rules by which traces will be sampled. Multiple policies
  # can be added to the same pipeline.
  policies:
    [ - <tailsamplingprocessor.policies> ... ]

  # Time that to wait before making a decision for a trace.
  # Longer wait times reduce the probability of sampling an incomplete trace at
  # the cost of higher memory usage.
  [ decision_wait: <duration> | default = 5s ]

  # Optional, number of traces kept in memory
  [ num_traces: <int> | default = 50000 ]

  # Optional, expected number of new traces (helps in allocating data structures)
  [ expected_new_traces_per_sec: <int> | default = 0 ]

# load_balancing configures load balancing of spans across multi agent deployments.
# It ensures that all spans of a trace are sampled in the same instance.
# It works by exporting spans based on their traceID via consistent hashing.
#
# Enabling this feature is required for "tail_sampling", "spanmetrics", and "service_graphs"
# to correctly work when spans are ingested by multiple agent instances.
#
# Load balancing works by layering two pipelines and consistently exporting
# spans belonging to a trace to the same agent instance.
# Agent instances need to be able to communicate with each other via gRPC.
#
# When load_balancing is enabled:
# 1. When an Agent receives spans from the configured "receivers".
# 2. If the "attributes" processor is configured, it will run through all the spans.
# 3. The spans will be exported using the "load_balancing" configuration to any of the Agent instances.
#    This may or may not be the same Agent which has already received the span.
# 4. The Agent which received the span from the loadbalancer will run these processors, 
#    in this order, if they are configured:
#    1. "spanmetrics"
#    2. "service_graphs"
#    3. "tail_sampling"
#    4. "automatic_logging"
#    5. "batch"
# 5. The spans are then remote written using the "remote_write" configuration.
# 
# Load balancing significantly increases CPU usage. This is because spans are
# exported an additional time between agents.
load_balancing:
  # resolver configures the resolution strategy for the involved backends
  # It can be either "static", "dns" or "kubernetes".
  resolver:
    static:
      # A fixed list of hostnames.
      hostnames:
        [ - <string> ... ]
    dns:
      # DNS hostname from which to resolve IP addresses.
      hostname: <string>
      # Port number to use with the resolved IP address when exporting spans.
      [ port: <int> | default = 4317 ]
      # Resolver interval
      [ interval: <duration> | default = 5s ]
      # Resolver timeout
      [ timeout: <duration> | default = 1s ]
    # The kubernetes resolver receives IP addresses of a Kubernetes service 
    # from the Kubernetes API. It does not require polling. The Kubernetes API
    # notifies the Agent when a new pod is available and when an old pod has exited.
    #
    # For the kubernetes resolver to work, Agent must be running under
    # a system account with "list", "watch" and "get" permissions.
    kubernetes:
      service: <string>
      [ ports: <int array> | default = 4317 ]

  # routing_key can be either "traceID" or "service":
  # * "service": exports spans based on their service name.
  # * "traceID": exports spans based on their traceID.
  [ routing_key: <string> | default = "traceID" ]

  # receiver_port is the port the instance will use to receive load balanced traces
  receiver_port: [ <int> | default = 4318 ]

  # Load balancing is done via an otlp exporter.
  # The remaining configuration is common with the remote_write block.
  exporter:
    # Controls whether compression is enabled.
    [ compression: <string> | default = "gzip" | supported = "none", "gzip"]

    # Controls whether or not TLS is required.
    [ insecure: <boolean> | default = false ]

    # Disable validation of the server certificate. Only used when insecure is set
    # to false.
    [ insecure_skip_verify: <bool> | default = false ]

    # Sets the `Authorization` header on every trace push with the
    # configured username and password.
    # password and password_file are mutually exclusive.
    basic_auth:
      [ username: <string> ]
      [ password: <secret> ]
      [ password_file: <string> ]

# service_graphs configures processing of traces for building service graphs in
# the form of prometheus metrics. The generated metrics represent edges between
# nodes in the graph. Nodes are represented by `client` and `server` labels.
#
#  e.g. tempo_service_graph_request_total{client="app", server="db"} 20
#
# Service graphs works by inspecting spans and looking for the tag `span.kind`.
# If it finds the span kind to be client or server, it stores the request in a
# local in-memory store.
#
# That request waits until its corresponding client or server pair span is
# processed or until the maximum waiting time has passed.
# When either of those conditions is reached, the request is processed and
# removed from the local store. If the request is complete by that time, it'll
# be recorded as an edge in the graph.
#
# Service graphs supports multi-agent deployments, allowing to group all spans
# of a trace in the same agent by load balancing the spans by trace ID between
# the instances.
# * To make use of this feature, check load_balancing above *
service_graphs:
  [ enabled: <bool> | default = false ]

  # configures the time the processor will wait since a span is consumed until
  # it's considered expired if its paired has not been processed.
  #
  # increasing the waiting time will increase the percentage of paired spans.
  # retaining unpaired spans for longer will make reaching max_items more likely.
  [ wait: <duration> | default = 10s ]

  # configures the max amount of edges that will be stored in memory.
  #
  # spans that arrive to the processor that do not pair with an already
  # processed span are dropped.
  #
  # a higher max number of items increases the max throughput of processed spans
  # with a higher memory consumption.
  [ max_items: <integer> | default = 10_000 ]

  # configures the number of workers that will process completed edges concurrently.
  # as edges are completed, they get queued to be collected as metrics for the graph.
  [ workers: <integer> | default = 10 ]

  # configures what status codes are considered as successful (e.g. HTTP 404).
  #
  # by default, a request is considered failed in the following cases:
  #   1. HTTP status is not 2XX
  #   1. gRPC status code is not OK
  #   1. span status is Error
  success_codes:
    # http status codes not to be considered as failure
    http:
      [ - <int> ... ]
    # grpc status codes not to be considered as failure
    grpc:
      [ - <int> ... ]

# jaeger_remote_sampling configures one or more jaeger remote sampling extensions.
# For more details about the configuration please consult the OpenTelemetry documentation:
# https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/v0.96.0/extension/jaegerremotesampling
#
# Example config:
#
# jaeger_remote_sampling:
#   - source:
#       remote:
#         endpoint: jaeger-collector:14250
#         tls:
#           insecure: true
#   - source:
#       reload_interval: 1s
#       file: /etc/otelcol/sampling_strategies.json
#
jaeger_remote_sampling:
  [ - <jaeger_remote_sampling> ... ]

More information on the following types can be found on the documentation for their respective projects:

integrations_config

Wed, 11 Sep 2024 17:43:35 +0000

integrations_config

The integrations_config block configures how the Agent runs integrations that scrape and send metrics without needing to run specific Prometheus exporters or manually write scrape_configs:

# Controls the Agent integration
agent:
  # Enables the Agent integration, allowing the Agent to automatically
  # collect and send metrics about itself.
  [enabled: <boolean> | default = false]

  # Sets an explicit value for the instance label when the integration is
  # self-scraped. Overrides inferred values.
  #
  # The default value for this integration is inferred from the agent hostname
  # and HTTP listen port, delimited by a colon.
  [instance: <string>]

  # Automatically collect metrics from this integration. If disabled,
  # the agent integration will be run but not scraped and thus not
  # remote_written. Metrics for the integration will be exposed at
  # /integrations/agent/metrics and can be scraped by an external process.
  [scrape_integration: <boolean> | default = <integrations_config.scrape_integrations>]

  # How often should the metrics be collected? Defaults to
  # prometheus.global.scrape_interval.
  [scrape_interval: <duration> | default = <global_config.scrape_interval>]

  # The timeout before considering the scrape a failure. Defaults to
  # prometheus.global.scrape_timeout.
  [scrape_timeout: <duration> | default = <global_config.scrape_timeout>]

  # How frequent to truncate the WAL for this integration.
  [wal_truncate_frequency: <duration> | default = "60m"]

  # Allows for relabeling labels on the target.
  relabel_configs:
    [- <relabel_config> ... ]

  # Relabel metrics coming from the integration, allowing to drop series
  # from the integration that you don't care about.
  metric_relabel_configs:
    [ - <relabel_config> ... ]

# Client TLS Configuration
# Client Cert/Key Values need to be defined if the server is requesting a certificate
#  (Client Auth Type = RequireAndVerifyClientCert || RequireAnyClientCert).
http_tls_config: <tls_config>

# Controls the apache_http integration
apache_http: <apache_http_config>

# Controls the node_exporter integration
node_exporter: <node_exporter_config>

# Controls the process_exporter integration
process_exporter: <process_exporter_config>

# Controls the mysqld_exporter integration
mysqld_exporter: <mysqld_exporter_config>

# Controls the oracledb integration
oracledb: <oracledb_config>

# Controls the redis_exporter integration
redis_exporter: <redis_exporter_config>

# Controls the dnsmasq_exporter integration
dnsmasq_exporter: <dnsmasq_exporter_config>

# Controls the elasticsearch_exporter integration
elasticsearch_exporter: <elasticsearch_exporter_config>

# Controls the memcached_exporter integration
memcached_exporter: <memcached_exporter_config>

# Controls the mssql integration
mssql: <mssql_config>

# Controls the postgres_exporter integration
postgres_exporter: <postgres_exporter_config>

# Controls the snmp_exporter integration
snmp_exporter: <snmp_exporter_config>

# Controls the snowflake integration
snowflake: <snowflake_config>

# Controls the statsd_exporter integration
statsd_exporter: <statsd_exporter_config>

# Controls the consul_exporter integration
consul_exporter: <consul_exporter_config>

# Controls the windows_exporter integration
windows_exporter: <windows_exporter_config>

# Controls the kafka_exporter integration
kafka_exporter: <kafka_exporter_config>

# Controls the mongodb_exporter integration
mongodb_exporter: <mongodb_exporter_config>

# Controls the github_exporter integration
github_exporter: <github_exporter_config>

# Controls the blackbox_exporter integration
blackbox: <blackbox_config>

# Controls the CloudWatch exporter integration
cloudwatch_exporter: <cloudwatch_exporter_config>

# Controls the azure_exporter integration
azure_exporter: <azure_exporter_config>

# Controls the gcp_exporter integration
gcp_exporter: <gcp_exporter_config>

# Controls the squid integration
squid: <squid_config>

# Automatically collect metrics from enabled integrations. If disabled,
# integrations will be run but not scraped and thus not remote_written. Metrics
# for integrations will be exposed at /integrations/<integration_key>/metrics
# and can be scraped by an external process.
[scrape_integrations: <boolean> | default = true]

# Extra labels to add to all samples coming from integrations.
labels:
  { <string>: <string> }

# The period to wait before restarting an integration that exits with an
# error.
[integration_restart_backoff: <duration> | default = "5s"]

# A list of remote_write targets. Defaults to global_config.remote_write.
# If provided, overrides the global defaults.
prometheus_remote_write:
  - [<remote_write>]

Scraping service (Beta)

Wed, 11 Sep 2024 17:43:35 +0000

Scraping service (Beta)

The Grafana Agent scraping service allows you to cluster a set of Agent processes and distribute the scrape load.

Determining what to scrape is done by writing instance configuration files to an API, which then stores the configuration files in a KV store backend. All agents in the cluster must use the same KV store to see the same set of configuration files.

Each process of the Grafana Agent can be running multiple independent “instances” at once, where an “instance” refers to the combination of:

Service discovery for all scrape_configs within that loaded configuration
Scrapes metrics from all discovered targets
Stores data in its own Write-Ahead Log specific to the loaded configuration
Remote Writes scraped metrics to the configured remote_write destinations specified within the loaded configuration.

The “instance configuration file,” then, is the configuration file that specifies the set of scrape_configs and remote_write endpoints. For example, a small instance configuration file looks like this:

scrape_configs:
  - job_name: self-scrape
    static_configs:
      - targets: ['localhost:9090']
        labels:
          process: 'agent'
remote_write:
  - url: http://cortex:9009/api/prom/push

The full set of supported options for an instance configuration file is available in the metrics-config.md file.

Multiple instance configuration files are necessary for sharding. Each config file is distributed to a particular agent on the cluster based on the hash of its contents.

When the scraping service is enabled, Agents disallow specifying instance configurations locally in the configuration file; using the KV store is required. agentctl can be used to manually sync instance configuration files to the Agent’s API server.

Distributed hash ring

The scraping service uses a Distributed Hash Ring (commonly called “the ring”) to cluster agents and to shard configurations within that ring. Each Agent joins the ring with a random distinct set of tokens that are used for sharding. The default number of generated tokens is 128.

The Distributed Hash Ring is also stored in a KV store. Since a KV store is also needed for storing configuration files, it is encouraged to re-use the same KV store for the ring.

When sharding, the Agent currently uses the name of a configuration file stored in the KV store for load distribution. Configuration names are guaranteed to be unique keys. The hash of the name is used as the lookup key in the ring and determines which agent (based on token) should be responsible for that configuration. “Price is Right” rules are used for the Agent lookup; the Agent owning the token with the closest value to the key without going over is responsible for the configuration.

All Agents are simultaneously watching the KV store for changes to the set of configuration files. When a configuration file is added or updated in the configuration store, each Agent will run the configuration name hash through their copy of the Hash Ring to determine if they are responsible for that config.

When an Agent receives a new configuration that it is responsible for, it launches a new instance from the instance configuration. If a configuration is deleted from the KV store, this will be detected by the owning Agent, and it will stop the metric collection process for that configuration file.

When an Agent receives an event for an updated configuration file that they used to be the owner of but are no longer the owner, the associated instance for that configuration file is stopped for that Agent. This can happen when the cluster size changes.

The scraping service currently does not support replication. Only one agent at a time will be responsible for scraping a certain configuration.

Resharding

When a new Agent joins or leaves the cluster, the set of tokens in the ring may cause configurations to hash to a new Agent. The process of responding to this action is called “resharding.”

Resharding is run:

When an Agent joins the ring
When an Agent leaves the ring
When the KV store sends a notification indicating a configuration has changed.
On a specified interval if KV change events have not fired.

The resharding process involves each Agent retrieving the full set of configurations stored in the KV store and determining if:

The configuration owned by the current resharding Agent has changed and needs to be reloaded.
The configuration is no longer owned by the current resharding Agent and the associated instance should be stopped.
The configuration has been deleted, and the associated instance should be stopped.

Best practices

Because distribution is determined by the number of configuration files and not how many targets exist per configuration file, the best amount of distribution is achieved when each configuration file has the lowest amount of targets possible. The best distribution will be achieved if each configuration file stored in the KV store is limited to one static configuration with only one target.

Example

Here’s an example agent.yaml configuration file that uses the same etcd server for both configuration storage and the distributed hash ring storage:

server:
  log_level: debug

metrics:
  global:
    scrape_interval: 1m
  scraping_service:
    enabled: true
    kvstore:
      store: etcd
      etcd:
        endpoints:
          - etcd:2379
    lifecycler:
      ring:
        replication_factor: 1
        kvstore:
          store: etcd
          etcd:
            endpoints:
              - etcd:2379

Note that there are no instance configurations present in this example; instance configurations must be passed to the API for the Agent to start scraping metrics.

agentctl

agentctl is a tool included with this repository that helps users interact with the new Config Management API. The agentctl config-sync subcommand uses local YAML files as a source of truth and syncs their contents with the API. Entries in the API not in the synced directory will be deleted.

agentctl is distributed in binary form with each release and as a Docker container with the grafana/agentctl image. Tanka configurations that utilize grafana/agentctl and sync a set of configurations to the API are planned for the future.

Debug Ring endpoint

You can use the /debug/ring endpoint to troubleshoot issues with the scraping service in Scraping Service Mode. It provides information about the Distributed Hash Ring and the current distribution of configurations among Agents in the cluster. It also allows you to forget an instance in the ring manually.

You can access this endpoint by making an HTTP request to the Agent’s API server.

Information returned by the /debug/ring endpoint includes:

The list of Agents in the cluster, and their respective tokens used for sharding.
The list of configuration files in the KV store and associated hash values used for lookup in the ring.
The unique instance ID assigned to each instance of the Agent running in the cluster. The instance ID is a unique identifier assigned to each running instance of the Agent within the cluster. The exact details of the instance ID generation might be specific to the implementation of the Grafana Agent.
The time of the “Last Heartbeat” of each instance. The Last Heartbeat is the last time the instance was active in the ring.

Agent Management - Experimental

Wed, 11 Sep 2024 17:43:35 +0000

Agent Management - Experimental

Agent Management is under active development. Backwards incompatible changes to its API are to be expected. Feedback is much appreciated. This is a feature that MAY NOT make it production.

Agent Management enables centralized management of fleets of Grafana Agents.

In this mode, Grafana Agent polls and dynamically reloads its configuration from a remote API server.

Remote Configurations are composed of a base configuration and a set of snippets. Snippets are applied conditionally via label matching.

Configuration

Agent Management can be used by passing the flag -enable-features=agent-management. When enabled, the file referred to -config.file will be loaded as an agent management configuration file.

Agent Management configuration files are YAML documents which conform the following schema:

# Agent Management configuration.
agent_management:
  # Host of the API server to connect to.
  host: <string>

  # Protocol to use when connecting to the API server (http|https).
  protocol: <string>

  # The polling interval for fetching the configuration.
  polling_interval: <string>

  # Sets the `Authorization` header on every request with the
  # configured username and password.
  basic_auth:
    [ username: <string> ]
    [ password_file: <string> ]

  # Optional proxy URL.
  [ proxy_url: <string> ]

  # Comma-separated string that can contain IPs, CIDR notation, domain names
  # that should be excluded from proxying. IP and domain names can
  # contain port numbers.
  [ no_proxy: <string> ]

  # Use proxy URL indicated by environment variables (HTTP_PROXY, https_proxy, HTTPs_PROXY, https_proxy, and no_proxy)
  [ proxy_from_environment: <boolean> | default: false ]

  # Specifies headers to send to proxies during CONNECT requests.
  [ proxy_connect_header:
    [ <string>: [<secret>, ...] ] ]

  # Fields specific to remote configuration.
  remote_configuration:
    # A path to a directory where the remote configuration will be cached. The directory must be writeable.
    cache_location: <string>

    # The namespace to use.
    namespace: <string>

    # Set of self-identifying labels used for snippet selection.
    labels:
      [ <labelname>: <labelvalue> ... ]

    # Whether to use labels from the label management service. If enabled, labels from the API supersede the ones configured in the agent. The agent_id field must be defined.
    label_management_enabled: <bool> | default = false

    # A unique ID for the agent, which is used to identify the agent.
    agent_id: <string>

    # Whether to accept HTTP 304 Not Modified responses from the API server. If enabled, the agent will use the cached configuration if the API server responds with HTTP 304 Not Modified. You can set this argument to `false` for debugging or testing.
    accept_http_not_modified: <bool> | default = true

API

Grafana Agents with Agent Management enabled continuously poll the API server for an up-to-date configuration. The API server is expected to implement a GET /agent-management/api/agent/v2/namespace/:namespace/remote_config HTTP endpoint returning a successful response with the following body format:

# The base configuration for the Agent.
base_config: |
  <grafana_agent_config>
# A set of snippets to be conditionally merged into the base configuration.
snippets:
  [ <snippet_name>: <snippet_content> ... ]

grafana_agent_config

This is a standard Grafana Agent static mode configuration. Typically used to configure the server, remote_writes, and other global configuration.

snippet_content

The snippet content is a YAML document which conforms to the following schema:

# Config provides the actual snippet configuration.
config: |
  [metrics_scrape_configs]:
  - [<scrape_config> ... ]
  [logs_scrape_configs]:
  - [<promtail.scrape_config> ... ]
  [integration_configs]:
    [<integrations_config> ... ]
# Selector is a set of labels used to decide which snippets to apply to the final configuration.
selector:
  [ <labelname>: <labelvalue> ... ]

Note: More information on the following types can be found in their respective documentation pages:

scrape_config

promtail.scrape_config

integrations_config

Note: Snippet selection is currently done in the API server. This behaviour is subject to change in the future.

Example response body

base_config: |
  server:
    log_level: info
  metrics:
    global:
      remote_write:
        - basic_auth:
            password_file: key.txt
            username: 123
          url: https://myserver.com/api/prom/push
  logs:
    positions_directory: /var/lib/grafana-agent
    global:
      clients:
        - basic_auth:
            password_file: key.txt
            username: 456
          url: https://myserver.com/loki/api/v1/push
snippets:
  snip1:
    config: |
      metrics_scrape_configs:
      - job_name: 'prometheus'
        scrape_interval: 60s
        static_configs:
        - targets: ['localhost:9090']
      logs_scrape_configs:
      - job_name: 'loki'
        static_configs:
        - targets: ['localhost:3100']
      integration_configs:
        node_exporter:
          enabled: true
    selector:
      os: linux
      app: app1

Note: Base configurations and snippets can contain go’s text/template actions. If you need preserve the literal value of a template action, you can escape it using backticks. For example:

{{ `{{ .template_var }}` }}

Configure static mode on Grafana Labs

Create a configuration file

Create a configuration file

Integrations

Prometheus config/migrating from Prometheus

Loki Config/Migrating from Promtail

Full config example

Command-line flags

Command-line flags

Basic

Experimental feature flags

Report information usage

Support bundles

Configuration file

Remote Configuration

Server

TLS Support

Metrics

server_config

server_config

server_tls_config

windows_certificate_filter_config

windows_client_config

windows_server_config

metrics_config

metrics_config

scraping_service_config

kvstore_config

lifecycler_config

scraping_service_client_config

global_config

metrics_instance_config

Data retention

Extended remote_write outages

Intermittent remote_write outages

Falling behind

WAL corruption

logs_config

logs_config

global_config

file_watch_config

logs_instance_config

traces_config

traces_config

traces_instance_config

integrations_config

integrations_config

Scraping service (Beta)

Scraping service (Beta)

Distributed hash ring

Resharding

Best practices

Example

agentctl

Debug Ring endpoint

Agent Management - Experimental

Agent Management - Experimental

Configuration

API

grafana_agent_config

snippet_content

Example response body

Extended `remote_write` outages

Intermittent `remote_write` outages