Agent Management - Experimental
Agent Management is under active development. Backwards incompatible changes to its API are to be expected. Feedback is much appreciated. This is a feature that MAY NOT make it production.
Agent Management enables centralized management of fleets of Grafana Agents.
In this mode, Grafana Agent polls and dynamically reloads its configuration from a remote API server.
Remote Configurations are composed of a base configuration and a set of snippets. Snippets are applied conditionally via label matching.
Agent Management can be used by passing the flag
-enable-features=agent-management. When enabled, the file referred to
-config.file will be loaded as an agent management configuration file.
Agent Management configuration files are YAML documents which conform the following schema:
# Agent Management configuration. agent_management: # Host of the API server to connect to. host: <string> # Protocol to use when connecting to the API server (http|https). protocol: <string> # The polling interval for fetching the configuration. polling_interval: <string> # Sets the `Authorization` header on every request with the # configured username and password. basic_auth: [ username: <string> ] [ password_file: <string> ] # Optional proxy URL. [ proxy_url: <string> ] # Comma-separated string that can contain IPs, CIDR notation, domain names # that should be excluded from proxying. IP and domain names can # contain port numbers. [ no_proxy: <string> ] # Use proxy URL indicated by environment variables (HTTP_PROXY, https_proxy, HTTPs_PROXY, https_proxy, and no_proxy) [ proxy_from_environment: <boolean> | default: false ] # Specifies headers to send to proxies during CONNECT requests. [ proxy_connect_header: [ <string>: [<secret>, ...] ] ] # Fields specific to remote configuration. remote_configuration: # A path to a directory where the remote configuration will be cached. The directory must be writeable. cache_location: <string> # The namespace to use. namespace: <string> # Set of self-identifying labels used for snippet selection. labels: [ <labelname>: <labelvalue> ... ] # Whether to use labels from the label management service. If enabled, labels from the API supersede the ones configured in the agent. label_management_enabled: <bool> | default = false # A unique ID for the agent, which is used to identify the agent. agent_id: <string>
Grafana Agents with Agent Management enabled continuously poll the API server for an up-to-date configuration. The API server is expected to implement a
GET /agent-management/api/agent/v2/namespace/:namespace/remote_config HTTP endpoint returning a successful response with the following body format:
# The base configuration for the Agent. base_config: | <grafana_agent_config> # A set of snippets to be conditionally merged into the base configuration. snippets: [ <snippet_name>: <snippet_content> ... ]
This is a standard Grafana Agent static mode configuration. Typically used to configure the server, remote_writes, and other global configuration.
The snippet content is a YAML document which conforms to the following schema:
# Config provides the actual snippet configuration. config: | [metrics_scrape_configs]: - [<scrape_config> ... ] [logs_scrape_configs]: - [<promtail.scrape_config> ... ] [integration_configs]: [<integrations_config> ... ] # Selector is a set of labels used to decide which snippets to apply to the final configuration. selector: [ <labelname>: <labelvalue> ... ]
Note: More information on the following types can be found in their respective documentation pages:
Note: Snippet selection is currently done in the API server. This behaviour is subject to change in the future.
Example response body
base_config: | server: log_level: info metrics: global: remote_write: - basic_auth: password_file: key.txt username: 123 url: https://myserver.com/api/prom/push logs: positions_directory: /var/lib/grafana-agent global: clients: - basic_auth: password_file: key.txt username: 456 url: https://myserver.com/loki/api/v1/push snippets: snip1: config: | metrics_scrape_configs: - job_name: 'prometheus' scrape_interval: 60s static_configs: - targets: ['localhost:9090'] logs_scrape_configs: - job_name: 'loki' static_configs: - targets: ['localhost:3100'] integration_configs: node_exporter: enabled: true selector: os: linux app: app1