Dynamic Configuration on Grafana Labs

template_paths:
  - "file:///etc/grafana/01_assets"

Tells the Grafana Agent where to load files from. It is important to note that dynamic configuration does NOT traverse directories. It will look at the directory specified only, if you need more directories then add them to the template_paths array. NOTE, if no protocol specified ie file:// above, then file access will be assumed. file:///etc/grafana/01_assets is equivalent to //etc/grafana/01_assets

Agent

Dynamic Configuration will find the first file matching pattern agent-*.yml and load that as the base. You can only have one agent template. If multiple matching templates are found then the configuration will fail to load.

server:
  log_level: debug
metrics:
  wal_directory: /tmp/grafana-agent-normal
  global:
    scrape_interval: 60s
    remote_write:
      - url: https://prometheus-us-central1.grafana.net/api/prom/push
        basic_auth:
          username: xyz
          password: secretpassword
integrations:
  node_exporter:
    enabled: true
  agent:
    enabled: true

Server

Dynamic configuration will find the first file matching pattern server-*.yml and replace the Server config block in the Agent Configuration. Note that you do NOT include the server: tag, dynamic configuration knows by the name that it is a configuration block.

You can only have 1 server template.

server-1.yml

log_level: info

Final

In the above example the log_level: debug block will be replaced with log_level: info from the server-1.yml

Instances

Thu, 27 Apr 2023 15:07:06 +0000

02 Instances

Dynamic configuration allows multiple prometheus instances to be loaded with a parent metric. This uses the same agent-1 and server-1 yml from 01.

docker run -v ${PWD}/:/etc/grafana grafana/agentctl:latest template-parse file:///etc/grafana/02_config.yml

Dynamic Configuration

Tells the Grafana Agent where to load files from.

Metrics

Dynamic Configuration will find the first file matching pattern metrics-*.yml and load that as the base. You can only have one metrics template.

metrics-1.yml

configs:
  - name: default
global:
  scrape_interval: 60s
  scrape_timeout: 20s
wal_directory: /tmp/grafana-agent-wal

Metrics Instances

You can have any number of metrics_instances and they are added to any existing metrics instances defined previously.

metrics_instances-1.yml

name: instance1
scrape_configs:
  - job_name: instance1_job
    static_configs:
      - targets:
          - localhost:4000

metrics_instances-2.yml

name: instance2
scrape_configs:
  - job_name: instance2_job
    static_configs:
      - targets:
          - localhost:5555

Final

In the above you will see the final.yml includes all the instance configurations

default
instance1
instance2

Integrations

Thu, 27 Apr 2023 15:07:06 +0000

03 Integrations

Dynamic configuration requires the use of integrations-next feature flag, to allow arrays of integrations. In this we will load integrations of various types. This is all built on the previous examples.

docker run -v ${PWD}/:/etc/grafana grafana/agentctl:latest template-parse file:///etc/grafana/03_config.yml

Dynamic Configuration

Tells the Grafana Agent where to load files from.

Integrations

Integrations are loaded from files matching integrations-*.yml and are combined together. You can declare for example multiple sets of redis_configs across several files.

integrations-node.yml

Note: You do NOT have to name the above file integrations-node.yml with node, integrations-1.yml would work the same. The name does NOT determine the type of integrations a template can contain and a template can contain integrations of different types.

node_exporter: {}

integrations-redis.yml

redis_configs:
  - redis_addr: localhost:6379
    autoscrape:
      metric_relabel_configs:
        - source_labels: [__address__]
          target_label: "banana"
          replacement: "apple"
  - redis_addr: localhost:6380

Final

The final result should have 3 integrations enabled, 1 node_exporter and 2 redis.

Logs and Traces

Thu, 27 Apr 2023 15:07:06 +0000

04 Logs and Traces

Logs and Traces can also be templated. This is built ontop of the previous examples.

docker run -v ${PWD}/:/etc/grafana grafana/agentctl:latest template-parse file:///etc/grafana/04_config.yml

Dynamic Configuration

Tells the Grafana Agent where to load files from.

Logs

Logs are loaded from a template matching logs-*.yml. There can ONLY be 1 template loaded

logs-1.yml

configs:
  - name: test_logs
    positions:
      filename: /tmp/positions.yaml
    scrape_configs:
      - job_name: test
        pipeline_stages:
          - regex:
            source: filename
            expression: '\\temp\\Logs\\(?P<log_app>.+?)\\'

traces.yml

configs:
  - name: test_traces
    automatic_logging:
      backend: stdout
      loki_name: default
      spans: true

Final

Looping

Thu, 27 Apr 2023 15:07:06 +0000

01 Basics of Templating

The templating is based on the excellent gomplate library. Currently using a custom fork to allow loading gomplate as a library in addition to some new commands. This will NOT try to cover the full range of gomplate, would recommend reading the documentation for full knowledge.

docker run -v ${PWD}/:/etc/grafana grafana/agentctl:latest template-parse file:///etc/grafana/01_config.yml

Looping

server:
  log_level: debug
metrics:
  wal_directory: /tmp/grafana-agent-normal
  global:
    scrape_interval: 60s
    remote_write:
      - url: https://prometheus-us-central1.grafana.net/api/prom/push
        basic_auth:
          username: xyz
          password: secretpassword
  configs:
    - name: default
  {{ range slice "apple" "banana" "pear" }}
    - name: {{ . }}
  {{ end }}

The templating engine uses directives that are wrapped in {{ command }}, in the above the dynamic configuration engine will loop over the three values, and those values can be accessed by {{ . }} which means current value.

Final

The final.yml contains 4 prometheus configs

default
apple
banana
pear

Datasources

Thu, 27 Apr 2023 15:07:06 +0000

02 Datasources

Datasources are a powerful concept in gomplate. They allow you to reach out to other files, systems and resources to pull data.

docker run -v ${PWD}/:/etc/grafana grafana/agentctl:latest template-parse file:///etc/grafana/02_config.yml

Config

The config.yml adds a new field sources. Sources can be any number of things defined in the gomplate datasources documentation. In this example using fruit.

template_paths:
  - "file:///etc/grafana/01_assets"
datasources:
  - name: fruit
    url: "file://etc/grafana/01_assets/fruit.json"

fruit.json

["mango","peach","orange"]

Usage

server:
  log_level: debug
metrics:
  wal_directory: /tmp/grafana-agent-normal
  global:
    scrape_interval: 60s
    remote_write:
      - url: https://prometheus-us-central1.grafana.net/api/prom/push
        basic_auth:
          username: xyz
          password: secretpassword
  configs:
    - name: default
  {{ range (datasource "fruit") }}
    - name: {{ . }}
  {{ end }}

A Datasource is reference by name and in this case it is an array and used exactly like the looping example.

Objects

Thu, 27 Apr 2023 15:07:06 +0000

02 Datasources

Datasources can also access objects.

docker run -v ${PWD}/:/etc/grafana grafana/agentctl:latest template-parse file:///etc/grafana/03_config.yml

Config

The config.yml adds a new field sources. Sources can be any number of things defined in the gomplate datasources documentation. In this example using fruit.

template_paths:
  - "file:///etc/grafana/03_assets"
datasources:
  - name: computers
    url: "file:///etc/grafana/03_assets/computers.json"

computers.json

[
  {
    "name": "webhost1",
    "ip" : "192.168.1.1",
    "enabled": true
  },
  {
    "name": "webhost2",
    "ip" : "192.168.1.2",
    "enabled": false

  },
  {
    "name": "webhost3",
    "ip" : "192.168.1.3",
    "enabled": true
  }
]

Usage

server:
  log_level: debug
metrics:
  wal_directory: /tmp/grafana-agent-normal
  global:
    scrape_interval: 60s
    remote_write:
      - url: https://prometheus-us-central1.grafana.net/api/prom/push
        basic_auth:
          username: xyz
          password: secretpassword
  configs:
    - name: default
      # Check for length so that if it is 0, we dont write any scrape configs
  {{ if $length := len (datasource "computers") }}
  {{ if gt $length 0 }}
  scrape_configs:
  {{ end }}
  {{ end }}
  {{ range (datasource "computers") }}
  # Only add if the computers are enabled
  # the . references our current object
  {{ if eq .enabled true }}
  - job_name: {{ .name }}
    static_configs:
      - targets:
          - {{ .ip }}
  {{ end }}
  {{ end }}

This is a much more complex example, in the above we are doing:

comparisons
creating and setting variables
looping over objects

The final output will only list webhost1 and webhost3 since webhost2 is not enabled.

Querying AWS

Thu, 27 Apr 2023 15:07:06 +0000

01 AWS

The AWS datasource assumes that you have appropriate credentials and environment variables set to access AWS resources. The custom fork of gomplate adds a new command to the existing AWS commands.

Unfortunately there is not a specific docker command but generic examples are below.

Looping

server:
  log_level: debug
metrics:
  wal_directory: /tmp/grafana-agent-normal
  global:
    scrape_interval: 60s
    remote_write:
      - url: https://prometheus-us-central1.grafana.net/api/prom/push
        basic_auth:
          username: xyz
          password: secretpassword
  configs:
    - name: default
      scrape_configs:
      {{ range $index , $value := aws.EC2Query "tag:service=webhost" -}}
      - job_name: {{ $value.InstanceId }}
        static_configs:
          - targets:
              - {{ $value.PrivateDnsName }}
        {{ end -}}

The aws.EC2Query command is a new command added for Grafana Agent and takes a string in the DescribeInstances format

Final