Concepts on Grafana Labs

Configuration language

Wed, 29 Jan 2025 11:00:55 +0000

Configuration language

Grafana Agent Flow dynamically configures and connects components with a custom configuration language called River.

River aims to reduce errors in configuration files by making configurations easier to read and write. River configurations use blocks that can be easily copied and pasted from the documentation to help you get started as quickly as possible.

A River configuration file tells Grafana Agent Flow which components to launch and how to bind them together into a pipeline.

The River syntax uses blocks, attributes, and expressions.

// Create a local.file component labeled my_file.
// This can be referenced by other components as local.file.my_file.
local.file "my_file" {
  filename = "/tmp/my-file.txt"
}

// Pattern for creating a labeled block, which the above block follows:
BLOCK_NAME "BLOCK_LABEL" {
  // Block body
  IDENTIFIER = EXPRESSION // Attribute
}

// Pattern for creating an unlabeled block:
BLOCK_NAME {
  // Block body
  IDENTIFIER = EXPRESSION // Attribute
}

River is designed with the following requirements in mind:

Fast: The configuration language must be fast so the component controller can quickly evaluate changes.
Simple: The configuration language must be easy to read and write to minimize the learning curve.
Debuggable: The configuration language must give detailed information when there’s a mistake in the configuration file.

River is similar to HCL, the language Terraform and other Hashicorp projects use. It’s a distinct language with custom syntax and features, such as first-class functions.

Blocks are a group of related settings and usually represent creating a component. Blocks have a name that consists of zero or more identifiers separated by ., an optional user label, and a body containing attributes and nested blocks.
Attributes appear within blocks and assign a value to a name.
Expressions represent a value, either literally or by referencing and combining other values. You use expressions to compute a value for an attribute.

River is declarative, so ordering components, blocks, and attributes within a block isn’t significant. The relationship between components determines the order of operations.

Attributes

You use Attributes to configure individual settings. Attributes always take the form of ATTRIBUTE_NAME = ATTRIBUTE_VALUE.

The following example shows how to set the log_level attribute to "debug".

log_level = "debug"

Expressions

You use expressions to compute the value of an attribute. The simplest expressions are constant values like "debug", 32, or [1, 2, 3, 4]. River supports complex expressions, for example:

Referencing the exports of components: local.file.password_file.content
Mathematical operations: 1 + 2, 3 * 4, (5 * 6) + (7 + 8)
Equality checks: local.file.file_a.content == local.file.file_b.content
Calling functions from River’s standard library: env("HOME") retrieves the value of the HOME environment variable.

You can use expressions for any attribute inside a component definition.

Referencing component exports

The most common expression is to reference the exports of a component, for example, local.file.password_file.content. You form a reference to a component’s exports by merging the component’s name (for example, local.file), label (for example, password_file), and export name (for example, content), delimited by a period.

Blocks

You use Blocks to configure components and groups of attributes. Each block can contain any number of attributes or nested blocks.

prometheus.remote_write "default" {
  endpoint {
    url = "http://localhost:9009/api/prom/push"
  }
}

The preceding example has two blocks:

prometheus.remote_write "default": A labeled block which instantiates a prometheus.remote_write component. The label is the string "default".
endpoint: An unlabeled block inside the component that configures an endpoint to send metrics to. This block sets the url attribute to specify the endpoint.

Tooling

You can use one or all of the following tools to help you write configuration files in River.

Experimental editor support for
- vim
- VSCode
- river-mode for Emacs
Code formatting using the agent fmt command

You can also start developing your own tooling using the Grafana Agent repository as a go package or use the tree-sitter grammar with other programming languages.

Components

Wed, 29 Jan 2025 11:00:55 +0000

Components

Components are the building blocks of Grafana Agent Flow. Each component handles a single task, such as retrieving secrets or collecting Prometheus metrics.

Components are composed of the following:

Arguments: Settings that configure a component.
Exports: Named values that a component exposes to other components.

Each component has a name that describes what that component is responsible for. For example, the local.file component is responsible for retrieving the contents of files on disk.

You specify components in the configuration file by first providing the component’s name with a user-specified label, and then by giving arguments to configure the component.

discovery.kubernetes "pods" {
  role = "pod"
}

discovery.kubernetes "nodes" {
  role = "node"
}

You reference components by combining the component name with its label. For example, you can reference a local.file component labeled foo as local.file.foo.

The combination of a component’s name and its label must be unique within the configuration file. Combining component names with a label means you can define multiple instances of a component as long as each instance has a different label value.

Pipelines

Most arguments for a component in a configuration file are constant values, such as setting a log_level attribute to the quoted string "debug".

log_level = "debug"

You use expressions to dynamically compute the value of an argument at runtime. You can use expressions to retrieve the value of an environment variable (log_level = env("LOG_LEVEL")) or to reference an exported field of another component (log_level = local.file.log_level.content).

You create a dependent relationship when a component’s argument references an exported field of another component. A component’s arguments now depend on another component’s exports. The input of the component is re-evaluated whenever the exports of the components it references are updated.

The flow of data through the set of references between components forms a pipeline.

An example pipeline may look like this:

A local.file component watches a file that contains an API key.
A prometheus.remote_write component is configured to receive metrics and forward them to an external database using the API key from the local.file for authentication.
A discovery.kubernetes component discovers and exports Kubernetes Pods where metrics can be collected.
A prometheus.scrape component references the exports of the previous component, and sends collected metrics to the prometheus.remote_write component.

The following configuration file represents the pipeline.

// Get our API key from disk.
//
// This component has an exported field called "content", holding the content
// of the file.
//
// local.file.api_key will watch the file and update its exports any time the
// file changes.
local.file "api_key" {
  filename  = "/var/data/secrets/api-key"

  // Mark this file as sensitive to prevent its value from being shown in the
  // UI.
  is_secret = true
}

// Create a prometheus.remote_write component, which other components can send
// metrics to.
//
// This component exports a "receiver" value, which can be used by other
// components to send metrics.
prometheus.remote_write "prod" {
  endpoint {
    url = "https://prod:9090/api/v1/write"

    basic_auth {
      username = "admin"

      // Use the password file to authenticate with the production database.
      password = local.file.api_key.content
    }
  }
}

// Find Kubernetes pods where we can collect metrics.
//
// This component exports a "targets" value, which contains the list of
// discovered pods.
discovery.kubernetes "pods" {
  role = "pod"
}

// Collect metrics from Kubernetes pods and send them to prod.
prometheus.scrape "default" {
  targets    = discovery.kubernetes.pods.targets
  forward_to = [prometheus.remote_write.prod.receiver]
}

Component controller

Wed, 29 Jan 2025 11:00:55 +0000

Component controller

The component controller is the core part of Grafana Agent Flow which manages components at runtime.

The component controller is responsible for:

Reading and validating the configuration file.
Managing the lifecycle of defined components.
Evaluating the arguments used to configure components.
Reporting the health of defined components.

Component graph

A relationship between components is created when an expression is used to set the argument of one component to an exported field of another component.

The set of all components and the relationships between them define a Directed Acyclic Graph (DAG), which informs the component controller which references are valid and in what order components must be evaluated.

For a configuration file to be valid, components must not reference themselves or contain a cyclic reference.

// INVALID: local.file.some_file can not reference itself:
local.file "self_reference" {
  filename = local.file.self_reference.content
}

// INVALID: cyclic reference between local.file.a and local.file.b:
local.file "a" {
  filename = local.file.b.content
}
local.file "b" {
  filename = local.file.a.content
}

Component evaluation

A component is evaluated when its expressions are computed into concrete values. The computed values configure the component’s runtime behavior. The component controller is finished loading once all components are evaluated, configured, and running.

The component controller only evaluates a given component after evaluating all of that component’s dependencies. Components that don’t depend on other components can be evaluated anytime during the evaluation process.

Component reevaluation

A component is dynamic. A component can update its exports any number of times throughout its lifetime.

A controller reevaluation is triggered when a component updates its exports. The component controller reevaluates any component that references the changed component, any components that reference those components, and so on, until all affected components are reevaluated.

Component health

At any given time, a component can have one of the following health states:

Unknown: The default state. The component isn’t running yet.
Healthy: The component is working as expected.
Unhealthy: The component isn’t working as expected.
Exited: The component has stopped and is no longer running.

By default, the component controller determines the health of a component. The component controller marks a component as healthy as long as that component is running and its most recent evaluation succeeded.

Some components can report their own component-specific health information. For example, the local.file component reports itself as unhealthy if the file it was watching gets deleted.

The overall health of a component is determined by combining the controller-reported health of the component with the component-specific health information.

An individual component’s health is independent of the health of any other components it references. A component can be marked as healthy even if it references an exported field of an unhealthy component.

Handling evaluation failures

When a component fails to evaluate, it’s marked as unhealthy with the reason for why the evaluation failed.

When an evaluation fails, the component continues operating as normal. The component continues using its previous set of evaluated arguments and can continue exporting new values.

This behavior prevents failure propagation. If your local.file component, which watches API keys, suddenly stops working, other components continue using the last valid API key until the component returns to a healthy state.

In-memory traffic

Components that expose HTTP endpoints, such as prometheus.exporter.unix, can expose an internal address that completely bypasses the network and communicate in-memory. Components within the same process can communicate with one another without needing to be aware of any network-level protections such as authentication or mutual TLS.

The internal address defaults to agent.internal:12345. If this address collides with a real target on your network, change it to something unique using the --server.http.memory-addr flag in the run command.

Components must opt-in to using in-memory traffic. Refer to the individual documentation for components to learn if in-memory traffic is supported.

Updating the configuration file

The /-/reload HTTP endpoint and the SIGHUP signal can inform the component controller to reload the configuration file. When this happens, the component controller synchronizes the set of running components with the ones in the configuration file, removing components no longer defined in the configuration file and creating new components added to the configuration file. All components managed by the controller are reevaluated after reloading.

Custom components

Wed, 18 Jun 2025 15:04:54 +0000

Custom components

Custom components are a way to create new components from a pipeline of built-in and other custom components.

A custom component is composed of:

Arguments: Settings that configure the custom component.
Exports: Values that a custom component exposes to its consumers.
Components: Built-in and custom components that are run as part of the custom component.

Creating custom components

You can create a new custom component using the declare configuration block. The label of the block determines the name of the custom component.

The following custom configuration blocks can be used inside a declare block:

argument: Create a new named argument, whose current value can be referenced using the expression argument.NAME.value. Argument values are determined by the user of a custom component.
export: Expose a new named value to custom component users.

Custom components are useful for reusing a common pipeline multiple times. To learn how to share custom components across multiple files, refer to Modules.

Example

This example creates a new custom component called add, which exports the sum of two arguments:

declare "add" {
    argument "a" { }
    argument "b" { }

    export "sum" {
        value = argument.a.value + argument.b.value
    }
}

add "example" {
    a = 15
    b = 17
}

// add.example.sum == 32

Modules

Wed, 18 Jun 2025 15:04:54 +0000

Modules

A Module is a unit of Grafana Agent Flow configuration, which combines all the other concepts, containing a mix of configuration blocks, instantiated components, and custom component definitions. The module passed as an argument to the run command is called the main configuration.

Modules can be imported to enable the reuse of custom components defined by that module.

Importing modules

A module can be imported, allowing the custom components defined by that module to be used by other modules, called the importing module. Modules can be imported from multiple locations using one of the import configuration blocks:

import.file: Imports a module from a file or a directory on disk.
import.git: Imports a module from a file located in a Git repository.
import.http: Imports a module from the response of an HTTP request.
import.string: Imports a module from a string.

Warning
You can’t import a module that contains top-level blocks other than declare or import.

Modules are imported into a namespace where the top-level custom components of the imported module are exposed to the importing module. The label of the import block specifies the namespace of an import. For example, if a configuration contains a block called import.file "my_module", then custom components defined by that module are exposed as my_module.CUSTOM_COMPONENT_NAME. Imported namespaces must be unique across a given importing module.

If an import namespace matches the name of a built-in component namespace, such as prometheus, the built-in namespace is hidden from the importing module, and only components defined in the imported module may be used.

Example

This example module defines a component to filter out debug-level and info-level log lines:

declare "log_filter" {
  // argument.write_to is a required argument that specifies where filtered
  // log lines are sent.
  //
  // The value of the argument is retrieved in this file with
  // argument.write_to.value.
  argument "write_to" {
    optional = false
  }

  // loki.process.filter is our component which executes the filtering,
  // passing filtered logs to argument.write_to.value.
  loki.process "filter" {
    // Drop all debug- and info-level logs.
    stage.match {
      selector = `{job!=""} |~ "level=(debug|info)"`
      action   = "drop"
    }

    // Send processed logs to our argument.
    forward_to = argument.write_to.value
  }

  // export.filter_input exports a value to the module consumer.
  export "filter_input" {
    // Expose the receiver of loki.process so the module importer can send
    // logs to our loki.process component.
    value = loki.process.filter.receiver
  }
}

You can save this module to a file called helpers.river and import it:

// Import our helpers.river module, exposing its custom components as
// helpers.COMPONENT_NAME.
import.file "helpers" {
  filename = "helpers.river"
}

loki.source.file "self" {
  targets = LOG_TARGETS

  // Forward collected logs to the input of our filter.
  forward_to = [helpers.log_filter.default.filter_input]
}

helpers.log_filter "default" {
  // Configure the filter to forward filtered logs to loki.write below.
  write_to = [loki.write.default.receiver]
}

loki.write "default" {
  endpoint {
    url = LOKI_URL
  }
}

Classic modules (deprecated)

Caution
Modules were redesigned in v0.40 to simplify concepts. This section outlines the design of the original modules prior to v0.40. Classic modules are scheduled to be removed in the release after v0.40.

You use Modules to create Grafana Agent Flow configurations that you can load as a component. Modules are a great way to parameterize a configuration to create reusable pipelines.

Modules are Grafana Agent Flow configurations which have:

Arguments: Settings that configure a module.
Exports: Named values that a module exposes to the consumer of the module.
Components: Grafana Agent Flow components to run when the module is running.

You use a Module loader to load Modules into Grafana Agent Flow.

Refer to argument block and export block to learn how to define arguments and exports for a module.

Module loaders

A Module loader is a Grafana Agent Flow component that retrieves a module and runs the defined components.

Module loader components are responsible for the following functions:

Retrieving the module source.
Creating a Component controller for the module.
Passing arguments to the loaded module.
Exposing exports from the loaded module.

Module loaders are typically called module.LOADER_NAME.

Note
Some module loaders may not support running modules with arguments or exports.

Refer to Components for more information about the module loader components.

Module sources

Modules are flexible, and you can retrieve their configuration anywhere, such as:

The local filesystem.
An S3 bucket.
An HTTP endpoint.

Each module loader component supports different ways of retrieving module.sources. The most generic module loader component, module.string, can load modules from the export of another Grafana Agent Flow component.

local.file "my_module" {
  filename = "PATH_TO_MODULE"
}

module.string "my_module" {
  content = local.file.my_module.content

  arguments {
    MODULE_ARGUMENT_NAME_1 = MODULE_ARGUMENT_VALUE_1
    MODULE_ARGUMENT_NAME_2 = MODULE_ARGUMENT_VALUE_2
    // ...
  }
}

Example module

This example module manages a pipeline that filters out debug-level and info-level log lines.

// argument.write_to is a required argument that specifies where filtered
// log lines are sent.
//
// The value of the argument is retrieved in this file with
// argument.write_to.value.
argument "write_to" {
  optional = false
}

// loki.process.filter is our component which executes the filtering, passing
// filtered logs to argument.write_to.value.
loki.process "filter" {
  // Drop all debug- and info-level logs.
  stage.match {
    selector = "{job!=\"\"} |~ \"level=(debug|info)\""
    action   = "drop"
  }

  // Send processed logs to our argument.
  forward_to = argument.write_to.value
}

// export.filter_input exports a value to the module consumer.
export "filter_input" {
  // Expose the receiver of loki.process so the module consumer can send
  // logs to our loki.process component.
  value = loki.process.filter.receiver
}

You can save the module to a file and then use it as a processing step before writing logs to Loki.

loki.source.file "self" {
  targets = LOG_TARGETS

  // Forward collected logs to the input of our filter.
  forward_to = [module.file.log_filter.exports.filter_input]
}

module.file "log_filter" {
  filename = "/path/to/modules/log_filter.river"

  arguments {
    // Configure the filter to forward filtered logs to loki.write below.
    write_to = [loki.write.default.receiver],
  }
}

loki.write "default" {
  endpoint {
    url = "LOKI_URL"
  }
}

Clustering

Wed, 29 Jan 2025 11:00:55 +0000

Clustering

Clustering enables a fleet of Grafana Agents to work together for workload distribution and high availability. It helps create horizontally scalable deployments with minimal resource and operational overhead.

To achieve this, Grafana Agent Flow makes use of an eventually consistent model that assumes all participating Grafana Agents are interchangeable and converge on using the same configuration file.

The behavior of a standalone, non-clustered Grafana Agent is the same as if it were a single-node cluster.

You configure clustering by passing cluster command-line flags to the run command.

Use cases

Target auto-distribution

Target auto-distribution is the most basic use case of clustering. It allows scraping components running on all peers to distribute the scrape load between themselves. Target auto-distribution requires that all Grafana Agent in the same cluster can reach the same service discovery APIs and scrape the same targets.

You must explicitly enable target auto-distribution on components by defining a clustering block.

prometheus.scrape "default" {
    clustering {
        enabled = true
    }

    ...
}

A cluster state change is detected when a new node joins or an existing node leaves. All participating components locally recalculate target ownership and re-balance the number of targets they’re scraping without explicitly communicating ownership over the network.

Target auto-distribution allows you to dynamically scale the number of Grafana Agents to distribute workload during peaks. It also provides resiliency because targets are automatically picked up by one of the node peers if a node leaves.

Grafana Agent Flow uses a local consistent hashing algorithm to distribute targets, meaning that, on average, only ~1/N of the targets are redistributed.

Refer to component reference documentation to discover whether it supports clustering, such as:

Cluster monitoring and troubleshooting

You can use the Grafana Agent Flow UI clustering page to monitor your cluster status. Refer to Debugging clustering issues for additional troubleshooting information.