Concepts on Grafana Labs

Components

Tue, 25 Apr 2023 17:26:25 +0000

Components

Components are the building blocks of Grafana Agent Flow. Each component is responsible for handling a single task, such as retrieving secrets or collecting Prometheus metrics.

Components are composed of two parts:

Arguments: settings which configure a component.
Exports: named values which a component exposes to other components.

Each component has a name which describes what that component is responsible for. For example, the local.file component is responsible for retrieving the contents of files on disk.

Components are specified in the config file by first providing the component’s name with a user-specified label, and then by providing arguments to configure the component:

discovery.kubernetes "pods" {
  role = "pod"
}

discovery.kubernetes "nodes" {
  role = "node"
}

Components are referenced by combining the component name with its label. For example, a local.file component labeled foo would be referenced as local.file.foo.

The combination of a component’s name and its label must be unique within the configuration file. This means multiple instances of a component may be defined as long as each instance has a different label value.

Pipelines

Most arguments for a component in a config file are constant values, such setting a log_level attribute to the quoted string "debug":

log_level = "debug"

Expressions can be used to dynamically compute the value of an argument at runtime. Among other things, expressions can be used to retrieve the value of an environment variable (log_level = env("LOG_LEVEL")) or to reference an exported field of another component (log_level = local.file.log_level.content).

When a component’s argument references an exported field of another component, a dependant relationship is created: a component’s input (arguments) now depends on another component’s output (exports). The input of the component will now be re-evaluated any time the exports of the components it references get updated.

The flow of data through the set of references between components forms a pipeline.

An example pipeline may look like this:

A local.file component watches a file on disk containing an API key.
A prometheus.remote_write component is configured to receive metrics and forward them to an external database using the API key from the local.file for authentication.
A discovery.kubernetes component discovers and exports Kubernetes Pods where metrics can be collected.
A prometheus.scrape component references the exports of the previous component, and sends collected metrics to the prometheus.remote_write component.

The following config file represents the above pipeline:

// Get our API key from disk.
//
// This component has an exported field called "content", holding the content
// of the file.
//
// local.file.api_key will watch the file and update its exports any time the
// file changes.
local.file "api_key" {
  filename  = "/var/data/secrets/api-key"

  // Mark this file as sensitive to prevent its value from being shown in the
  // UI.
  is_secret = true
}

// Create a prometheus.remote_write component which other components can send
// metrics to.
//
// This component exports a "receiver" value which can be used by other
// components to send metrics.
prometheus.remote_write "prod" {
  endpoint {
    url = "https://prod:9090/api/v1/write"

    basic_auth {
      username = "admin"

      // Use our password file for authenticating with the production database.
      password = local.file.api_key.content
    }
  }
}

// Find Kubernetes pods where we can collect metrics.
//
// This component exports a "targets" value which contains the list of
// discovered pods.
discovery.kubernetes "pods" {
  role = "pod"
}

// Collect metrics from Kubernetes pods and send them to prod.
prometheus.scrape "default" {
  targets    = discovery.kubernetes.pods.targets
  forward_to = [prometheus.remote_write.prod.receiver]
}

Component controller

Wed, 12 Mar 2025 11:45:31 +0100

Component controller

The component controller is the core part of Grafana Agent Flow which manages components at runtime.

It is responsible for:

Reading and validating the config file.
Managing the lifecycle of defined components.
Evaluating the arguments used to configure components.
Reporting the health of defined components.

Component graph

As discussed in Components, a relationship between components is created when an expression is used to set the argument of one component to an exported field of another component.

The set of all components and the relationships between them define a directed acyclic graph (DAG), which informs the component controller which references are valid and in what order components must be evaluated.

For a config file to be valid, components must not reference themselves or contain a cyclic reference:

// INVALID: local.file.some_file can not reference itself:
local.file "self_reference" {
  filename = local.file.self_reference.content
}

// INVALID: cyclic reference between local.file.a and local.file.b:
local.file "a" {
  filename = local.file.b.content
}
local.file "b" {
  filename = local.file.a.content
}

Component evaluation

A component is evaluated when its expressions are computed into concrete values. The computed values are then used to configure the component’s runtime behavior. The component controller is finished loading once all components are evaluated, configured, and running.

The component controller only evaluates a given component after evaluating all of that component’s dependencies. Component that do not depend on other components can be evaluated at any time during the evaluation process.

Component reevaluation

As mentioned in Components, a component is dynamic: a component can update its exports any number of times throughout its lifetime.

When a component updates its exports, a controller reevaluation is triggered: the component controller reevaluates any component which references the changed component, any components which reference those components, and so on, until all affected components are reevaluated.

Component health

At any given time, a component can have one of the following health states:

Unknown: default state, the component isn’t running yet.
Healthy: the component is working as expected.
Unhealthy: the component is not working as expected.
Exited: the component has stopped and is no longer running.

By default, the component controller determines the health of a component. The component controller marks a component as healthy as long as that component is running and its most recent evaluation succeeded.

Some components can report their own component-specific health information. For example, the local.file component reports itself as unhealthy if the file it was watching gets deleted.

The overall health of a component is determined by combining the controller-reported health of the component with the component-specific health information.

An individual component’s health is independent of the health of any other components it references: a component can be marked as healthy even if it references an exported field of an unhealthy component.

Handling evaluation failures

When a component fails to evaluate, it is marked as unhealthy with the reason for why the evaluation failed.

When an evaluation fails, the component continue operating as normal: it continues using its previous set of evaluated arguments, and it can continue exporting new values.

This prevents failure propagation: if your local.file component which watches API keys suddenly stops working, other components continues using the last valid API key until the component returns to a healthy state.

Updating the config file

Both the /-/reload HTTP endpoint and the SIGHUP signal can be used to inform the component controller to reload the config file. When this happens, the component controller will synchronize the set of running components with the ones in the config file, removing components which are no longer defined in the config file and creating new components which were added to the config file. All components managed by the controller will be reevaluated after reloading.

Modules

Wed, 12 Mar 2025 11:45:31 +0100

Modules

Modules are a way to create Grafana Agent Flow configurations which can be loaded as a component. Modules are a great way to parameterize a configuration to create reusable pipelines.

Modules are Grafana Agent Flow configurations which have:

Arguments: settings which configure a module.
Exports: named values which a module exposes to the consumer of the module.
Components: Grafana Agent Flow Components to run when the module is running.

Modules are loaded into Grafana Agent Flow by using a Module loader.

Refer to the documentation for the argument block and export block to learn how to define arguments and exports for a module.

Module loaders

A Module loader is a Grafana Agent Flow component which retrieves a module and runs the components defined inside of it.

Module loader components are responsible for:

Retrieving the module source to run.
Creating a Component controller for the module to run in.
Passing arguments to the loaded module.
Exposing exports from the loaded module.

Module loaders typically are called module.LOADER_NAME. The list of module loader components can be found in the list of Grafana Agent Flow Components.

Some module loaders may not support running modules with arguments or exports; refer to the documentation for the module loader you are using for more information.

Module sources

Modules are designed to be flexible, and can have their configuration retrieved from anywhere, such as:

The local filesystem
An S3 bucket
An HTTP endpoint

Each module loader component will support different ways of retrieving module sources. The most generic module loader component, module.string, can load modules from the export of another Flow component:

local.file "my_module" {
  filename = "PATH_TO_MODULE"
}

module.string "my_module" {
  content = local.file.my_module.content

  arguments {
    MODULE_ARGUMENT_NAME_1 = MODULE_ARGUMENT_VALUE_1
    MODULE_ARGUMENT_NAME_2 = MODULE_ARGUMENT_VALUE_2
    // ...
  }
}

Example module

This example module manages a pipeline which filters out debug- and info-level log lines which are given to it:

// argument.write_to is a required argument which specifies where filtered
// log lines should be sent.
//
// The value of the argument can be retrieved in this file with
// argument.write_to.value.
argument "write_to" {
  optional = false
}

// loki.process.filter is our component which executes the filtering, passing
// filtered logs to argument.write_to.value.
loki.process "filter" {
  // Drop all debug- and info-level logs.
  stage.match {
    selector = "{job!=\"\"} |~ \"level=(debug|info)\""
    action   = "drop"
  }

  // Send processed logs to our argument.
  forward_to = argument.write_to.value
}

// export.filter_input exports a value to the consumer of the module.
export "filter_input" {
  // Expose the receiver of loki.process so the module consumer can send
  // logs to our loki.process component.
  value = loki.process.filter.receiver
}

It can then be used as a processing step before writing logs to Loki:

loki.source.file "self" {
  targets = LOG_TARGETS

  // Forward collected logs to the input of our filter.
  forward_to = [module.string.filter.exports.logs_input]
}

module.file "log_filter" {
  filename = "/path/to/modules/log_filter.river"

  arguments {
    // Configure the filter to forward filtered logs to loki.echo below.
    logs_output = [loki.write.default.receiver],
  }
}

loki.write "default" {
  endpoint {
    url = "LOKI_URL"
  }
}

Configuration language

Wed, 12 Mar 2025 11:45:31 +0100

Configuration language

The Grafana Agent Flow configuration language refers to the language used in configuration files which define and configure components to run.

The configuration language is called River, a Terraform/HCL-inspired language:

prometheus.scrape "default" {
  targets = [{
    "__address__" = "demo.robustperception.io:9090",
  }]
  forward_to = [prometheus.remote_write.default.receiver]
}

prometheus.remote_write "default" {
  endpoint {
    url = "http://localhost:9009/api/prom/push"
  }
}

River was designed with the following requirements in mind:

Fast: The configuration language must be fast so the component controller can evaluate changes as quickly as possible.
Simple: The configuration language must be easy to read and write to minimize the learning curve.
Debuggable: The configuration language must give detailed information when there’s a mistake in the config file.

Attributes

Attributes are used to configure individual settings. They always take the form of ATTRIBUTE_NAME = ATTRIBUTE_VALUE.

log_level = "debug"

This sets the log_level attribute to "debug".

Expressions

Expressions are used to compute the value of an attribute. The simplest expressions are constant values like "debug", 32, or [1, 2, 3, 4]. River supports more complex expressions, such as:

Referencing the exports of components: local.file.password_file.content
Mathematical operations: 1 + 2, 3 * 4, (5 * 6) + (7 + 8)
Equality checks: local.file.file_a.content == local.file.file_b.content
Calling functions from River’s standard library: env("HOME") (retrieve the value of the HOME environment variable)

Expressions may be used for any attribute inside a component definition.

Referencing component exports

The most common expression is to reference the exports of a component like local.file.password_file.content. A reference to a component’s exports is formed by merging the component’s name (e.g., local.file), label (e.g., password_file), and export name (e.g., content), delimited by period.

For components that don’t use labels, like prometheus.exporter.unix, only combine the component name with export name: prometheus.exporter.unix.targets.

Blocks

Blocks are used to configure components and groups of attributes. Each block can contain any number of attributes or nested blocks.

prometheus.remote_write "default" {
  endpoint {
    url = "http://localhost:9009/api/prom/push"
  }
}

This file has two blocks:

prometheus.remote_write "default": A labeled block which instantiates a prometheus.remote_write component. The label is the string "default".
endpoint: An unlabeled block inside the component which configures an endpoint to send metrics to. This block sets the url attribute to specify what the endpoint is.

More information

River is documented in detail in Configuration language section of the Grafana Agent Flow docs.