Flow by example on Grafana Labs

Get started

Tue, 10 Mar 2026 03:19:46 +0000

Get started

This set of tutorials contains a collection of examples that build on each other to demonstrate how to configure and use Grafana Agent Flow. It assumes you have a basic understanding of what Grafana Agent is and telemetry collection in general. It also assumes a base level of familiarity with Prometheus and PromQL, Loki and LogQL, and basic Grafana navigation. It assumes no knowledge of Grafana Agent Flow or River concepts.

What is Flow?

Flow is a new way to configure Grafana Agent Flow. It is a declarative configuration language that allows you to define a pipeline of telemetry collection, processing, and output. It is built on top of the River configuration language, which is designed to be fast, simple, and debuggable.

What do I need to get started?

You will need a Linux or Unix environment with Docker installed. The examples are designed to be run on a single host so that you can run them on your laptop or in a VM. You are encouraged to follow along with the examples using a config.river file and experiment with the examples yourself.

To run the examples, you should have a Grafana Agent binary available. You can follow the instructions on how to Install Grafana Agent as a Standalone Binary to get a binary.

How should I follow along?

You can use this docker-compose file to set up a local Grafana instance alongside Loki and Prometheus pre-configured as datasources. The examples are designed to be run locally, so you can follow along and experiment with them yourself.

version: '3'
services:
  loki:
    image: grafana/loki:2.9.0
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml
  prometheus:
    image: prom/prometheus:v2.47.0
    command:
      - --web.enable-remote-write-receiver
      - --config.file=/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"
  grafana:
    environment:
      - GF_PATHS_PROVISIONING=/etc/grafana/provisioning
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
    entrypoint:
      - sh
      - -euc
      - |
        mkdir -p /etc/grafana/provisioning/datasources
        cat <<EOF > /etc/grafana/provisioning/datasources/ds.yaml
        apiVersion: 1
        datasources:
        - name: Loki
          type: loki
          access: proxy
          orgId: 1
          url: http://loki:3100
          basicAuth: false
          isDefault: false
          version: 1
          editable: false
        - name: Prometheus
          type: prometheus
          orgId: 1
          url: http://prometheus:9090
          basicAuth: false
          isDefault: true
          version: 1
          editable: false
        EOF
        /run.sh
    image: grafana/grafana:latest
    ports:
      - "3000:3000"

After running docker-compose up, open http://localhost:3000 in your browser to view the Grafana UI.

The tutorials are designed to be followed in order and generally build on each other. Each example explains what it does and how it works. They are designed to be run locally, so you can follow along and experiment with them yourself.

The Recommended Reading sections in each tutorial provide a list of documentation topics. To help you understand the concepts used in the example, read the recommended topics in the order given.

First components and introducing the standard library

Tue, 10 Mar 2026 03:19:46 +0000

First components and the standard library

This tutorial covers the basics of the River language and the standard library. It introduces a basic pipeline that collects metrics from the host and sends them to Prometheus.

River basics

Recommended reading

River is an HCL-inspired configuration language used to configure Grafana Agent Flow. A River file is comprised of three things:

Attributes

key = value pairs used to configure individual settings.
Alloy
```
url = "http://localhost:9090"
```
Expressions

Expressions are used to compute values. They can be constant values (for example, "localhost:9090"), or they can be more complex (for example, referencing a component’s export: prometheus.exporter.unix.targets. They can also be a mathematical expression: (1 + 2) * 3, or a standard library function call: env("HOME")). We will use more expressions as we go along the examples. If you are curious, you can find a list of available standard library functions in the Standard library documentation.
Blocks

Blocks are used to configure components with groups of attributes or nested blocks. The following example block can be used to configure the logging output of Grafana Agent Flow:
Alloy
```
logging {
    level  = "debug"
    format = "json"
}
```
Note
The default log level is info and the default log format is logfmt.

Try pasting this into config.river and running /path/to/agent run config.river to see what happens.

Congratulations, you’ve just written your first River file! You’ve also just written your first Grafana Agent Flow configuration file. This configuration won’t do anything, so let’s add some components to it.

Note
Comments in River are prefixed with // and are single-line only. For example: // This is a comment.

Components

Recommended reading

Components are the building blocks of a Grafana Agent Flow configuration. They are configured and linked to create pipelines that collect, process, and output your telemetry data. Components are configured with Arguments and have Exports that may be referenced by other components.

Let’s look at a simple example pipeline:

local.file "example" {
    path = env("HOME") + "file.txt"
}

prometheus.remote_write "local_prom" {
    endpoint {
        url = "http://localhost:9090/api/v1/write"

        basic_auth {
            username = "admin"
            password = local.file.example.content
        }
    }
}

Note
A list of all available components can be found in the Component reference. Each component has a link to its documentation, which contains a description of what the component does, its arguments, its exports, and examples.

This pipeline has two components: local.file and prometheus.remote_write. The local.file component is configured with a single argument, path, which is set by calling the env standard library function to retrieve the value of the HOME environment variable and concatenating it with the string "file.txt". The local.file component has a single export, content, which contains the contents of the file.

The prometheus.remote_write component is configured with an endpoint block, containing the url attribute and a basic_auth block. The url attribute is set to the URL of the Prometheus remote write endpoint. The basic_auth block contains the username and password attributes, which are set to the string "admin" and the content export of the local.file component, respectively. The content export is referenced by using the syntax local.file.example.content, where local.file.example is the fully qualified name of the component (the component’s type + its label) and content is the name of the export.

Note
The local.file component’s label is set to "example", so the fully qualified name of the component is local.file.example. The prometheus.remote_write component’s label is set to "local_prom", so the fully qualified name of the component is prometheus.remote_write.local_prom.

This example pipeline still doesn’t do anything, so let’s add some more components to it.

Shipping your first metrics

Recommended reading

Make a simple pipeline with a prometheus.exporter.unix component, a prometheus.scrape component to scrape it, and a prometheus.remote_write component to send the scraped metrics to Prometheus.

prometheus.exporter.unix "localhost" {
    // This component exposes a lot of metrics by default, so we will keep all of the default arguments.
}

prometheus.scrape "default" {
    // Setting the scrape interval lower to make it faster to be able to see the metrics
    scrape_interval = "10s"

    targets    = prometheus.exporter.unix.localhost.targets
    forward_to = [
        prometheus.remote_write.local_prom.receiver,
    ]
}

prometheus.remote_write "local_prom" {
    endpoint {
        url = "http://localhost:9090/api/v1/write"
    }
}

Run Grafana Agent Flow with:

/path/to/agent run config.river

Navigate to http://localhost:3000/explore in your browser. After ~15-20 seconds, you should be able to see the metrics from the prometheus.exporter.unix component! Try querying for node_memory_Active_bytes to see the active memory of your host.

Visualizing the relationship between components

The following diagram is an example pipeline:

The preceding configuration defines three components:

prometheus.scrape - A component that scrapes metrics from components that export targets.
prometheus.exporter.unix - A component that exports metrics from the host, built around node_exporter.
prometheus.remote_write - A component that sends metrics to a Prometheus remote-write compatible endpoint.

The prometheus.scrape component references the prometheus.exporter.unix component’s targets export, which is a list of scrape targets. The prometheus.scrape component then forwards the scraped metrics to the prometheus.remote_write component.

One rule is that components can’t form a cycle. This means that a component can’t reference itself directly or indirectly. This is to prevent infinite loops from forming in the pipeline.

Exercise for the reader

Recommended Reading

Optional: prometheus.exporter.redis

Let’s start a container running Redis and configure Grafana Agent Flow to scrape metrics from it.

docker container run -d --name flow-redis -p 6379:6379 --rm redis

Try modifying the pipeline to scrape metrics from the Redis exporter. You can refer to the prometheus.exporter.redis component documentation for more information on how to configure it.

To give a visual hint, you want to create a pipeline that looks like this:

Note
You may find the concat standard library function useful.

You can run Grafana Agent Flow with the new configuration file by running:

/path/to/agent run config.river

Navigate to http://localhost:3000/explore in your browser. After the first scrape, you should be able to query for redis metrics as well as node metrics.

To shut down the Redis container, run:

docker container stop flow-redis

If you get stuck, you can always view a solution here:

// Configure your first components, learn about the standard library, and learn how to run Grafana Agent

// prometheus.exporter.redis collects information about Redis and exposes
// targets for other components to use
prometheus.exporter.redis "local_redis" {
    redis_addr = "localhost:6379"
}

prometheus.exporter.unix "localhost" { }

// prometheus.scrape scrapes the targets that it is configured with and forwards
// the metrics to other components (typically prometheus.relabel or prometheus.remote_write)
prometheus.scrape "default" {
    // This is scraping too often for typical use-cases, but is easier for testing and demo-ing!
    scrape_interval = "10s"

    // Here, prometheus.exporter.redis.local_redis.targets refers to the 'targets' export
    // of the prometheus.exporter.redis component with the label "local_redis".
    //
    // If you have more than one set of targets that you would like to scrape, you can use
    // the 'concat' function from the standard library to combine them.
    targets    = concat(prometheus.exporter.redis.local_redis.targets, prometheus.exporter.unix.localhost.targets)
    forward_to = [prometheus.remote_write.local_prom.receiver]
}

// prometheus.remote_write exports a 'receiver', which other components can forward
// metrics to and it will remote_write them to the configured endpoint(s)
prometheus.remote_write "local_prom" {
    endpoint {
        url = "http://localhost:9090/api/v1/write"
    }
}

Finishing up and next steps

You might have noticed that running Grafana Agent Flow with the configurations created a directory called data-agent in the directory you ran Grafana Agent Flow from. This directory is where components can store data, such as the prometheus.exporter.unix component storing its WAL (Write Ahead Log). If you look in the directory, do you notice anything interesting? The directory for each component is the fully qualified name.

If you’d like to store the data elsewhere, you can specify a different directory by supplying the --storage.path flag to Grafana Agent’s run command, for example, /path/to/agent run config.river --storage.path /etc/grafana-agent. Generally, you can use a persistent directory for this, as some components may use the data stored in this directory to perform their function.

In the next tutorial, you will look at how to configure Grafana Agent Flow to collect logs from a file and send them to Loki. You will also look at using different components to process metrics and logs before sending them.

Logs and relabeling basics

Tue, 10 Mar 2026 03:19:46 +0000

Logs and relabeling basics

This tutorial assumes you have completed the First components and introducing the standard library tutorial, or are at least familiar with the concepts of components, attributes, and expressions and how to use them. You will cover some basic metric relabeling, followed by how to send logs to Loki.

Relabel metrics

Recommended reading

Optional: prometheus.relabel

Before moving on to logs, let’s look at how we can use the prometheus.relabel component to relabel metrics. The prometheus.relabel component allows you to perform Prometheus relabeling on metrics and is similar to the relabel_configs section of a Prometheus scrape config.

Let’s add a prometheus.relabel component to a basic pipeline and see how to add labels.

prometheus.exporter.unix "localhost" { }

prometheus.scrape "default" {
    scrape_interval = "10s"

    targets    = prometheus.exporter.unix.localhost.targets
    forward_to = [
        prometheus.relabel.example.receiver,
    ]
}

prometheus.relabel "example" {
    forward_to = [
        prometheus.remote_write.local_prom.receiver,
    ]

    rule {
        action       = "replace"
        target_label = "os"
        replacement  = constants.os
    }
}

prometheus.remote_write "local_prom" {
    endpoint {
        url = "http://localhost:9090/api/v1/write"
    }
}

We have now created the following pipeline:

This pipeline has a prometheus.relabel component that has a single rule. This rule has the replace action, which will replace the value of the os label with a special value: constants.os. This value is a special constant that is replaced with the OS of the host Grafana Agent is running on. You can see the other available constants in the constants documentation. This example has one rule block, but you can have as many as you want. Each rule block is applied in order.

If you run Grafana Agent and navigate to localhost:3000/explore, you can see the os label on the metrics. Try querying for node_context_switches_total and look at the labels.

Relabeling uses the same rules as Prometheus. You can always refer to the prometheus.relabel documentation for a full list of available options.

Note
You can forward multiple components to one prometheus.relabel component. This allows you to apply the same relabeling rules to multiple pipelines.

Warning
There is an issue commonly faced when relabeling and using labels that start with __ (double underscore). These labels are considered internal and are dropped before relabeling rules from a prometheus.relabel component are applied. If you would like to keep or act on these kinds of labels, use a discovery.relabel component.

Send logs to Loki

Recommended reading

Optional: local.file_match
Optional: loki.source.file
Optional: loki.write

Now that you’re comfortable creating components and chaining them together, let’s collect some logs and send them to Loki. We will use the local.file_match component to perform file discovery, the loki.source.file to collect the logs, and the loki.write component to send the logs to Loki.

Before doing this, we need to ensure we have a log file to scrape. We will use the echo command to create a file with some log content.

mkdir -p /tmp/flow-logs
echo "This is a log line" > /tmp/flow-logs/log.log

Now that we have a log file, let’s create a pipeline to scrape it.

local.file_match "tmplogs" {
    path_targets = [{"__path__" = "/tmp/flow-logs/*.log"}]
}

loki.source.file "local_files" {
    targets    = local.file_match.tmplogs.targets
    forward_to = [loki.write.local_loki.receiver]
}

loki.write "local_loki" {
    endpoint {
        url = "http://localhost:3100/loki/api/v1/push"
    }
}

The rough flow of this pipeline is:

If you navigate to localhost:3000/explore and switch the Datasource to Loki, you can query for {filename="/tmp/flow-logs/log.log"} and see the log line we created earlier. Try running the following command to add more logs to the file.

echo "This is another log line!" >> /tmp/flow-logs/log.log

If you re-execute the query, you can see the new log lines.

If you are curious how Grafana Agent keeps track of where it is in a log file, you can look at data-agent/loki.source.file.local_files/positions.yml. If you delete this file, Grafana Agent starts reading from the beginning of the file again, which is why keeping the Grafana Agent’s data directory in a persistent location is desirable.

Exercise

Recommended reading

Add a Label to Logs

This exercise will have two parts, building on the previous example. Let’s start by adding an os label (just like the Prometheus example) to all of the logs we collect.

Modify the following snippet to add the label os with the value of the os constant.

local.file_match "tmplogs" {
    path_targets = [{"__path__" = "/tmp/flow-logs/*.log"}]
}

loki.source.file "local_files" {
    targets    = local.file_match.tmplogs.targets
    forward_to = [loki.write.local_loki.receiver]
}

loki.write "local_loki" {
    endpoint {
        url = "http://localhost:3100/loki/api/v1/push"
    }
}

Note
You can use the loki.relabel component to relabel and add labels, just like you can with the prometheus.relabel component.

Once you have your completed configuration, run Grafana Agent and execute the following:

echo 'level=info msg="INFO: This is an info level log!"' >> /tmp/flow-logs/log.log
echo 'level=warn msg="WARN: This is a warn level log!"' >> /tmp/flow-logs/log.log
echo 'level=debug msg="DEBUG: This is a debug level log!"' >> /tmp/flow-logs/log.log

Navigate to localhost:3000/explore and switch the Datasource to Loki. Try querying for {filename="/tmp/flow-logs/log.log"} and see if you can find the new label!

Now that we have added new labels, we can also filter on them. Try querying for {os!=""}. You should only see the lines you added in the previous step.

// Let's learn about relabeling and send logs to Loki!

local.file_match "tmplogs" {
    path_targets = [{"__path__" = "/tmp/flow-logs/*.log"}]
}

loki.source.file "local_files" {
    targets    = local.file_match.tmplogs.targets
    forward_to = [loki.relabel.add_static_label.receiver]
}

loki.relabel "add_static_label" {
    forward_to = [loki.write.local_loki.receiver]

    rule {
        target_label = "os"
        replacement  = constants.os
    }
}

loki.write "local_loki" {
    endpoint {
        url = "http://localhost:3100/loki/api/v1/push"
    }
}

Extract and add a Label from Logs

Note
This exercise is more challenging than the previous one. If you are having trouble, skip it and move to the next section, which will cover some of the concepts used here. You can always come back to this exercise later.

This exercise will build on the previous one, though it’s more involved.

Let’s say we want to extract the level from the logs and add it as a label. As a starting point, look at loki.process. This component allows you to perform processing on logs, including extracting values from log contents.

Try modifying your configuration from the previous section to extract the level from the logs and add it as a label. If needed, you can find a solution to the previous exercise at the end of the previous section.

Note
The stage.logfmt and stage.labels blocks for loki.process may be helpful.

Once you have your completed config, run Grafana Agent and execute the following:

echo 'level=info msg="INFO: This is an info level log!"' >> /tmp/flow-logs/log.log
echo 'level=warn msg="WARN: This is a warn level log!"' >> /tmp/flow-logs/log.log
echo 'level=debug msg="DEBUG: This is a debug level log!"' >> /tmp/flow-logs/log.log

Navigate to localhost:3000/explore and switch the Datasource to Loki. Try querying for {level!=""} to see the new labels in action.

// Let's learn about relabeling and send logs to Loki!

local.file_match "tmplogs" {
    path_targets = [{"__path__" = "/tmp/flow-logs/*.log"}]
}

loki.source.file "local_files" {
    targets    = local.file_match.tmplogs.targets
    forward_to = [loki.process.add_new_label.receiver]
}

loki.process "add_new_label" {
    // Extract the value of "level" from the log line and add it to the extracted map as "extracted_level"
    // You could also use "level" = "", which would extract the value of "level" and add it to the extracted map as "level"
    // but to make it explicit for this example, we will use a different name.
    //
    // The extracted map will be covered in more detail in the next section.
    stage.logfmt {
        mapping = {
            "extracted_level" = "level",
        }
    }

    // Add the value of "extracted_level" from the extracted map as a "level" label
    stage.labels {
        values = {
            "level" = "extracted_level",
        }
    }

    forward_to = [loki.relabel.add_static_label.receiver]
}

loki.relabel "add_static_label" {
    forward_to = [loki.write.local_loki.receiver]

    rule {
        target_label = "os"
        replacement  = constants.os
    }
}

loki.write "local_loki" {
    endpoint {
        url = "http://localhost:3100/loki/api/v1/push"
    }
}

Finishing up and next steps

You have learned the concepts of components, attributes, and expressions. You have also seen how to use some standard library components to collect metrics and logs. In the next tutorial, you will learn more about how to use the loki.process component to extract values from logs and use them.

Processing Logs

Tue, 10 Mar 2026 03:19:46 +0000

Processing Logs

This tutorial assumes you are familiar with setting up and connecting components. It covers using loki.source.api to receive logs over HTTP, processing and filtering them, and sending them to Loki.

Receive logs over HTTP and Process

Recommended reading

Optional: loki.source.api

The loki.source.api component can receive logs over HTTP. It can be useful for receiving logs from other Grafana Agents or collectors, or directly from applications that can send logs over HTTP, and then processing them centrally.

Your pipeline is going to look like this:

Let’s start by setting up the loki.source.api component:

loki.source.api "listener" {
    http {
        listen_address = "127.0.0.1"
        listen_port    = 9999
    }

    labels = { "source": "api" }

    forward_to = [loki.process.process_logs.receiver]
}

This is a simple configuration. You are configuring the loki.source.api component to listen on 127.0.0.1:9999 and attach a source="api" label to the received log entries, which are then forwarded to the loki.process.process_logs component’s exported receiver. Next, you can configure the loki.process and loki.write components.

Process and Write Logs

Recommended reading

// Let's send and process more logs!

loki.source.api "listener" {
    http {
        listen_address = "127.0.0.1"
        listen_port    = 9999
    }

    labels = { "source" = "api" }

    forward_to = [loki.process.process_logs.receiver]
}

loki.process "process_logs" {

    // Stage 1
    stage.json {
        expressions = {
            log = "",
            ts  = "timestamp",
        }
    }

    // Stage 2
    stage.timestamp {
        source = "ts"
        format = "RFC3339"
    }

    // Stage 3
    stage.json {
        source = "log"

        expressions = {
            is_secret = "",
            level     = "",
            log_line  = "message",
        }
    }

    // Stage 4
    stage.drop {
        source = "is_secret"
        value  = "true"
    }

    // Stage 5
    stage.labels {
        values = {
            level = "",
        }
    }

    // Stage 6
    stage.output {
        source = "log_line"
    }

    // This stage adds static values to the labels on the log line
    stage.static_labels {
        values = {
            source = "demo-api",
        }
    }

    forward_to = [loki.write.local_loki.receiver]
}

loki.write "local_loki" {
    endpoint {
        url = "http://localhost:3100/loki/api/v1/push"
    }
}

You can skip to the next section if you successfully completed the previous section’s exercises. If not, or if you were unsure how things worked, let’s break down what is happening in the loki.process component.

Many of the stage.* blocks in loki.process act on reading or writing a shared map of values extracted from the logs. You can think of this extracted map as a hashmap or table that each stage has access to, and it is referred to as the “extracted map” from here on. In subsequent stages, you can use the extracted map to filter logs, add or remove labels, or even modify the log line.

Note
stage.* blocks are executed in the order they appear in the component, top down.

Let’s use an example log line to illustrate this, then go stage by stage, showing the contents of the extracted map. Here is our example log line:

{
    "log": {
        "is_secret": "true",
        "level": "info",
        "message": "This is a secret message!",
    },
    "timestamp": "2023-11-16T06:01:50Z",
}

Stage 1

stage.json {
    expressions = {
        log = "",
        ts  = "timestamp",
    }
}

This stage parses the log line as JSON, extracts two values from it, log and timestamp, and puts them into the extracted map with keys log and ts, respectively.

Note
Supplying an empty string is shorthand for using the same key as in the input log line (so log = "" is the same as log = "log"). The keys of the expressions object end up as the keys in the extracted map, and the values are used as keys to look up in the parsed log line.

If this were Python, it would be roughly equivalent to:

extracted_map = {}
log_line      = {"log": {"is_secret": "true", "level": "info", "message": "This is a secret message!"}, "timestamp": "2023-11-16T06:01:50Z"}

extracted_map["log"] = log_line["log"]
extracted_map["ts"]  = log_line["timestamp"]

Extracted map before performing this stage:

{}

Extracted map after performing this stage:

{
    "log": {
        "is_secret": "true",
        "level": "info",
        "message": "This is a secret message!",
    },
    "ts": "2023-11-16T06:01:50Z",
}

Stage 2

stage.timestamp {
    source = "ts"
    format = "RFC3339"
}

This stage acts on the ts value in the map you extracted in the previous stage. The value of ts is parsed in the format of RFC3339 and added as the timestamp to be ingested by Loki. This is useful if you want to use the timestamp present in the log itself, rather than the time the log is ingested. This stage does not modify the extracted map.

Stage 3

stage.json {
    source = "log"

    expressions = {
        is_secret = "",
        level     = "",
        log_line  = "message",
    }
}

This stage acts on the log value in the extracted map, which is a value that you extracted in the previous stage. This value is also a JSON object, so you can extract values from it as well. This stage extracts three values from the log value, is_secret, level, and log_line, and puts them into the extracted map with keys is_secret, level, and log_line.

If this were Python, it would be roughly equivalent to:

extracted_map = {
    "log": {
        "is_secret": "true",
        "level": "info",
        "message": "This is a secret message!",
    },
    "ts": "2023-11-16T06:01:50Z",
}

source = extracted_map["log"]

extracted_map["is_secret"] = source["is_secret"]
extracted_map["level"]     = source["level"]
extracted_map["log_line"]  = source["message"]

Extracted map before performing this stage:

{
    "log": {
        "is_secret": "true",
        "level": "info",
        "message": "This is a secret message!",
    },
    "ts": "2023-11-16T06:01:50Z",
}

Extracted map after performing this stage:

{
    "log": {
        "is_secret": "true",
        "level": "info",
        "message": "This is a secret message!",
    },
    "ts": "2023-11-16T06:01:50Z",
    "is_secret": "true",
    "level": "info",
    "log_line": "This is a secret message!",
}

Stage 4

stage.drop {
    source = "is_secret"
    value  = "true"
}

This stage acts on the is_secret value in the extracted map, which is a value that you extracted in the previous stage. This stage drops the log line if the value of is_secret is "true" and does not modify the extracted map. There are many other ways to filter logs, but this is a simple example. Refer to the loki.process#stage.drop documentation for more information.

Stage 5

stage.labels {
    values = {
        level = "",
    }
}

This stage adds a label to the log using the same shorthand as above (so this is equivalent to using values = { level = "level" }). This stage adds a label with key level and the value of level in the extracted map to the log ("info" from our example log line). This stage does not modify the extracted map.

Stage 6

stage.output {
    source = "log_line"
}

This stage uses the log_line value in the extracted map to set the actual log line that is forwarded to Loki. Rather than sending the entire JSON blob to Loki, you are only sending original_log_line["log"]["message"], along with some labels that you attached.

This stage does not modify the extracted map.

Putting it all together

Now that you have all of the pieces, let’s run the Grafana Agent and send some logs to it. Modify config.river with the config from the previous example and start the Grafana Agent with:

/path/to/agent run config.river

To get the current time in RFC3339 format, you can run:

date -u +"%Y-%m-%dT%H:%M:%SZ"

Try executing the following, replacing the "timestamp" value:

curl localhost:9999/loki/api/v1/raw -XPOST -H "Content-Type: application/json" -d '{"log": {"is_secret": "false", "level": "debug", "message": "This is a debug message!"}, "timestamp": <YOUR TIMESTAMP HERE>}'

Now that you have sent some logs, let’s see how they look in Grafana. Navigate to localhost:3000/explore and switch the Datasource to Loki. Try querying for {source="demo-api"} and see if you can find the logs you sent.

Try playing around with the values of "level", "message", "timestamp", and "is_secret" and see how the logs change. You can also try adding more stages to the loki.process component to extract more values from the logs, or add more labels.

Exercise

Since you are already using Docker and Docker exports logs, let’s get those logs into Loki. You can refer to the discovery.docker and loki.source.docker documentation for more information.

To ensure proper timestamps and other labels, make sure you use a loki.process component to process the logs before sending them to Loki.

Although you have not used it before, let’s use a discovery.relabel component to attach the container name as a label to the logs. You can refer to the discovery.relabel documentation for more information. The discovery.relabel component is very similar to the prometheus.relabel component, but is used to relabel discovered targets rather than metrics.

// Discover docker containers to collect logs from
discovery.docker "docker_containers" {
    // Note that if you are using Docker Desktop Engine this may need to be changed to
    // something like "unix:///${HOME}/.docker/desktop/docker.sock"
    host = "unix:///var/run/docker.sock"
}

// Extract container name from __meta_docker_container_name label and add as label
discovery.relabel "docker_containers" {
    targets = discovery.docker.docker_containers.targets

    rule {
        source_labels = ["__meta_docker_container_name"]
        target_label  = "container"
    }
}

// Scrape logs from docker containers and send to be processed
loki.source.docker "docker_logs" {
    host    = "unix:///var/run/docker.sock"
    targets = discovery.relabel.docker_containers.output
    forward_to = [loki.process.process_logs.receiver]
}

// Process logs and send to Loki
loki.process "process_logs" {
    stage.docker { }

    forward_to = [loki.write.local_loki.receiver]
}

loki.write "local_loki" {
    endpoint {
        url = "http://localhost:3100/loki/api/v1/push"
    }
}