Important: This documentation is about an older version. It's relevant only to the release noted, many of the features and functions have been updated or replaced. Please view the current version.

Clients

Promtail

Pipelines

Open source

Pipelines

A detailed look at how to set up Promtail to process your log lines, including extracting metrics and labels.

Pipeline

A pipeline is used to transform a single log line, its labels, and its timestamp. A pipeline is comprised of a set of stages. There are 4 types of stages:

Parsing stages parse the current log line and extract data out of it. The extracted data is then available for use by other stages.
Transform stages transform extracted data from previous stages.
Action stages take extracted data from previous stages and do something with them. Actions can:
1. Add or modify existing labels to the log line
2. Change the timestamp of the log line
3. Change the content of the log line
4. Create a metric based on the extracted data
Filtering stages optionally apply a subset of stages or drop entries based on some condition.

Typical pipelines will start with a parsing stage (such as a regex or json stage) to extract data from the log line. Then, a series of action stages will be present to do something with that extracted data. The most common action stage will be a labels stage to turn extracted data into a label.

A common stage will also be the match stage to selectively apply stages or drop entries based on a LogQL stream selector and filter expressions.

Note that pipelines can not currently be used to deduplicate logs; Grafana Loki will receive the same log line multiple times if, for example:

Two scrape configs read from the same file
Duplicate log lines in a file are sent through a pipeline. Deduplication is not done.

However, Loki will perform some deduplication at query time for logs that have the exact same nanosecond timestamp, labels, and log contents.

This documented example gives a good glimpse of what you can achieve with a pipeline:

scrape_configs:
- job_name: kubernetes-pods-name
  kubernetes_sd_configs: ....
  pipeline_stages:

  # This stage is only going to run if the scraped target has a label
  # of "name" with value "promtail".
  - match:
      selector: '{name="promtail"}'
      stages:
      # The regex stage parses out a level, timestamp, and component. At the end
      # of the stage, the values for level, timestamp, and component are only
      # set internally for the pipeline. Future stages can use these values and
      # decide what to do with them.
      - regex:
          expression: '.*level=(?P<level>[a-zA-Z]+).*ts=(?P<timestamp>[T\d-:.Z]*).*component=(?P<component>[a-zA-Z]+)'

      # The labels stage takes the level and component entries from the previous
      # regex stage and promotes them to a label. For example, level=error may
      # be a label added by this stage.
      - labels:
          level:
          component:

      # Finally, the timestamp stage takes the timestamp extracted from the
      # regex stage and promotes it to be the new timestamp of the log entry,
      # parsing it as an RFC3339Nano-formatted value.
      - timestamp:
          format: RFC3339Nano
          source: timestamp

  # This stage is only going to run if the scraped target has a label of
  # "name" with a value of "nginx" and if the log line contains the word "GET"
  - match:
      selector: '{name="nginx"} |= "GET"'
      stages:
      # This regex stage extracts a new output by matching against some
      # values and capturing the rest.
      - regex:
          expression: \w{1,3}.\w{1,3}.\w{1,3}.\w{1,3}(?P<output>.*)

      # The output stage changes the content of the captured log line by
      # setting it to the value of output from the regex stage.
      - output:
          source: output

  # This stage is only going to run if the scraped target has a label of
  # "name" with a value of "jaeger-agent".
  - match:
      selector: '{name="jaeger-agent"}'
      stages:
      # The JSON stage reads the log line as a JSON string and extracts
      # the "level" field from the object for use in further stages.
      - json:
          expressions:
            level: level

      # The labels stage pulls the value from "level" that was extracted
      # from the previous stage and promotes it to a label.
      - labels:
          level:
- job_name: kubernetes-pods-app
  kubernetes_sd_configs: ....
  pipeline_stages:
  # This stage will only run if the scraped target has a label of "app"
  # with a name of *either* grafana or prometheus.
  - match:
      selector: '{app=~"grafana|prometheus"}'
      stages:
      # The regex stage will extract a level and component for use in further
      # stages, allowing the level to be defined as either lvl=<level> or
      # level=<level> and the component to be defined as either
      # logger=<component> or component=<component>
      - regex:
          expression: ".*(lvl|level)=(?P<level>[a-zA-Z]+).*(logger|component)=(?P<component>[a-zA-Z]+)"

      # The labels stage then promotes the level and component extracted from
      # the regex stage to labels.
      - labels:
          level:
          component:

  # This stage will only run if the scraped target has a label "app"
  # with a value of "some-app" and the log line doesn't contain the word "info"
  - match:
      selector: '{app="some-app"} != "info"'
      stages:
      # The regex stage tries to extract a Go panic by looking for panic:
      # in the log message.
      - regex:
          expression: ".*(?P<panic>panic: .*)"

      # The metrics stage is going to increment a panic_total metric counter
      # which Promtail exposes. The counter is only incremented when panic
      # was extracted from the regex stage.
      - metrics:
          panic_total:
            type: Counter
            description: "total count of panic"
            source: panic
            config:
              action: inc

Data Accessible to Stages

The following sections further describe the types that are accessible to each stage (although not all may be used):

Label Set

The current set of labels for the log line. Initialized to be the set of labels that were scraped along with the log line. The label set is only modified by an action stage, but filtering stages read from it.

The final label set will be index by Loki and can be used for queries.

Extracted Map

A collection of key-value pairs extracted during a parsing stage. Subsequent stages operate on the extracted map, either transforming them or taking action with them. At the end of a pipeline, the extracted map is discarded; for a parsing stage to be useful, it must always be paired with at least one action stage.

The extracted map is initialized with the same set of initial labels that were scraped along with the log line. This initial data allows for taking action on the values of labels inside pipeline stages that only manipulate the extracted map. For example, log entries tailed from files have the label filename whose value is the file path that was tailed. When a pipeline executes for that log entry, the initial extracted map would contain filename using the same value as the label.