This is documentation for the next version of Loki. For the latest stable release, go to the latest version.
Pipelines
A detailed look at how to set up Promtail to process your log lines, including extracting metrics and labels.
Pipeline
A pipeline is used to transform a single log line, its labels, and its timestamp. A pipeline is comprised of a set of stages. There are 4 types of stages:
- Parsing stages parse the current log line and extract data out of it. The extracted data is then available for use by other stages.
- Transform stages transform extracted data from previous stages.
- Action stages take extracted data from previous stages and do something
with them. Actions can:
- Add or modify existing labels to the log line
- Change the timestamp of the log line
- Change the content of the log line
- Create a metric based on the extracted data
- Filtering stages optionally apply a subset of stages or drop entries based on some condition.
Typical pipelines will start with a parsing stage (such as a regex or json stage) to extract data from the log line. Then, a series of action stages will be present to do something with that extracted data. The most common action stage will be a labels stage to turn extracted data into a label.
A common stage will also be the match stage to selectively apply stages or drop entries based on a LogQL stream selector and filter expressions.
Note that pipelines can not currently be used to deduplicate logs; Grafana Loki will receive the same log line multiple times if, for example:
- Two scrape configs read from the same file
- Duplicate log lines in a file are sent through a pipeline. Deduplication is not done.
However, Loki will perform some deduplication at query time for logs that have the exact same nanosecond timestamp, labels, and log contents.
This documented example gives a good glimpse of what you can achieve with a pipeline:
Data Accessible to Stages
The following sections further describe the types that are accessible to each stage (although not all may be used):
Label Set
The current set of labels for the log line. Initialized to be the set of labels that were scraped along with the log line. The label set is only modified by an action stage, but filtering stages read from it.
The final label set will be index by Loki and can be used for queries.
Extracted Map
A collection of key-value pairs extracted during a parsing stage. Subsequent stages operate on the extracted map, either transforming them or taking action with them. At the end of a pipeline, the extracted map is discarded; for a parsing stage to be useful, it must always be paired with at least one action stage.
The extracted map is initialized with the same set of initial labels that were
scraped along with the log line. This initial data allows for taking action on
the values of labels inside pipeline stages that only manipulate the extracted
map. For example, log entries tailed from files have the label filename
whose
value is the file path that was tailed. When a pipeline executes for that log
entry, the initial extracted map would contain filename
using the same value
as the label.
Log Timestamp
The current timestamp for the log line. Action stages can modify this value. If left unset, it defaults to the time when the log was scraped.
The final value for the timestamp is sent to Loki.
Log Line
The current log line, represented as text. Initialized to be the text that Promtail scraped. Action stages can modify this value.
The final value for the log line is sent to Loki as the text content for the given log entry.
Stages
Refer to the Promtail Stages Configuration Reference for the schema on the various supported stages supported.