Promtail pipeline stages on Grafana Labs

cri

Sat, 11 Apr 2026 00:53:11 +0000

cri

The cri stage is a parsing stage that reads the log line using the standard CRI logging format.

Schema

cri:
  # Max buffer size to hold partial lines.
  [max_partial_lines: <int> | default = 100]

  # Max line size to hold a single partial line, if max_partial_line_size_truncate is true. Example: 262144.
  [max_partial_line_size: <int> | default = 0]

  # Allows to pretruncate partial lines before storing in partial buffer.
  [max_partial_line_size_truncate: <bool> | default = false]

Unlike most stages, the cri stage provides no configuration options and only supports the specific CRI log format. CRI specifies log lines as space-delimited values with the following components:

time: The timestamp string of the log
stream: Either stdout or stderr
flags: CRI flags including F or P
log: The contents of the log line

No whitespace is permitted between the components. In the following example, only the first log line can be properly formatted using the cri stage:

"2019-01-01T01:00:00.000000001Z stderr P test\ngood"
"2019-01-01 T01:00:00.000000001Z stderr testgood"
"2019-01-01T01:00:00.000000001Z testgood"

Examples

For the given pipeline:

- cri: {}

Given the following log line:

"2019-04-30T02:12:41.8443515Z stdout F message"

The following key-value pairs would be created in the set of extracted data:

content: message
stream: stdout
flags: F
timestamp: 2019-04-30T02:12:41.8443515 - The cri-stage both extracts the timestamp as a label and set it as the timestamp of the log entry.

decolorize

Sat, 11 Apr 2026 00:53:11 +0000

decolorize

The decolorize stage is a transform stage that lets you strip ANSI color codes from the log line, thus making it easier to parse logs further.

There are examples below to help explain.

Decolorize stage schema

decolorize:
  # Currently this stage has no configurable options

Examples

The following is an example showing the use of the decolorize stage.

Given the pipeline:

- decolorize:

Would turn each line having a color code into a non-colored one, e.g.

[2022-11-04 22:17:57.811] \033[0;32http\033[0m: GET /_health (0 ms) 204

is turned into

[2022-11-04 22:17:57.811] http: GET /_health (0 ms) 204

docker

Sat, 11 Apr 2026 00:53:11 +0000

docker

The docker stage is a parsing stage that reads log lines in the standard format of Docker log files.

Schema

docker: {}

Unlike most stages, the docker stage provides no configuration options and only supports the specific Docker log format. Each log line from Docker is written as JSON with the following keys:

log: The content of log line
stream: Either stdout or stderr
time: The timestamp string of the log line

Examples

For the given pipeline:

- docker: {}

Given the following log line:

{"log":"log message\n","stream":"stderr","time":"2019-04-30T02:12:41.8443515Z"}

The following key-value pairs would be created in the set of extracted data:

output: log message\n
stream: stderr
timestamp: 2019-04-30T02:12:41.8443515

drop

Sat, 11 Apr 2026 00:53:11 +0000

drop

The drop stage is a filtering stage that lets you drop logs based on several options.

It’s important to note that if you provide multiple options they will be treated like an AND clause, where each option has to be true to drop the log.

If you wish to drop with an OR clause, then specify multiple drop stages.

There are examples below to help explain.

Drop stage schema

drop:
  # Single name or names list of extracted data. If empty, uses the log message.
  [source: [<string>] | <string>]

  # Separator placed between concatenated extracted data names. The default separator is a semicolon.
  [separator: <string> | default = ";"]

  # RE2 regular expression. If `source` is provided and it's a list, the regex will attempt to match
  # the concatenated sources. If no source is provided, then the regex attempts
  # to match the log line.
  # If the provided regex matches the log line or the source, the line will be dropped.
  [expression: <string>]

  # value can only be specified when source is specified. If `source` is provided and it's a list,
  # the value will attempt to match the concatenated sources. It is an error to specify value and expression.
  # If the value provided is an exact match for the `source` the line will be dropped.
  [value: <string>]

  # older_than will be parsed as a Go duration: https://golang.org/pkg/time/#ParseDuration
  # If the log line timestamp is older than the current time minus the provided duration it will be dropped.
  [older_than: <duration>]

  # longer_than is a value in bytes, any log line longer than this value will be dropped.
  # Can be specified as an exact number of bytes in integer format: 8192
  # Or can be expressed with a suffix such as 8kb
  [longer_than: <string>|<int>]

  # Every time a log line is dropped the metric `logentry_dropped_lines_total`
  # will be incremented.  By default the reason label will be `drop_stage`
  # however you can optionally specify a custom value to be used in the `reason`
  # label of that metric here.
  [drop_counter_reason: <string> | default = "drop_stage"]

Examples

The following are examples showing the use of the drop stage.

Simple drops

Simple drop stage configurations only specify one of the options, or two options when using the source option.

Given the pipeline:

- drop:
    source: ["level","msg"]

Drops any log line that has an extracted data field of at least level or msg.

Regex match a line

This example pipeline drops any log line with the substring “debug” in it:

- drop:
    expression: ".*debug.*"

Regex match concatenated sources

Given the pipeline:

- json:
    expressions:
     level:
     msg:
- drop:
    source:     ["level","msg"]
    separator:   "#"
    expression:  "(error|ERROR)#.*\/loki\/api\/push.*"

Drops both of these log lines:

{"time":"2019-01-01T01:00:00.000000001Z", "level": "error", "msg":"11.11.11.11 - "POST /loki/api/push/ HTTP/1.1" 200 932 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 GTB6"}
{"time":"2019-01-01T01:00:00.000000001Z", "level": "ERROR", "msg":"11.11.11.11 - "POST /loki/api/push/ HTTP/1.1" 200 932 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 GTB6"}

Value match a source

Given the pipeline:

- json:
    expressions:
     level:
     msg:
- drop:
    source: "level"
    value:  "error"

Would drop this log line:

{"time":"2019-01-01T01:00:00.000000001Z", "level": "error", "msg":"11.11.11.11 - "POST /loki/api/push/ HTTP/1.1" 200 932 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 GTB6"}

Drop old log lines

Given the pipeline:

- json:
    expressions:
     time:
     msg:
- timestamp:
    source: time
    format: RFC3339
- drop:
    older_than: 24h
    drop_counter_reason: "line_too_old"

With a current ingestion time of 2020-08-12T12:00:00Z would drop this log line when read from a file:

{"time":"2020-08-11T11:00:00Z", "level": "error", "msg":"11.11.11.11 - "POST /loki/api/push/ HTTP/1.1" 200 932 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 GTB6"}

However it would not drop this log line:

{"time":"2020-08-11T13:00:00Z", "level": "error", "msg":"11.11.11.11 - "POST /loki/api/push/ HTTP/1.1" 200 932 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 GTB6"}

In this example the current time is 2020-08-12T12:00:00Z and older_than is 24h. All log lines which have a timestamp older than 2020-08-11T12:00:00Z will be dropped.

All lines dropped by this drop stage would also increment the logentry_dropped_lines_total metric with a label reason="line_too_old"

Dropping long log lines

Given the pipeline:

- drop:
    longer_than: 8kb
    drop_counter_reason: "line_too_long"

Would drop any log line longer than 8kb bytes, this is useful when Loki would reject a line for being too long.

All lines dropped by this drop stage would also increment the logentry_dropped_lines_total metric with a label reason="line_too_long"

Complex drops

Complex drop stage configurations specify multiple options in one stage or specify multiple drop stages

Drop logs by regex AND length

Given the pipeline:

- drop:
    expression: ".*debug.*"
    longer_than: 1kb

Would drop all logs that contain the word debug AND are longer than 1kb bytes

Drop logs by time OR length OR regex

Given the pipeline:

- json:
    expressions:
     time:
     msg:
- timestamp:
    source: time
    format: RFC3339
- drop:
    older_than: 24h
- drop:
    longer_than: 8kb
- drop:
    source: "msg"
    expression: ".*trace.*"

Would drop all logs older than 24h OR longer than 8kb bytes OR have a json msg field containing the word trace

eventlogmessage

Sat, 11 Apr 2026 00:53:11 +0000

eventlogmessage

The eventlogmessage stage is a parsing stage that extracts data from the Message string that appears in the Windows Event Log.

Schema

eventlogmessage:
  # Name from extracted data to parse, defaulting to the name
  # used by the windows_events scraper
  [source: <string> | default = message]

  # If previously extracted data exists for a key that occurs
  # in the Message, when true, the previous value will be
  # overwriten by the value in the Message. Otherwise,
  # '_extracted' will be appended to the key that is used for
  # the value in the Message.
  [overwrite_existing: <bool> | default = false]

  # When true, keys extracted from the Message that are not
  # valid labels will be dropped, otherwise they will be
  # automatically converted into valid labels replacing invalid
  # characters with underscores
  [drop_invalid_labels: <bool> | default = false]

The extracted data can hold non-string values and this stage does not do any type conversions; downstream stages will need to perform correct type conversion of these values as necessary. Please refer to the the template stage for how to do this.

Example combined with json

For the given pipeline:

- json:
    expressions:
      message:
      Overwritten:
- eventlogmessage:
    source: message
    overwrite_existing: true

Given the following log line:

{"event_id": 1, "Overwritten": "old", "message": "Message type:\r\nOverwritten: new\r\nImage: C:\\Users\\User\\promtail.exe"}

The first stage would create the following key-value pairs in the set of extracted data:

message: Message type:\r\nOverwritten: new\r\nImage: C:\Users\User\promtail.exe
Overwritten: old

The second stage will parse the value of message from the extracted data and append/overwrite the following key-value pairs to the set of extracted data:

Image: C:\\Users\\User\\promtail.exe
Message_type: (empty string)
Overwritten: new

geoip

Sat, 11 Apr 2026 00:53:11 +0000

geoip

The geoip stage is a parsing stage that reads an ip address and populates the labelset with geoip fields. Maxmind’s GeoIP2 database is used for the lookup.

Populated fields for City db:

geoip_city_name
geoip_country_name
geoip_continent_name
geoip_continent_code
geoip_location_latitude
geoip_location_longitude
geoip_postal_code
geoip_timezone
geoip_subdivision_name
geoip_subdivision_code

Populated fields for ASN (Autonomous System Number) db:

geoip_autonomous_system_number
geoip_autonomous_system_organization

Schema

geoip:
  # Path to the Maxmind DB file
  [db: <string>]

  # IP from extracted data to parse.
  [source: <string>]

  # Maxmind DB type. Allowed values are "city", "asn"
  [db_type: <string>]

GeoIP with City database example

For the given pipeline

- regex:
    expression: "^(?P<ip>\S+) .*"
- geoip:
    db: "/path/to/GeoIP2-City.mmdb"
    source: "ip"
    db_type: "city"

And the log line:

"34.120.177.193 - "POST /loki/api/push/ HTTP/1.1" 200 932 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 GTB6"

The regex stage parses the log line and ip is extracted. Then the extracted ip value is given as source to geoip stage. The geoip stage performs a lookup on the ip and populates the following labels:

geoip_city_name: Kansas City
geoip_country_name: United States
geoip_continent_name: North America
geoip_continent_code: NA
geoip_location_latitude: "39.1027
geoip_location_longitude: -94.5778
geoip_postal_code: 64184
geoip_timezone: America/Chicago
geoip_subdivision_name: Missouri
geoip_subdivision_code: MO

If only a subset of these labels are required, you can chain the above pipeline with the labeldrop or labelallow stage.

labelallow example

- regex:
    expression: "^(?P<ip>\S+) .*"
- geoip:
    db: "/path/to/GeoCity.mmdb"
    source: "ip"
    db_type: "city"
- labelallow:
  - geoip_city_name
  - geoip_country_name
  - geoip_location_latitude
  - geoip_location_longitude

Only the labels listed under labelallow will be sent to Loki.

labeldrop example

- regex:
    expression: "^(?P<ip>\S+) .*"
- geoip:
    db: "/path/to/GeoCity.mmdb"
    source: "ip"
    db_type: "city"
- labeldrop:
  - geoip_postal_code
  - geoip_subdivision_code

All the labels except the ones listed under labeldrop will be sent to Loki.

GeoIP with ASN (Autonomous System Number) database example

- regex:
    expression: "^(?P<ip>\S+) .*"
- geoip:
    db: "/path/to/GeoIP2-ASN.mmdb"
    source: "ip"
    db_type: "asn"

And the log line:

"34.120.177.193 - "POST /loki/api/push/ HTTP/1.1" 200 932 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 GTB6"

geoip_autonomous_system_number: 396982
geoip_autonomous_system_organization: GOOGLE-CLOUD-PLATFORM

For more information and real life example, see Protect PII and add geolocation data: Monitoring legacy systems with Grafana which has real-life examples on how to infuse dashboards with geo-location data.

json

Sat, 11 Apr 2026 00:53:11 +0000

json

The json stage is a parsing stage that reads the log line as JSON and accepts JMESPath expressions to extract data.

Schema

json:
  # Set of key/value pairs of JMESPath expressions. The key will be
  # the key in the extracted data while the expression will be the value,
  # evaluated as a JMESPath from the source data.
  #
  # Literal JMESPath expressions can be done by wrapping a key in
  # double quotes, which then must be wrapped in single quotes in
  # YAML so they get passed to the JMESPath parser.
  expressions:
    [ <string>: <string> ... ]

  # Name from extracted data to parse. If empty, uses the log message.
  [source: <string>]

  # When true, then any lines that cannot be successfully parsed as valid JSON objects
  # will be dropped instead of being sent to Loki.
  [drop_malformed: <bool> | default = false]

This stage uses the Go JSON unmarshaler, which means non-string types like numbers or booleans will be unmarshaled into those types. The extracted data can hold non-string values and this stage does not do any type conversions; downstream stages will need to perform correct type conversion of these values as necessary. Please refer to the the template stage for how to do this.

If the value extracted is a complex type, such as an array or a JSON object, it will be converted back into a JSON string before being inserted into the extracted data.

Examples

Using log line

For the given pipeline:

- json:
    expressions:
      output: log
      stream: stream
      timestamp: time

Given the following log line:

{"log":"log message\n","stream":"stderr","time":"2019-04-30T02:12:41.8443515Z"}

The following key-value pairs would be created in the set of extracted data:

output: log message\n
stream: stderr
timestamp: 2019-04-30T02:12:41.8443515

Using extracted data

For the given pipeline:

- json:
    expressions:
      output:    log
      stream:    stream
      timestamp: time
      extra:
- json:
    expressions:
      user:
    source: extra

And the given log line:

{"log":"log message\n","stream":"stderr","time":"2019-04-30T02:12:41.8443515Z","extra":"{\"user\":\"marco\"}"}

The first stage would create the following key-value pairs in the set of extracted data:

output: log message\n
stream: stderr
timestamp: 2019-04-30T02:12:41.8443515
extra: {"user": "marco"}

The second stage will parse the value of extra from the extracted data as JSON and append the following key-value pairs to the set of extracted data:

user: marco

Using a JMESPath Literal

This pipeline uses a literal JMESPath expression to parse JSON fields with special characters in the name, like @ or .

For the given pipeline:

- json:
    expressions:
      output: log
      stream: '"grpc.stream"'
      timestamp: time

Given the following log line:

{"log":"log message\n","grpc.stream":"stderr","time":"2019-04-30T02:12:41.8443515Z"}

The following key-value pairs would be created in the set of extracted data:

output: log message\n
stream: stderr
timestamp: 2019-04-30T02:12:41.8443515

labelallow

Sat, 11 Apr 2026 00:53:11 +0000

labelallow

The labelallow stage is an action stage that allows only the provided labels to be included in the label set that is sent to Loki with the log entry.

Schema

labelallow:
  - [<string>]
  ...

Examples

For the given pipeline:

kubernetes_sd_configs:
 - role: pod 
pipeline_stages:
- docker: {}    
- labelallow:
    - kubernetes_pod_name
    - kubernetes_container_name

Given the following incoming labels:

kubernetes_pod_name: "loki-pqrs"
kubernetes_container_name: "loki"
kubernetes_pod_template_hash: "79f5db67b"
kubernetes_controller_revision_hash: "774858987d"

Only the below labels would be sent to loki

kubernetes_pod_name: "loki-pqrs"
kubernetes_container_name: "loki"

labeldrop

Sat, 11 Apr 2026 00:53:11 +0000

labeldrop

The labeldrop stage is an action stage that drops labels from the label set that is sent to Loki with the log entry.

Schema

labeldrop:
  - [<string>]
  ...

Examples

For the given pipeline:

- replace:
    expression: "(.*)"
    replace: "pod_name:{{ .kubernetes_pod_name }} {{ .Value }}"
- labeldrop:
    - kubernetes_pod_name

Given the following log line:

log message\n

The first stage would append the value of thekubernetes_pod_name label into the beginning of the log line. The labeldrop stage would drop the label from being sent to Loki, and it would now be part of the log line instead.

labels

Sat, 11 Apr 2026 00:53:11 +0000

labels

The labels stage is an action stage that takes data from the extracted map and modifies the label set that is sent to Loki with the log entry.

Schema

labels:
  # Key is REQUIRED and the name for the label that will be created.
  # Value is optional and will be the name from extracted data whose value
  # will be used for the value of the label. If empty, the value will be
  # inferred to be the same as the key.
  [ <string>: [<string>] ... ]

Examples

For the given pipeline:

- json:
    expressions:
      stream: stream
- labels:
    stream:

Given the following log line:

{"log":"log message\n","stream":"stderr","time":"2019-04-30T02:12:41.8443515Z"}

The first stage would extract stream into the extracted map with a value of stderr. The labels stage would turn that key-value pair into a label, so the log line sent to Loki would include the label stream with a value of stderr.

limit

Sat, 11 Apr 2026 00:53:11 +0000

limit

The limit stage is a rate-limiting stage that throttles logs based on several options.

Limit stage schema

This pipeline stage places limits on the rate or burst quantity of log lines that Promtail pushes to Loki. The concept of having distinct burst and rate limits mirrors the approach to limits that can be set for the Loki distributor component: ingestion_rate_mb and ingestion_burst_size_mb, as defined in limits_config.

limit:
  # The rate limit in lines per second that Promtail will push to Loki
  [rate: <int>]

  # The cap in the quantity of burst lines that Promtail will push to Loki
  [burst: <int>]
   
  # Ratelimit each label value independently. If label is not found, log line is not
  # considered for ratelimiting. Drop must be true if this is set.
  [by_label_name: <string>]  
    
  # When ratelimiting by label is enabled, keep track of this many last used labels
  [max_distinct_labels: <int> | default = 10000]  

  # When drop is true, log lines that exceed the current rate limit will be discarded.
  # When drop is false, log lines that exceed the current rate limit will only wait
  # to enter the back pressure mode. 
  [drop: <bool> | default = false]

Examples

The following are examples showing the use of the limit stage.

limit

Simple limit stage configurations.

Match a line and throttle

Given the pipeline:

- limit:
    rate: 10
    burst: 10

Would throttle any log line.

Match a line and drop

Given the pipeline:

- limit:
    rate: 10
    burst: 10
    drop: true

Would throttle any log line and drop logs when rate limit.

Ratelimit by a label

Given the pipeline:

- limit:
    rate: 10
    burst: 10
    drop: true
    by_label_name: "namespace"

Would ratelimit messages originating from each namespace independently. Any message without namespace label will not be ratelimited.

logfmt

Sat, 11 Apr 2026 00:53:11 +0000

logfmt

The logfmt stage is a parsing stage that reads the log line as logfmt and allows extraction of data into labels.

Schema

logfmt:
  # Set of key/value pairs for mapping of logfmt fields to extracted labels. The YAML key will be
  # the key in the extracted data, while the expression will be the YAML value. If the value
  # is empty, then the logfmt field with the same name is extracted.
  mapping:
    [ <string>: <string> ... ]

  # Name from extracted data to parse. If empty, uses the log message.
  [source: <string>]

This stage uses the go-logfmt unmarshaler, which means non-string types like numbers or booleans will be unmarshaled into those types. The extracted data can hold non-string values, and this stage does not do any type conversions; downstream stages will need to perform correct type conversion of these values as necessary. Please refer to the template stage for how to do this.

If the value extracted is a complex type, its value is extracted as a string.

Examples

Using log line

For the given pipeline:

- logfmt:
    mapping:
      timestamp: time
      app:
      duration:
      unknown:

Given the following log line:

time=2012-11-01T22:08:41+00:00 app=loki level=WARN duration=125 message="this is a log line" extra="user=foo"

The following key-value pairs would be created in the set of extracted data:

timestamp: 2012-11-01T22:08:41+00:00
app: loki
duration: 125

Using extracted data

For the given pipeline:

- logfmt:
    mapping:
      extra:
- logfmt:
    mapping:
      user:
    source: extra

And the given log line:

time=2012-11-01T22:08:41+00:00 app=loki level=WARN duration=125 message="this is a log line" extra="user=foo"

The first stage would create the following key-value pairs in the set of extracted data:

extra: user=foo

The second stage will parse the value of extra from the extracted data as logfmt and append the following key-value pairs to the set of extracted data:

user: foo

match

Sat, 11 Apr 2026 00:53:11 +0000

match

The match stage is a filtering stage that conditionally applies a set of stages or drop entries when a log entry matches a configurable LogQL stream selector and filter expressions.

Schema

match:
  # LogQL stream selector and line filter expressions.
  selector: <string>

  # Names the pipeline. When defined, creates an additional label in
  # the pipeline_duration_seconds histogram, where the value is
  # concatenated with job_name using an underscore.
  [pipeline_name: <string>]

  # Determines what action is taken when the selector matches the log
  # line. Defaults to keep. When set to drop, entries will be dropped
  # and no later metrics will be recorded.
  # Stages must be not defined when dropping entries.
  [action: <string> | default = "keep"]
  
  # If you specify `action: drop` the metric `logentry_dropped_lines_total` 
  # will be incremented for every line dropped.  By default the reason
  # label will be `match_stage` however you can optionally specify a custom value 
  # to be used in the `reason` label of that metric here.
  [drop_counter_reason: <string> | default = "match_stage"]

  # Nested set of pipeline stages only if the selector
  # matches the labels of the log entries:
  stages:
    [<stages>...]

Refer to the Promtail Stages Configuration Reference for the schema on the various stages supported here.

Example

For the given pipeline:

pipeline_stages:
- json:
    expressions:
      app:
- labels:
    app:
- match:
    selector: '{app="loki"}'
    stages:
    - json:
        expressions:
          msg: message
- match:
    pipeline_name: "app2"
    selector: '{app="pokey"}'
    action: keep
    stages:
    - json:
        expressions:
          msg: msg
- match:
    selector: '{app="promtail"} |~ ".*noisy error.*"'
    action: drop
    drop_counter_reason: promtail_noisy_error
- output:
    source: msg

And given log lines:

{ "time":"2012-11-01T22:08:41+00:00", "app":"loki", "component": ["parser","type"], "level" : "WARN", "message" : "app1 log line" }
{ "time":"2012-11-01T22:08:41+00:00", "app":"promtail", "component": ["parser","type"], "level" : "ERROR", "message" : "foo noisy error" }

The first stage will add app with a value of loki into the extracted map for the first log line, while the second stage will add app as a label (again with the value of loki). The second line will follow the same flow and will be added the label app with a value of promtail.

The third stage uses LogQL to only execute the nested stages when there is a label of app whose value is loki. This matches the first line in our case; the nested json stage then adds msg into the extracted map with a value of app1 log line.

The fourth stage uses LogQL to only executed the nested stages when there is a label of app whose value is pokey. This does not match in our case, so the nested json stage is not ran.

The fifth stage will drop any entries from the application promtail that matches the regex .*noisy error. and will also increment the logentry_dropped_lines_total metric with a label reason="promtail_noisy_error"

The final output stage changes the contents of the log line to be the value of msg from the extracted map. In this case, the log line is changed to app1 log line.

metrics

Sat, 11 Apr 2026 00:53:11 +0000

metrics

The metrics stage is an action stage that allows for defining and updating metrics based on data from the extracted map. Note that created metrics are not pushed to Loki and are instead exposed via Promtail’s /metrics endpoint. Prometheus should be configured to scrape Promtail to be able to retrieve the metrics configured by this stage. If Promtail’s configuration is reloaded, all metrics will be reset.

Schema

# A map where the key is the name of the metric and the value is a specific
# metric type.
metrics:
  [<string>: [ <metric_counter> | <metric_gauge> | <metric_histogram> ] ...]

metric_counter

Defines a counter metric whose value only goes up.

# The metric type. Must be Counter.
type: Counter

# Describes the metric.
[description: <string>]

# Defines custom prefix name for the metric. If undefined, default name "promtail_custom_" will be prefixed.
[prefix: <string>]

# Key from the extracted data map to use for the metric,
# defaulting to the metric's name if not present.
[source: <string>]

# Label values on metrics are dynamic which can cause exported metrics
# to go stale (for example when a stream stops receiving logs).
# To prevent unbounded growth of the /metrics endpoint any metrics which
# have not been updated within this time will be removed.
# Must be greater than or equal to '1s', if undefined default is '5m'
[max_idle_duration: <string>]

config:
  # If present and true all log lines will be counted without attempting
  # to match the `value` to the field specified by `source` in the extracted map.
  # It is an error to specify `match_all: true` and also specify a `value`
  [match_all: <bool>]

  # If present and true all log line bytes will be counted.
  # It is an error to specify `count_entry_bytes: true` without specifying `match_all: true`
  # It is an error to specify `count_entry_bytes: true` without specifying `action: add`
  [count_entry_bytes: <bool>]

  # Filters down source data and only changes the metric
  # if the targeted value exactly matches the provided string.
  # If not present, all data will match.
  [value: <string>]

  # Must be either "inc" or "add" (case insensitive). If
  # inc is chosen, the metric value will increase by 1 for each
  # log line received that passed the filter. If add is chosen,
  # the extracted value must be convertible to a positive float
  # and its value will be added to the metric.
  action: <string>

metric_gauge

Defines a gauge metric whose value can go up or down.

# The metric type. Must be Gauge.
type: Gauge

# Describes the metric.
[description: <string>]

# Defines custom prefix name for the metric. If undefined, default name "promtail_custom_" will be prefixed.
[prefix: <string>]

# Key from the extracted data map to use for the metric,
# defaulting to the metric's name if not present.
[source: <string>]

# Label values on metrics are dynamic which can cause exported metrics
# to go stale (for example when a stream stops receiving logs).
# To prevent unbounded growth of the /metrics endpoint any metrics which
# have not been updated within this time will be removed.
# Must be greater than or equal to '1s', if undefined default is '5m'
[max_idle_duration: <string>]

config:
  # Filters down source data and only changes the metric
  # if the targeted value exactly matches the provided string.
  # If not present, all data will match.
  [value: <string>]

  # Must be either "set", "inc", "dec"," add", or "sub". If
  # add, set, or sub is chosen, the extracted value must be
  # convertible to a positive float. inc and dec will increment
  # or decrement the metric's value by 1 respectively.
  action: <string>

metric_histogram

Defines a histogram metric whose values are bucketed.

# The metric type. Must be Histogram.
type: Histogram

# Describes the metric.
[description: <string>]

# Defines custom prefix name for the metric. If undefined, default name "promtail_custom_" will be prefixed.
[prefix: <string>]

# Key from the extracted data map to use for the metric,
# defaulting to the metric's name if not present.
[source: <string>]

# Label values on metrics are dynamic which can cause exported metrics
# to go stale (for example when a stream stops receiving logs).
# To prevent unbounded growth of the /metrics endpoint any metrics which
# have not been updated within this time will be removed.
# Must be greater than or equal to '1s', if undefined default is '5m'
[max_idle_duration: <string>]

config:
  # Filters down source data and only changes the metric
  # if the targeted value exactly matches the provided string.
  # If not present, all data will match.
  [value: <string>]

  # Holds all the numbers in which to bucket the metric.
  buckets:
    - <int>

Examples

Counter

- metrics:
    log_lines_total:
      type: Counter
      description: "total number of log lines"
      prefix: my_promtail_custom_
      max_idle_duration: 24h
      config:
        match_all: true
        action: inc
    log_bytes_total:
      type: Counter
      description: "total bytes of log lines"
      prefix: my_promtail_custom_
      max_idle_duration: 24h
      config:
        match_all: true
        count_entry_bytes: true
        action: add

This pipeline creates a log_lines_total counter which increments for every log line received by using the match_all: true parameter.

It also creates a log_bytes_total counter which adds the byte size of every log line received to the counter by using the count_entry_bytes: true parameter.

Those two metrics will disappear after 24h if they don’t receive new entries, this is useful to reduce the building up of stage metrics.

The combination of these two metric stages will give you two counters to track the volume of every log stream in both number of lines and bytes, which can be useful in identifying sources of very high volume, as well as helping understand why you may have too much cardinality.

These stages should be placed towards the end of your pipeline after any labels stages

- regex:
    expression: "^.*(?P<order_success>order successful).*$"
- metrics:
    successful_orders_total:
      type: Counter
      description: "log lines with the message `order successful`"
      source: order_success
      config:
        action: inc

This pipeline first tries to find order successful in the log line, extracting it as the order_success field in the extracted map. The metrics stage then creates a metric called successful_orders_total whose value only increases when order_success was found in the extracted map.

The result of this pipeline is a metric whose value only increases when a log line with the text order successful was scraped by Promtail.

- regex:
    expression: "^.* order_status=(?P<order_status>.*?) .*$"
- metrics:
    successful_orders_total:
      type: Counter
      description: "successful orders"
      source: order_status
      config:
        value: success
        action: inc
    failed_orders_total:
      type: Counter
      description: "failed orders"
      source: order_status
      config:
        value: fail
        action: inc

This pipeline first tries to find text in the format order_status=<value> in the log line, pulling out the <value> into the extracted map with the key order_status.

The metrics stage creates successful_orders_total and failed_orders_total metrics that only increment when the value of order_status in the extracted map is success or fail respectively.

Gauge

Gauge examples will be very similar to Counter examples with additional action values.

- regex:
    expression: "^.* retries=(?P<retries>\d+) .*$"
- metrics:
    retries_total:
      type: Gauge
      description: "total retries"
      source: retries
      config:
        action: add

This pipeline first tries to find text in the format retries=<value> in the log line, pulling out the <value> into the extracted map with the key retries. Note that the regex only parses numbers for the value in retries.

The metrics stage then creates a Gauge whose current value will be added to the number in the retries field from the extracted map.

Histogram

- metrics:
    http_response_time_seconds:
      type: Histogram
      description: "distribution of log response time"
      source: response_time
      config:
        buckets: [0.001,0.0025,0.005,0.010,0.025,0.050]

This pipeline creates a histogram that reads response_time from the extracted map and places it into a bucket, both increasing the count of the bucket and the sum for that particular bucket.

Supported values

The metric values extracted from the log data are internally converted to floating points. The supported values are the following:

integers, floating point numbers
string - two types of string formats are supported:
- strings that represent floating point numbers: e.g., "0.804" is converted to 0.804.
- duration format strings. Valid time units are “ns”, “us”, “ms”, “s”, “m”, “h”. Values in this format are converted as a floating point number of seconds. E.g., "0.5ms" is converted to 0.0005.
boolean:
- true is converted to 1
- false is converted to 0

multiline

Sat, 11 Apr 2026 00:53:11 +0000

multiline

The multiline stage merges multiple lines into a multiline block before passing it on to the next stage in the pipeline.

A new block is identified by the firstline regular expression. Any line that does not match the expression is considered to be part of the block of the previous match.

Schema

multiline:
  # RE2 regular expression, if matched will start a new multiline block.
  # This expression must be provided.
  firstline: <string>

  # The maximum wait time will be parsed as a Go duration: https://golang.org/pkg/time/#ParseDuration.
  # If no new logs arrive within this maximum wait time, the current block will be sent on.
  # This is useful if the observed application dies with, for example, an exception.
  # No new logs will arrive and the exception
  # block is sent *after* the maximum wait time expires.
  # It defaults to 3s.
  max_wait_time: <duration>

  # Maximum number of lines a block can have. If the block has more lines, a new block is started.
  # The default is 128 lines.
  max_lines: <integer>

Examples

Predefined Log Format

Consider these logs from a simple flask service.

[2020-12-03 11:36:20] "GET /hello HTTP/1.1" 200 -
[2020-12-03 11:36:23] ERROR in app: Exception on /error [GET]
Traceback (most recent call last):
  File "/home/pallets/.pyenv/versions/3.8.5/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/pallets/.pyenv/versions/3.8.5/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/pallets/.pyenv/versions/3.8.5/lib/python3.8/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/pallets/.pyenv/versions/3.8.5/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/home/pallets/.pyenv/versions/3.8.5/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/pallets/.pyenv/versions/3.8.5/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/pallets/src/deployment_tools/hello.py", line 10, in error
    raise Exception("Sorry, this route always breaks")
Exception: Sorry, this route always breaks
[2020-12-03 11:36:23] "GET /error HTTP/1.1" 500 -
[2020-12-03 11:36:26] "GET /hello HTTP/1.1" 200 -
[2020-12-03 11:36:27] "GET /hello HTTP/1.1" 200 -

We would like to collapse all lines of the traceback into one multiline block. In this example, all blocks start with a timestamp in brackets. Thus, we configure a multiline stage with the firstline regular expression ^\[\d{4}-\d{2}-\d{2} \d{1,2}:\d{2}:\d{2}\]. This will match the start of the traceback, but not the following lines until Exception: Sorry, this route always breaks. These will be part of a multiline block and one log entry in Loki.

- multiline:
    # Identify timestamps as first line of a multiline block. Enclose the string in single quotes.
    firstline: '^\[\d{4}-\d{2}-\d{2} \d{1,2}:\d{2}:\d{2}\]'
    max_wait_time: 3s
- regex:
    # Flag (?s:.*) needs to be set for regex stage to capture full traceback log in the extracted map.
    expression: '^(?P<time>\[\d{4}-\d{2}-\d{2} \d{1,2}:\d{2}:\d{2}\]) (?P<message>(?s:.*))$'

Custom Log Format

The example assumed you had no control over the log format. Thus, it required a more elaborate regular expression to match the first line. If you can control the log format of the system under observation, we can simplify the first line matching.

This time we are looking at the logs of a simple Akka HTTP service.

[2021-01-07 14:17:43,494] [DEBUG] [akka.io.TcpListener] [HelloAkkaHttpServer-akka.actor.default-dispatcher-26] [akka://HelloAkkaHttpServer/system/IO-TCP/selectors/$a/0] - New connection accepted
[2021-01-07 14:17:43,499] [ERROR] [akka.actor.ActorSystemImpl] [HelloAkkaHttpServer-akka.actor.default-dispatcher-3] [akka.actor.ActorSystemImpl(HelloAkkaHttpServer)] - Error during processing of request: 'oh no! oh is unknown'. Completing with 500 Internal Server Error response. To change default exception handling behavior, provide a custom ExceptionHandler.
java.lang.Exception: oh no! oh is unknown
	at com.grafana.UserRoutes.$anonfun$userRoutes$6(UserRoutes.scala:28)
	at akka.http.scaladsl.server.Directive$.$anonfun$addByNameNullaryApply$2(Directive.scala:166)
	at akka.http.scaladsl.server.ConjunctionMagnet$$anon$2.$anonfun$apply$3(Directive.scala:234)
	at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$mapRouteResult$2(BasicDirectives.scala:68)
	at akka.http.scaladsl.server.directives.BasicDirectives.$anonfun$textract$2(BasicDirectives.scala:161)
	at akka.http.scaladsl.server.RouteConcatenation$RouteWithConcatenation.$anonfun$$tilde$2(RouteConcatenation.scala:47)
	at akka.http.scaladsl.util.FastFuture$.strictTransform$1(FastFuture.scala:40)
  ...

At first sight these seem be like the others. Let’s look at the log format.

<configuration>
    <appender name="FILE" class="ch.qos.logback.core.FileAppender">
        <file>crasher.log</file>
        <append>true</append>
        <encoder>
            <pattern>&ZeroWidthSpace;[%date{ISO8601}] [%level] [%logger] [%thread] [%X{akkaSource}] - %msg%n</pattern>
        </encoder>
    </appender>

    <appender name="ASYNC" class="ch.qos.logback.classic.AsyncAppender">
        <queueSize>1024</queueSize>
        <neverBlock>true</neverBlock>
        <appender-ref ref="STDOUT" />
    </appender>

    <root level="DEBUG">
        <appender-ref ref="ASYNC"/>
    </root>

</configuration>

There is nothing special for a Logback configuration except for &ZeroWidthSpace; at the beginning of each log line. This is the HTML-code for the Zero-width space character. It makes identifying first lines much simpler and is not visible. Thus it will not change the view of the log. The new first line matching regular expression is then \x{200B}\[. 200B is the Unicode code point for the zero-width space character.

multiline:
  # Identify zero-width space as first line of a multiline block.
  # Note the string should be in single quotes.
  firstline: '^\x{200B}\['

  max_wait_time: 3s

Zero-width space might not suite everyone. Any special character that is unlikely to be part of your regular logs should do just fine.

output

Sat, 11 Apr 2026 00:53:11 +0000

output

The output stage is an action stage that takes data from the extracted map and changes the log line that will be sent to Loki.

Schema

output:
  # Name from extracted data to use for the log entry.
  source: <string>

Example

For the given pipeline:

- json:
    expressions:
      user: user
      message: message
- labels:
    user:
- output:
    source: message

And the given log line:

{"user": "alexis", "message": "hello, world!"}

Then the first stage will extract the following key-value pairs into the extracted map:

user: alexis
message: hello, world!

The second stage will then add user=alexis to the label set for the outgoing log line, and the final output stage will change the log line from the original JSON to hello, world!

pack

Sat, 11 Apr 2026 00:53:11 +0000

pack

The pack stage is a transform stage which lets you embed extracted values and labels into the log line by packing the log line and labels inside a JSON object.

For example, if you wanted to remove the labels container and pod but still wanted to keep their values you could use this stage to create the following output:

{
  "container": "myapp",
  "pod": "pod-3223f",
  "_entry": "original log message"
}

The original message will be stored under the _entry key.

This stage is useful if you have some label or other metadata you would like to keep but it doesn’t make a good label (isn’t useful for querying or is too high cardinality)

The querying capabilities of Loki make it easy to still access this data and filter/aggregate on it at query time.

Pack stage schema

pack:
  # Name from extracted data and/or line labels
  # Labels provided here are automatically removed from the output labels.
  labels:
    - [<string>]

  # If the resulting log line should use any existing timestamp or use time.Now() when the line was processed.
  # To avoid out-of-order issues with Loki, when combining several log streams (separate source files) into one
  # you will want to set a new timestamp on the log line, `ingest_timestamp: true`
  # If you are not combining multiple source files or you know your log lines won't have interlaced timestamps
  # you can set this value to false.
  [ingest_timestamp: <bool> | default = true]

Examples

Removing the container label and embed it into the log line (Kubernetes pods could have multiple containers)

pack:
  labels:
    - container

This would create a log line

{
  "container": "myapp",
  "_entry": "original log message"
}

Loki 2.2 also includes a new unpack parser to work with the pack stage.

For example:

{cluster="us-central1", job="myjob"} | unpack

Will automatically unpack embedded labels and log line and replace the log line with the original log line automatically.

More Examples

Use the packed labels for filtering:

{cluster="us-central1", job="myjob"} | unpack | container="myapp"

You can even use the json parser twice if your original message was json:

{cluster="us-central1", job="myjob"} | unpack | container="myapp" | json | val_from_original_log_json="foo"

Or any other parser

{cluster="us-central1", job="myjob"} | unpack | container="myapp" | logfmt | val_from_original_log_json="foo"

regex

Sat, 11 Apr 2026 00:53:11 +0000

regex

The regex stage is a parsing stage that parses a log line using a regular expression. Named capture groups in the regex support adding data into the extracted map.

Schema

regex:
  # The RE2 regular expression. Each capture group must be named.
  expression: <string>

  # Name from extracted data to parse. If empty, uses the log message.
  [source: <string>]

expression needs to be a Go RE2 regex string. Every capture group (re) will be set into the extracted map, every capture group must be named: (?P<name>re). The name of the capture group will be used as the key in the extracted map.

Because of how YAML treats backslashes in double-quoted strings, note that all backslashes in a regex expression must be escaped when using double quotes. For example, all of these are valid:

expression: \w*
expression: '\w*'
expression: "\\w*"

But these are not:

expression: \\w* (only escape backslashes when using double quotes)
expression: '\\w*' (only escape backslashes when using double quotes)
expression: "\w*" (backslash must be escaped)

If you run Promtail with the --config.expand-env=true flag the configuration will run through envsubst which will replace double backslashes with single backslashes. Because of this when using expand-env=true you need to use double backslashes for each single backslash. For example:

expression: '\w*' must be expression: '\\w*'
expression: "\\w*" must be expression: "\\\\w*"

Example

Without `source`

Given the pipeline:

- regex:
    expression: "^(?s)(?P<time>\\S+?) (?P<stream>stdout|stderr) (?P<flags>\\S+?) (?P<content>.*)$"

And the log line:

2019-01-01T01:00:00.000000001Z stderr P i'm a log message!

The following key-value pairs would be added to the extracted map:

time: 2019-01-01T01:00:00.000000001Z,
stream: stderr,
flags: P,
content: i'm a log message

With `source`

Given the pipeline:

- json:
    expressions:
      time:
- regex:
    expression: "^(?P<year>\\d+)"
    source:     "time"

And the log line:

{"time":"2019-01-01T01:00:00.000000001Z"}

The first stage would add the following key-value pairs into the extracted map:

time: 2019-01-01T01:00:00.000000001Z

While the regex stage would then parse the value for time in the extracted map and append the following key-value pairs back into the extracted map:

year: 2019

replace

Sat, 11 Apr 2026 00:53:11 +0000

replace

The replace stage is a parsing stage that parses a log line using a regular expression and replaces the log line. Named capture groups in the regex support adding data into the extracted map.

Schema

replace:
  # The RE2 regular expression. Each named capture group will be added to extracted.
  # Each capture group and named capture group will be replaced with the value given in `replace`
  expression: <string>

  # Name from extracted data to parse. If empty, uses the log message.
  # The replaced value will be assigned back to soure key
  [source: <string>]

  # Value to which the captured group will be replaced. The captured group or the named captured group will be
  # replaced with this value and the log line will be replaced with new replaced values. An empty value will
  # remove the captured group from the log line.
  [replace: <string>]

expression needs to be a Go RE2 regex string. Every named capture group (?P<name>re) will be set into the extracted map. The name of the capture group will be used as the key in the extracted map.

Because of how YAML treats backslashes in double-quoted strings, note that all backslashes in a regex expression must be escaped when using double quotes. For example, all of these are valid:

expression: \w*
expression: '\w*'
expression: "\\w*"

But these are not:

expression: \\w* (only escape backslashes when using double quotes)
expression: '\\w*' (only escape backslashes when using double quotes)
expression: "\w*" (backslash must be escaped)

Example

Without `source`

Given the pipeline:

- replace:
    expression: "password (\\S+)"
    replace: "****"

And the log line:

2019-01-01T01:00:00.000000001Z stderr P i'm a log message who has sensitive information with password xyz!

The log line becomes

2019-01-01T01:00:00.000000001Z stderr P i'm a log message who has sensitive information with password ****!

With `source`

Given the pipeline:

- json:
    expressions:
     level:
     msg:
- replace:
    expression: "\\S+ - \"POST (\\S+) .*"
    source:     "msg"
    replace: "/loki/api/v1/push"

And the log line:

{"time":"2019-01-01T01:00:00.000000001Z", "level": "info", "msg":"11.11.11.11 - "POST /loki/api/push/ HTTP/1.1" 200 932 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 GTB6"}

The first stage would add the following key-value pairs into the extracted map:

time: 2019-01-01T01:00:00.000000001Z
level: info
msg: 11.11.11.11 - "POST /loki/api/push/ HTTP/1.1" 200 932 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 GTB6"

While the replace stage would then parse the value for msg in the extracted map and replaces the msg value. msg in extracted will now become

msg: 11.11.11.11 - "POST /loki/api/v1/push/ HTTP/1.1" 200 932 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 GTB6"

With `replace` value in `template` format

Given the pipeline:

- replace:
    expression: "^(\\S+) (\\S+) (\\S+) \\[([\\w:/]+\\s[+\\-]\\d{4})\\] \"(\\S+)\\s?(\\S+)?\\s?(\\S+)?\" (\\d{3}|-) (\\d+|-)\\s?\"?([^\"]*)\"?\\s?\"?([^\"]*)?\"?$"
    replace: '{{ if eq .Value "200" }}{{ Replace .Value "200" "HttpStatusOk" -1 }}{{ else }}{{ .Value | ToUpper }}{{ end }}'

And the log line:

11.11.11.11 - frank [25/Jan/2000:14:00:01 -0500] "GET /1986.js HTTP/1.1" 200 932 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 GTB6"

The replace stage parses the log line and if the captured group has a value 200 it replaces the value to HttpStatusOk.

The log line would become

11.11.11.11 - frank [25/Jan/2000:14:00:01 -0500] "GET /1986.js HTTP/1.1" HttpStatusOk 932 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 GTB6"

With `replace` value in `template` format with hashing for obfuscating data

To obfuscate sensitive data, you can combine the replace stage with the Hash template method.

- replace:
    # SSN
    expression: '([0-9]{3}-[0-9]{2}-[0-9]{4})'
    replace: '*SSN*{{ .Value | Hash "salt" }}*'
- replace:
    # IP4
    expression: '(\d{1,3}[.]\d{1,3}[.]\d{1,3}[.]\d{1,3})'
    replace: '*IP4*{{ .Value | Hash "salt" }}*'    
- replace:
    # email
    expression: '([\w\.=-]+@[\w\.-]+\.[\w]{2,64})'
    replace: '*email*{{ .Value | Hash "salt" }}*'  
- replace:
    # creditcard
    expression: '((?:\d[ -]*?){13,16})'
    replace: '*creditcard*{{ .Value | Hash "salt" }}*'

`replace` with named captured group

Given the pipeline:

- replace:
    expression: "^(?P<ip>\\S+) (?P<identd>\\S+) (?P<user>\\S+) \\[(?P<timestamp>[\\w:/]+\\s[+\\-]\\d{4})\\] \"(?P<action>\\S+)\\s?(?P<path>\\S+)?\\s?(?P<protocol>\\S+)?\" (?P<status>\\d{3}|-) (?P<size>\\d+|-)\\s?\"?(?P<referer>[^\"]*)\"?\\s?\"?(?P<useragent>[^\"]*)?\"?$"
    replace: '{{ .Value | ToUpper }}'

And the log line:

11.11.11.11 - frank [25/Jan/2000:14:00:01 -0500] "GET /1986.js HTTP/1.1" 200 932 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 GTB6"

The replace stage parses the log line and replaces the value of all named captured groups to upper case. The named captured groups will be extracted to

ip: 11.11.11.11
identd: -
user: FRANK
timestamp: 25/JAN/2000:14:00:01 -0500
action: GET
path: /1986.JS
protocol: HTTP/1.1
status: 200
size: 932
referer: -
useragent: MOZILLA/5.0 (WINDOWS; U; WINDOWS NT 5.1; DE; RV:1.9.1.7) GECKO/20091221 FIREFOX/3.5.7 GTB6"

The log line would become

11.11.11.11 - FRANK [25/JAN/2000:14:00:01 -0500] "GET /1986.JS HTTP/1.1" 200 932 "-" "MOZILLA/5.0 (WINDOWS; U; WINDOWS NT 5.1; DE; RV:1.9.1.7) GECKO/20091221 FIREFOX/3.5.7 GTB6"

`replace` with both named captured group and only captured group

Given the pipeline:

- replace:
    expression: "^(?P<ip>\\S+) (?P<identd>\\S+) (\\S+) \\[(?P<timestamp>[\\w:/]+\\s[+\\-]\\d{4})\\] \"(?P<action>\\S+)\\s?(?P<path>\\S+)?\\s?(?P<protocol>\\S+)?\" (?P<status>\\d{3}|-) (?P<size>\\d+|-)\\s?\"?(?P<referer>[^\"]*)\"?\\s?\"?(?P<useragent>[^\"]*)?\"?$"
    replace: '{{ .Value | ToUpper }}'

And the log line:

11.11.11.11 - frank [25/Jan/2000:14:00:01 -0500] "GET /1986.js HTTP/1.1" 200 932 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 GTB6"

The replace stage parses the log line and replaces the value of all named captured groups to upper case. The named captured groups will be extracted to. Observe here that user is not extracted since it was just (\\S+) and not a named captured group like this (?P<user>\\S+)

ip: 11.11.11.11
identd: -
timestamp: 25/JAN/2000:14:00:01 -0500
action: GET
path: /1986.JS
protocol: HTTP/1.1
status: 200
size: 932
referer: -
useragent: MOZILLA/5.0 (WINDOWS; U; WINDOWS NT 5.1; DE; RV:1.9.1.7) GECKO/20091221 FIREFOX/3.5.7 GTB6"

The log line would become

11.11.11.11 - FRANK [25/JAN/2000:14:00:01 -0500] "GET /1986.JS HTTP/1.1" 200 932 "-" "MOZILLA/5.0 (WINDOWS; U; WINDOWS NT 5.1; DE; RV:1.9.1.7) GECKO/20091221 FIREFOX/3.5.7 GTB6"

With empty `replace`

Given the pipeline:

- replace:
    expression: "11.11.11.11 - (\\S+\\s)"
    replace: ""

And the log line:

11.11.11.11 - frank [25/Jan/2000:14:00:01 -0500] "GET /1986.js HTTP/1.1" 200 932 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 GTB6"

The log line becomes

11.11.11.11 - [25/Jan/2000:14:00:01 -0500] "GET /1986.js HTTP/1.1" 200 932 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; de; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 GTB6"

sampling

Sat, 11 Apr 2026 00:53:11 +0000

sampling

The sampling stage is a stage that sampling logs.

Sampling stage schema

The sampling stage is used to sampling the logs. Configuring the value rate: 0.1 means that 10% of the logs will be pushed to the Loki server.

sampling:
  # The rate sampling in lines per second that Promtail will push to Loki.The value is between 0 and 1, where a value of 0 means no logs are sampled and a value of 1 means 100% of logs are sampled.
  [rate: <int>]

Examples

The following are examples showing the use of the sampling stage.

sampling

Simple sampling

Given the pipeline:

- sampling:
    rate: 0.1

Match a line and sampling

Given the pipeline:

pipeline_stages:
- json:
    expressions:
      app:
- match:
    pipeline_name: "app2"
    selector: '{app="poki"}'
    stages:
    - sampling:
        rate: 0.1

Complex sampling stage configurations.

static_labels

Sat, 11 Apr 2026 00:53:11 +0000

static_labels

The static_labels stage is an action stage that adds static-labels to the label set that is sent to Loki with the log entry.

Schema

static_labels:
  [ <string>: [<string>] ... ]

Examples

For the given pipeline:

- json:
    expressions:
      stream: stream
- labels:
    stream:
- static_labels:
    custom_key: custom_val

Given the following log line:

{"log":"log message\n","stream":"stderr","time":"2019-04-30T02:12:41.8443515Z"}

The first stage would extract stream into the extracted map with a value of stderr. The labels stage would turn that key-value pair into a label. The static_labels stage would add the provided static labels into the label set.

The resulting entry that is sent to Loki will contain stream="stderr" and custom_key="custom_val" as labels.

structured_metadata

Sat, 11 Apr 2026 00:53:11 +0000

structured_metadata

The structured_metadata stage is an action stage that takes data from the extracted map and modifies the structured metadata that is sent to Loki with the log entry.

Schema

structured_metadata:
  # Key is REQUIRED and the name for the label of structured metadata that will be created.
  # Value is optional and will be the name from extracted data whose value
  # will be used for the value of the label. If empty, the value will be
  # inferred to be the same as the key.
  [ <string>: [<string>] ... ]

Examples

Parse from log entry

For the given pipeline:

- json:
    expressions:
      stream: stream
      traceID: traceID
- labels:
    stream:
- structured_metadata:
    traceID:

Given the following log line:

{"log":"log message\n","stream":"stderr","traceID":"0242ac120002","time":"2019-04-30T02:12:41.8443515Z"}

The first stage would extract stream with a value of stderr and traceID with a value of 0242ac120002 into the extracted data set. The labels stage would turn that stream and stderr key-value pair into a stream label. The structured_metadata stage would attach the traceID and 0242ac120002 key-value pair as a structured metadata to the log line.

Parse from service discovery labels

For the configuration below, you can use labels from relabel_configs to use as structured_metadata:

pipeline_stages:
  - structured_metadata:
      pod_uid:
      pod_host_ip:
relabel_configs:
  - action: replace
    source_labels:
      - __meta_kubernetes_pod_uid
    target_label: pod_uid
  - action: replace
    source_labels:
      - __meta_kubernetes_pod_host_ip
    target_label: pod_host_ip

Given the following discovered labels below with a log line sample log:

- Discovered label	- Value	- Target label
__meta_kubernetes_pod_host_ip	127.0.0.1	pod_host_ip
__meta_kubernetes_pod_uid	b3937321-fe90-4e15-ac94-495c8fdb9202	pod_uid

The structured_metadata stage would turn the discovered labels pod_uid and pod_host_ip into key-value pairs as structured metadata for the log line sample log and exclude them from creating high-cardinality streams.

template

Sat, 11 Apr 2026 00:53:11 +0000

template

The template stage is a transform stage that lets use manipulate the values in the extracted map using Go’s template syntax.

The template stage is primarily useful for manipulating data from other stages before setting them as labels, such as to replace spaces with underscores or converting an uppercase string into a lowercase one. template can also be used to construct messages with multiple keys.

The template stage can also create new keys in the extracted map.

Schema

template:
  # Name from extracted data to parse. If key in extract data doesn't exist, an
  # entry for it will be created.
  source: <string>

  # Go template string to use. In additional to normal template
  # functions, ToLower, ToUpper, Replace, Trim, TrimLeft, TrimRight,
  # TrimPrefix, TrimSuffix, and TrimSpace are available as functions.
  template: <string>

Examples

- template:
    source: new_key
    template: 'hello world!'

Assuming no data has been added to the extracted map yet, this stage will first add new_key with a blank value into the extracted map. Then its value will be set to hello world!.

- template:
    source: app
    template: '{{ .Value }}_some_suffix'

This pipeline takes the value of the app key in the existing extracted map and appends _some_suffix to its value. For example, if the extracted map had a key of app and a value of loki, this stage would modify the value from loki to loki_some_suffix.

- template:
    source: app
    template: '{{ ToLower .Value }}'

This pipeline takes the current value of app from the extracted map and converts its value to be all lowercase. For example, if the extracted map contained app with a value of LOKI, this pipeline would change its value to loki.

- template:
    source: output_msg
    template: '{{ .level }} for app {{ ToUpper .app }}'

This pipeline takes the current value of level and app from the extracted map and a new key output_msg will be added to extracted map with evaluated template.

For example, if the extracted map contained app with a value of loki, this pipeline would change its value to LOKI. Assuming value of level is warn. A new key output_msg will be added to extracted map with value warn for app LOKI.

Any previously extracted keys can be used in template. All extracted keys are available for template to expand.

- template:
    source: app
    template: '{{ .level }} for app {{ ToUpper .Value }} in module {{.module}}'

This pipeline takes the current value of level, app and module from the extracted map and converts value of app to the evaluated template.

For example, if the extracted map contained app with a value of loki, this pipeline would change its value to LOKI. Assuming value of level is warn and value of module is test. Pipeline will change the value of app to warn for app LOKI in module test.

Any previously extracted keys can be used in template. All extracted keys are available for template to expand. Also, if source is available it can be referred as .Value in template. Here, app is provided as source. So, it can be referred as .Value in template.

- template:
    source: app
    template: '{{ Replace .Value "loki" "blokey" 1 }}'

The template here uses Go’s string.Replace function. When the template executes, the entire contents of the app key from the extracted map will have at most 1 instance of loki changed to blokey.

A special key named Entry can be used to reference the current line, this can be useful when you need to append/prepend the log line.

- template:
    source: message
    template: '{{.app }}: {{ .Entry }}'
- output:
    source: message

The snippet above will for instance prepend the log line with the application name.

- template:
    source: time
    template: "\
      {{ .date_local | substr 6 10 }}-{{ .date_local | substr 0 2 }}-{{ .date_local | substr 3 5 }}T\
      {{ if eq (.time_local | substr 0 2) \"12\" }}\
          {{ if eq .hour_period \"AM\" }}00{{ else }}12{{ end }}{{ .time_local | substr 2 10}}Z\
      {{ else }}\
          {{ if eq .hour_period \"AM\" }}\
              {{ if or (eq (.time_local | substr 0 2) \"11\") (eq (.time_local | substr 0 2) \"10\") }}{{ .time_local }}Z\
                  {{ else }}0{{ .time_local }}Z\
              {{ end }}\
          {{ else }}\
              {{ if eq (.time_local | substr 0 2) \"11\" }}23{{ .time_local | substr 2 10 }}Z{{ end }}\
              {{ if eq (.time_local | substr 0 2) \"10\" }}22{{ .time_local | substr 2 10 }}Z{{ end }}\
              {{ if eq (.time_local | substr 0 2) \"9:\" }}21{{ .time_local | substr 1 10 }}Z{{ end }}\
              {{ if eq (.time_local | substr 0 2) \"8:\" }}20{{ .time_local | substr 1 10 }}Z{{ end }}\
              {{ if and (le (.time_local | substr 0 1) \"7\") (eq (.time_local | substr 1 2) \":\") }}1{{ add (.time_local | substr 0 1) 2 }}{{ .time_local | substr 1 10 }}Z{{ end }}\
          {{ end }}\
      {{ end }}"
  - timestamp:
      source: time
      format: RFC3339

The snippet above is an example of a multiline template without spaces. The extracted data from logs using regex will be used to convert 11/08/2022, 12:53:24 PM to RFC3339 time format. It also makes use of functions like eq, substr and shows how we can use if with and, or in go templates.

Supported Functions

ToLower and ToUpper

ToLower and ToUpper convert the entire string respectively to lowercase and uppercase.

Examples:

- template:
    source: out
    template: '{{ ToLower .app }}'

- template:
    source: out
    template: '{{ .app | ToUpper }}'

Replace

Replace returns a copy of the string s with the first n non-overlapping instances of old replaced by new. If old is empty, it matches at the beginning of the string and after each UTF-8 sequence, yielding up to k+1 replacements for a k-rune string. If n < 0, there is no limit on the number of replacements.

The example below will replace the first two words loki by Loki.

- template:
    source: output
    template: '{{ Replace .Value "loki" "Loki" 2 }}'

Trim

Trim returns a slice of the string s with all leading and trailing Unicode code points contained in cutset removed.

TrimLeft and TrimRight are the same as Trim except that it respectively trim only leading and trailing characters.

- template:
    source: output
    template: '{{ Trim .Value ",. " }}'

TrimSpace TrimSpace returns a slice of the string s, with all leading and trailing white space removed, as defined by Unicode.

- template:
    source: output
    template: '{{ TrimSpace .Value }}'

TrimPrefix and TrimSuffix will trim respectively the prefix or suffix supplied.

- template:
    source: output
    template: '{{ TrimPrefix .Value "--" }}'

Regex

regexReplaceAll returns a copy of the input string, replacing matches of the Regexp with the replacement string replacement. Inside string replacement, $ signs are interpreted as in Expand, so for instance $1 represents the text of the first submatch

- template:
    source: output
    template: '{{ regexReplaceAll "(a*)bc" .Value "${1}a" }}'

regexReplaceAllLiteral returns a copy of the input string, replacing matches of the Regexp with the replacement string replacement The replacement string is substituted directly, without using Expand.

- template:
    source: output
    template: '{{ regexReplaceAllLiteral "(ts=)" .Value "timestamp=" }}'

Hash and Sha2Hash

Hash returns a Sha3_256 hash of the string, represented as a hexadecimal number of 64 digits. You can use it to obfuscate sensitive data / PII in the logs. It requires a (fixed) salt value, to add complexity to low input domains (e.g. all possible Social Security Numbers).

- template:
    source: output
    template: '{{ Hash .Value "salt" }}'

Alternatively, you can use Sha2Hash for calculating the Sha2_256 of the string. Sha2_256 is faster and requires less CPU than Sha3_256, however it is less secure.

We recommend using Hash as it has a stronger hashing algorithm which we plan to keep strong over time without requiring client config changes.

- template:
    source: output
    template: '{{ Sha2Hash .Value "salt" }}'

tenant

Sat, 11 Apr 2026 00:53:11 +0000

tenant

The tenant stage is an action stage that sets the tenant ID for the log entry picking it from a field in the extracted data map. If the field is missing, the default promtail client tenant_id will be used.

Schema

tenant:
  # Either label, source or value config option is required, but not all (they
  # are mutually exclusive).

  # Name from labels to whose value should be set as tenant ID.
  [ label: <string> ]

  # Name from extracted data to whose value should be set as tenant ID.
  [ source: <string> ]

  # Value to use to set the tenant ID when this stage is executed. Useful
  # when this stage is included within a conditional pipeline with "match".
  [ value: <string> ]

Example: extract the tenant ID from a structured log

For the given pipeline:

pipeline_stages:
  - json:
      expressions:
        customer_id: customer_id
  - tenant:
      source: customer_id

Given the following log line:

{"customer_id":"1","log":"log message\n","stream":"stderr","time":"2019-04-30T02:12:41.8443515Z"}

The first stage would extract customer_id into the extracted map with a value of 1. The tenant stage would set the X-Scope-OrgID request header (used by Loki to identify the tenant) to the value of the customer_id extracted data, which is 1.

Example: override the tenant ID with the configured value

For the given pipeline:

pipeline_stages:
  - json:
      expressions:
        app:
        message:
  - labels:
      app:
  - match:
      selector: '{app="api"}'
      stages:
        - tenant:
            value: "team-api"
  - output:
      source: message

Given the following log line:

{"app":"api","log":"log message\n","stream":"stderr","time":"2019-04-30T02:12:41.8443515Z"}

The pipeline would:

Decode the JSON log
Set the label app="api"
Process the match stage checking if the {app="api"} selector matches and - whenever it matches - run the sub stages. The tenant sub stage would override the tenant with the value "team-api".

Example: extract the tenant ID from kubernetes sd

scrape_configs:
  - job_name: kubernetes-pods-name

    kubernetes_sd_configs:
      - role: pod

    relabel_configs:
      - action: replace
        source_labels:
          - __meta_kubernetes_namespace
        target_label: namespace

    pipeline_stages:
    - match:
        selector: '{namespace=~".+"}'
        stages:
          - tenant:
              label: "namespace"
    - output:
         source: message

The pipeline would:

Match any log where the namespace label matched the regexp .+
Process the match stage checking if the {namespace=~".+"} selector matches and - whenever it matches - run the sub stages. The tenant sub stage would override the tenant with the value with the value of the namespace label, if it was set.

timestamp

Sat, 11 Apr 2026 00:53:11 +0000

timestamp

The timestamp stage is an action stage that can change the timestamp of a log line before it is sent to Loki. When a timestamp stage is not present, the timestamp of a log line defaults to the time when the log entry is scraped.

Schema

timestamp:
  # Name from extracted data to use for the timestamp.
  source: <string>

  # Determines how to parse the time string. Can use
  # pre-defined formats by name: [ANSIC UnixDate RubyDate RFC822
  # RFC822Z RFC850 RFC1123 RFC1123Z RFC3339 RFC3339Nano Unix
  # UnixMs UnixUs UnixNs].
  format: <string>

  # Fallback formats to try if the format fails to parse the value
  # Can use pre-defined formats by name: [ANSIC UnixDate RubyDate RFC822
  # RFC822Z RFC850 RFC1123 RFC1123Z RFC3339 RFC3339Nano Unix
  # UnixMs UnixUs UnixNs].
  [fallback_formats: []<string>]

  # IANA Timezone Database string.
  [location: <string>]

  # Which action should be taken in case the timestamp can't
  # be extracted or parsed. Valid values are: [skip, fudge].
  # Defaults to "fudge".
  [action_on_failure: <string>]

Reference Time

The format field can be how the reference time, defined to be Mon Jan 2 15:04:05 -0700 MST 2006, would be interpreted in the format or can alternatively be one of the following common forms:

ANSIC: Mon Jan _2 15:04:05 2006
UnixDate: Mon Jan _2 15:04:05 MST 2006
RubyDate: Mon Jan 02 15:04:05 -0700 2006
RFC822: 02 Jan 06 15:04 MST
RFC822Z: 02 Jan 06 15:04 -0700
RFC850: Monday, 02-Jan-06 15:04:05 MST
RFC1123: Mon, 02 Jan 2006 15:04:05 MST
RFC1123Z: Mon, 02 Jan 2006 15:04:05 -0700
RFC3339: 2006-01-02T15:04:05-07:00
RFC3339Nano: 2006-01-02T15:04:05.999999999-07:00

Additionally, support for common Unix timestamps is supported with the following format values:

Unix: 1562708916 or with fractions 1562708916.000000123
UnixMs: 1562708916414
UnixUs: 1562708916414123
UnixNs: 1562708916000000123

Custom formats are passed directly to the layout parameter in Go’s time.Parse function. If the custom format has no year component specified, Promtail will assume that the current year according to the system’s clock should be used.

The syntax used by the custom format defines the reference date and time using specific values for each component of the timestamp (i.e., Mon Jan 2 15:04:05 -0700 MST 2006). The following table shows supported reference values which should be used in the custom format.

Timestamp component	Format value
Year	`06`, `2006`
Month	`1`, `01`, `Jan`, `January`
Day	`2`, `02`, `_2` (two digits right justified)
Day of the week	`Mon`, `Monday`
Hour	`3` (12-hour), `03` (12-hour zero prefixed), `15` (24-hour)
Minute	`4`, `04`
Second	`5`, `05`
Fraction of second	`.000` (ms zero prefixed), `.000000` (μs), `.000000000` (ns), `.999` (ms without trailing zeroes), `.999999` (μs), `.999999999` (ns)
12-hour period	`pm`, `PM`
Timezone name	`MST`
Timezone offset	`-0700`, `-070000` (with seconds), `-07`, `07:00`, `-07:00:00` (with seconds)
Timezone ISO-8601	`Z0700` (Z for UTC or time offset), `Z070000`, `Z07`, `Z07:00`, `Z07:00:00`

In order to correctly format the time, for the timestamp 2006/01/02 03:04:05.000:

If you want a 24-hour format you should be using 15:04:0.000.
If you want a 12-hour format you should be using either 3:04:05.000 PM or 03:04:05.000 PM.

Action on Failure

The action_on_failure setting defines which action should be taken by the stage in case the source field doesn’t exist in the extracted data or the timestamp parsing fails. The supported actions are:

fudge (default): change the timestamp to the last known timestamp, summing up 1 nanosecond (to guarantee log entries ordering)
skip: do not change the timestamp and keep the time when the log entry has been scraped by Promtail

Examples

- timestamp:
    source: time
    format: RFC3339Nano

This stage looks for a time field in the extracted map and reads its value in RFC3339Nano form (e.g., 2006-01-02T15:04:05.999999999-07:00). The resulting time value will be used as the timestamp sent with the log line to Loki.