Send log data to Grafana Enterprise Logs on Grafana Labs

Configure Promtail for Grafana Enterprise Logs

Sat, 11 Apr 2026 00:53:11 +0000

Configure Promtail for Grafana Enterprise Logs

Promtail is an agent that ships the contents of local logs to Grafana Enterprise Logs (GEL).

Verify that Promtail is running

Because there are several ways to Install Promtail, there are also several ways to verify that it is running. If you need help at all, submit a Support Request.

For the following common installation methods, check that Promtail is running:

binary: To check that the executable is running, use ps -ef | grep promtail.
Docker: To check if a Promtail container is running, run docker ps.
Kubernetes: To check that Promtail pods are running, use the kubectl get pods | grep promtail command when you are in the correct namespace.

Configure Promtail to write to GEL

To configure Promtail to send logs to the demo tenant, add a clients block to its configuration.

Example of a minimal configuration:

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://enterprise-logs:3100/loki/api/v1/push
    basic_auth:
      username: demo
      password: <token>

scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/*log

For more information, refer to Configuring Promtail.

Run this Promtail configuration using Docker

To run this configuration, store the preceding configuration locally in the file promtail.yaml, and run the following code:

$ docker run \
   -d \
   --name=demo-promtail \
   --network=enterprise-logs \
   -v $(pwd)/promtail.yaml:/etc/promtail/config.yml:z \
   grafana/promtail:2.1.0

Verify that Promtail is writing to GEL

From Grafana Explore, you can query the data from your GEL cluster via the Enterprise Logs data source.

Ingesting logs to Loki using Alloy

Sat, 11 Apr 2026 00:53:11 +0000

Ingesting logs to Loki using Alloy

Grafana Alloy is a versatile observability collector that can ingest logs in various formats and send them to Loki. We recommend Alloy as the primary method for sending logs to Loki, as it provides a more robust and feature-rich solution for building a highly scalable and reliable observability pipeline.

Installing Alloy

To get started with Grafana Alloy and send logs to Loki, you need to install and configure Alloy. You can follow the Alloy documentation to install Alloy on your preferred platform.

Components of Alloy for logs

Alloy pipelines are built using components that perform specific functions. For logs these can be broken down into three categories:

Collector: These components collect/receive logs from various sources. This can be scraping logs from a file, receiving logs over HTTP, gRPC or ingesting logs from a message queue.
Transformer: These components can be used to manipulate logs before they are sent to a writer. This can be used to add additional metadata, filter logs, or batch logs before sending them to a writer.
Writer: These components send logs to the desired destination. Our documentation will focus on sending logs to Loki, but Alloy supports sending logs to various destinations.

Log components in Alloy

Here is a non-exhaustive list of components that can be used to build a log pipeline in Alloy. For a complete list of components, refer to the components list.

Type	Component
Collector	loki.source.api
Collector	loki.source.awsfirehose
Collector	loki.source.azure_event_hubs
Collector	loki.source.cloudflare
Collector	loki.source.docker
Collector	loki.source.file
Collector	loki.source.gcplog
Collector	loki.source.gelf
Collector	loki.source.heroku
Collector	loki.source.journal
Collector	loki.source.kafka
Collector	loki.source.kubernetes
Collector	loki.source.kubernetes_events
Collector	loki.source.podlogs
Collector	loki.source.syslog
Collector	loki.source.windowsevent
Collector	otelcol.receiver.loki
Transformer	loki.relabel
Transformer	loki.process
Writer	loki.write
Writer	otelcol.exporter.loki

Learning path

Start your learning experience with Grafana Learning Paths

Grafana Learning Paths provide a clear, structured path that leads you from beginner concepts to advanced use cases. Learn about this Grafana feature on Send logs to Grafana Cloud using Alloy.

Start learning

Interactive Tutorials

To learn more about how to configure Alloy to send logs to Loki within different scenarios, follow these interactive tutorials:

Install Promtail

Sat, 11 Apr 2026 00:53:11 +0000

Install Promtail

Promtail is distributed as a binary, in a Docker container, or there is a Helm chart to install it in a Kubernetes cluster.

Install the binary

Every Grafana Loki release includes binaries for Promtail which can be found on the Releases page as part of the release assets.

Install using APT or RPM package manager

See the instructions here.

Install using Docker

Make sure to modify the tag to the most recent version.
Bash
```
docker pull grafana/promtail:3.6.0
```
Create your Promtail configuration file in a file called promtail-config.yaml. Refer to the Promtail configuration reference for more details.
Note that you will need to replace <local-path> in the commands with your local path and may need to update the release version to match the current release.
Bash
```
docker run -v <local-path>:/mnt/config -v /var/log:/var/log --link loki grafana/promtail:3.6.0 --config.file=/mnt/config/promtail-config.yaml
```

Install on MacOS with Homebrew

If necessary, install Homebrew.

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

Update Homebrew.
Bash
```
brew update
```
Navigate to the homebrew page for Promtail.
To install Promtail, run the following command:
Bash
```
brew install promtail
```
Check that installation was successful.
1. Check that promtail exists in its install directory.
  Bash
```
which promtail
```
2. Run promtail
  Bash
```
promtail
```

Install using Helm

Make sure that Helm is installed. See Installing Helm.
Then you can add Grafana’s chart repository to Helm:
Bash
```
helm repo add grafana https://grafana.github.io/helm-charts
```
Update the chart repository:
Bash
```
helm repo update
```
Create the configuration file values.yaml. The example below illustrates a connection to the locally deployed loki server:
YAML
```
config:
# publish data to loki
  clients:
    - url: http://loki-gateway/loki/api/v1/push
      tenant_id: 1
```

Finally, Promtail can be deployed with:

# The default helm configuration deploys promtail as a daemonSet (recommended)
helm upgrade --values values.yaml --install promtail grafana/promtail

Install as Kubernetes daemonSet (recommended)

A DaemonSet will deploy Promtail on every node within a Kubernetes cluster.

The DaemonSet deployment works well at collecting the logs of all containers within a cluster. It’s the best solution for a single-tenant model. Replace {YOUR_LOKI_ENDPOINT} with your Loki endpoint.

--- # Daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: promtail-daemonset
spec:
  selector:
    matchLabels:
      name: promtail
  template:
    metadata:
      labels:
        name: promtail
    spec:
      serviceAccount: promtail-serviceaccount
      containers:
      - name: promtail-container
        image: grafana/promtail
        args:
        - -config.file=/etc/promtail/promtail.yaml
        env: 
        - name: 'HOSTNAME' # needed when using kubernetes_sd_configs
          valueFrom:
            fieldRef:
              fieldPath: 'spec.nodeName'
        volumeMounts:
        - name: logs
          mountPath: /var/log
        - name: promtail-config
          mountPath: /etc/promtail
        - mountPath: /var/lib/docker/containers
          name: varlibdockercontainers
          readOnly: true
      volumes:
      - name: logs
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: promtail-config
        configMap:
          name: promtail-config
--- # configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: promtail-config
data:
  promtail.yaml: |
    server:
      http_listen_port: 9080
      grpc_listen_port: 0

    clients:
    - url: https://<YOUR_LOKI_ENDPOINT>/loki/api/v1/push

    positions:
      filename: /tmp/positions.yaml
    target_config:
      sync_period: 10s
    scrape_configs:
    - job_name: pod-logs
      kubernetes_sd_configs:
        - role: pod
      pipeline_stages:
        - docker: {}
      relabel_configs:
        - source_labels:
            - __meta_kubernetes_pod_node_name
          target_label: __host__
        - action: labelmap
          regex: __meta_kubernetes_pod_label_(.+)
        - action: replace
          replacement: $1
          separator: /
          source_labels:
            - __meta_kubernetes_namespace
            - __meta_kubernetes_pod_name
          target_label: job
        - action: replace
          source_labels:
            - __meta_kubernetes_namespace
          target_label: namespace
        - action: replace
          source_labels:
            - __meta_kubernetes_pod_name
          target_label: pod
        - action: replace
          source_labels:
            - __meta_kubernetes_pod_container_name
          target_label: container
        - replacement: /var/log/pods/*$1/*.log
          separator: /
          source_labels:
            - __meta_kubernetes_pod_uid
            - __meta_kubernetes_pod_container_name
          target_label: __path__

--- # Clusterrole.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: promtail-clusterrole
rules:
  - apiGroups: [""]
    resources:
    - nodes
    - services
    - pods
    verbs:
    - get
    - watch
    - list

--- # ServiceAccount.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: promtail-serviceaccount

--- # Rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: promtail-clusterrolebinding
subjects:
    - kind: ServiceAccount
      name: promtail-serviceaccount
      namespace: default
roleRef:
    kind: ClusterRole
    name: promtail-clusterrole
    apiGroup: rbac.authorization.k8s.io

Configure Promtail

Sat, 11 Apr 2026 00:53:11 +0000

Configure Promtail

Promtail is configured in a YAML file (usually referred to as config.yaml) which contains information on the Promtail server, where positions are stored, and how to scrape logs from files.

Printing Promtail Config At Runtime

If you pass Promtail the flag -print-config-stderr or -log-config-reverse-order, (or -print-config-stderr=true) Promtail will dump the entire config object it has created from the built in defaults combined first with overrides from config file, and second by overrides from flags.

The result is the value for every config object in the Promtail config struct.

Some values may not be relevant to your install, this is expected as every option has a default value if it is being used or not.

This config is what Promtail will use to run, it can be invaluable for debugging issues related to configuration and is especially useful in making sure your config files and flags are being read and loaded properly.

-print-config-stderr is nice when running Promtail directly e.g. ./promtail as you can get a quick output of the entire Promtail config.

-log-config-reverse-order is the flag we run Promtail with in all our environments, the config entries are reversed so that the order of configs reads correctly top to bottom when viewed in Grafana’s Explore.

Configuration File Reference

To specify which configuration file to load, pass the -config.file flag at the command line. The file is written in YAML format, defined by the schema below. Brackets indicate that a parameter is optional. For non-list parameters the value is set to the specified default.

For more detailed information on configuring how to discover and scrape logs from targets, see Scraping. For more information on transforming logs from scraped targets, see Pipelines.

Reload at runtime

Promtail can reload its configuration at runtime. If the new configuration is not well-formed, the changes will not be applied. A configuration reload is triggered by sending a SIGHUP to the Promtail process or sending a HTTP POST request to the /reload endpoint (when the --server.enable-runtime-reload flag is enabled).

Use environment variables in the configuration

You can use environment variable references in the configuration file to set values that need to be configurable during deployment. To do this, pass -config.expand-env=true and use:

${VAR}

Where VAR is the name of the environment variable.

Each variable reference is replaced at startup by the value of the environment variable. The replacement is case-sensitive and occurs before the YAML file is parsed. References to undefined variables are replaced by empty strings unless you specify a default value or custom error text.

To specify a default value, use:

${VAR:-default_value}

Where default_value is the value to use if the environment variable is undefined.

Generic placeholders

<boolean>: a boolean that can take the values true or false
<int>: any integer matching the regular expression [1-9]+[0-9]*
<duration>: a duration matching the regular expression [0-9]+(ms|[smhdwy])
<labelname>: a string matching the regular expression [a-zA-Z_][a-zA-Z0-9_]*
<labelvalue>: a string of Unicode characters
<filename>: a valid path relative to current working directory or an absolute path.
<host>: a valid string consisting of a hostname or IP followed by an optional port number
<string>: a string
<secret>: a string that represents a secret, such as a password

Supported contents and default values of `config.yaml`:

# Configures global settings which impact all targets.
[global: <global_config>]

# Configures the server for Promtail.
[server: <server_config>]

# Describes how Promtail connects to multiple instances
# of Grafana Loki, sending logs to each.
# WARNING: If one of the remote Loki servers fails to respond or responds
# with any error which is retryable, this will impact sending logs to any
# other configured remote Loki servers.  Sending is done on a single thread!
# It is generally recommended to run multiple Promtail clients in parallel
# if you want to send to multiple remote Loki instances.
clients:
  - [<client_config>]

# Describes how to save read file offsets to disk
[positions: <position_config>]

scrape_configs:
  - [<scrape_config>]

# Configures global limits for this instance of Promtail
[limits_config: <limits_config>]

# Configures how tailed targets will be watched.
[target_config: <target_config>]

# Configures additional promtail configurations.
[options: <options_config>]

# Configures tracing support
[tracing: <tracing_config>]

global

The global block configures global settings which impact all scrape targets:

# Configure how frequently log files from disk get polled for changes.
[file_watch_config: <file_watch_config>]

file_watch_config

The file_watch_config block configures how often to poll log files from disk for changes:

# Minimum frequency to poll for files. Any time file changes are detected, the
# poll frequency gets reset to this duration.
[min_poll_frequency: <duration> | default = "250ms"]

# Maximum frequency to poll for files. Any time no file changes are detected,
# the poll frequency doubles in value up to the maximum duration specified by
# this value.
#
# The default is set to the same as min_poll_frequency.
[max_poll_frequency: <duration> | default = "250ms"]

server

The server block configures Promtail’s behavior as an HTTP server:

# Disable the HTTP and GRPC server.
[disable: <boolean> | default = false]

# Enable the /debug/fgprof and /debug/pprof endpoints for profiling.
[profiling_enabled: <boolean> | default = false]

# HTTP server listen host
[http_listen_address: <string>]

# HTTP server listen port (0 means random port)
[http_listen_port: <int> | default = 80]

# gRPC server listen host
[grpc_listen_address: <string>]

# gRPC server listen port (0 means random port)
[grpc_listen_port: <int> | default = 9095]

# Register instrumentation handlers (/metrics, etc.)
[register_instrumentation: <boolean> | default = true]

# Timeout for graceful shutdowns
[graceful_shutdown_timeout: <duration> | default = 30s]

# Read timeout for HTTP server
[http_server_read_timeout: <duration> | default = 30s]

# Write timeout for HTTP server
[http_server_write_timeout: <duration> | default = 30s]

# Idle timeout for HTTP server
[http_server_idle_timeout: <duration> | default = 120s]

# Max gRPC message size that can be received
[grpc_server_max_recv_msg_size: <int> | default = 4194304]

# Max gRPC message size that can be sent
[grpc_server_max_send_msg_size: <int> | default = 4194304]

# Limit on the number of concurrent streams for gRPC calls (0 = unlimited)
[grpc_server_max_concurrent_streams: <int> | default = 100]

# Log only messages with the given severity or above. Supported values [debug,
# info, warn, error]
[log_level: <string> | default = "info"]

# Base path to server all API routes from (e.g., /v1/).
[http_path_prefix: <string>]

# Target managers check flag for Promtail readiness, if set to false the check is ignored
[health_check_target: <bool> | default = true]

# Enable reload via HTTP request.
[enable_runtime_reload: <bool> | default = false]

clients

The clients block configures how Promtail connects to instances of Loki:

# The URL where Loki is listening, denoted in Loki as http_listen_address and
# http_listen_port. If Loki is running in microservices mode, this is the HTTP
# URL for the Distributor. Path to the push API needs to be included.
# Example: http://example.com:3100/loki/api/v1/push
url: <string>

# Custom HTTP headers to be sent along with each push request.
# Be aware that headers that are set by Promtail itself (e.g. X-Scope-OrgID) can't be overwritten.
headers:
  # Example: CF-Access-Client-Id: xxx
  [ <labelname>: <labelvalue> ... ]

# The tenant ID used by default to push logs to Loki. If omitted or empty
# it assumes Loki is running in single-tenant mode and no X-Scope-OrgID header
# is sent.
[tenant_id: <string>]

# Maximum amount of time to wait before sending a batch, even if that
# batch isn't full.
[batchwait: <duration> | default = 1s]

# Maximum batch size (in bytes) of logs to accumulate before sending
# the batch to Loki.
[batchsize: <int> | default = 1048576]

# If using basic auth, configures the username and password
# sent.
basic_auth:
  # The username to use for basic auth
  [username: <string>]

  # The password to use for basic auth
  [password: <string>]

  # The file containing the password for basic auth
  [password_file: <filename>]

# Optional OAuth 2.0 configuration
# Cannot be used at the same time as basic_auth or authorization
oauth2:
  # Client id and secret for oauth2
  [client_id: <string>]
  [client_secret: <secret>]

  # Read the client secret from a file
  # It is mutually exclusive with `client_secret`
  [client_secret_file: <filename>]

  # Optional scopes for the token request
  scopes:
    [ - <string> ... ]

  # The URL to fetch the token from
  token_url: <string>

  # Optional parameters to append to the token URL
  endpoint_params:
    [ <string>: <string> ... ]

# Bearer token to send to the server.
[bearer_token: <secret>]

# File containing bearer token to send to the server.
[bearer_token_file: <filename>]

# HTTP proxy server to use to connect to the server.
[proxy_url: <string>]

# If connecting to a TLS server, configures how the TLS
# authentication handshake will operate.
tls_config:
  # The CA file to use to verify the server
  [ca_file: <string>]

  # The cert file to send to the server for client auth
  [cert_file: <filename>]

  # The key file to send to the server for client auth
  [key_file: <filename>]

  # Validates that the server name in the server's certificate
  # is this value.
  [server_name: <string>]

  # If true, ignores the server certificate being signed by an
  # unknown CA.
  [insecure_skip_verify: <boolean> | default = false]

# Configures how to retry requests to Loki when a request
# fails.
# Default backoff schedule:
# 0.5s, 1s, 2s, 4s, 8s, 16s, 32s, 64s, 128s, 256s(4.267m)
# For a total time of 511.5s(8.5m) before logs are lost
backoff_config:
  # Initial backoff time between retries
  [min_period: <duration> | default = 500ms]

  # Maximum backoff time between retries
  [max_period: <duration> | default = 5m]

  # Maximum number of retries to do
  [max_retries: <int> | default = 10]

# Disable retries of batches that Loki responds to with a 429 status code (TooManyRequests). This reduces
# impacts on batches from other tenants, which could end up being delayed or dropped due to exponential backoff.
[drop_rate_limited_batches: <boolean> | default = false]

# Static labels to add to all logs being sent to Loki.
# Use map like {"foo": "bar"} to add a label foo with
# value bar.
# These can also be specified from command line:
# -client.external-labels=k1=v1,k2=v2
# (or --client.external-labels depending on your OS)
# labels supplied by the command line are applied
# to all clients configured in the `clients` section.
# NOTE: values defined in the config file will replace values
# defined on the command line for a given client if the
# label keys are the same.
external_labels:
  [ <labelname>: <labelvalue> ... ]

# Maximum time to wait for a server to respond to a request
[timeout: <duration> | default = 10s]

positions

The positions block configures where Promtail will save a file indicating how far it has read into a file. It is needed for when Promtail is restarted to allow it to continue from where it left off.

# Location of positions file
[filename: <string> | default = "/var/log/positions.yaml"]

# How often to update the positions file
[sync_period: <duration> | default = 10s]

# Whether to ignore & later overwrite positions files that are corrupted
[ignore_invalid_yaml: <boolean> | default = false]

scrape_configs

The scrape_configs block configures how Promtail can scrape logs from a series of targets using a specified discovery method. Promtail uses the same Prometheus scrape_configs. This means if you already own a Prometheus instance, the config will be very similar:

# Name to identify this scrape config in the Promtail UI.
job_name: <string>

# Describes how to transform logs from targets.
[pipeline_stages: <pipeline_stages>]

# Defines decompression behavior for the given scrape target.
decompression:
  # Whether decompression should be tried or not.
  [enabled: <boolean> | default = false]

  # Initial delay to wait before starting the decompression.
  # Especially useful in scenarios where compressed files are found before the compression is finished.
  [initial_delay: <duration> | default = 0s]

  # Compression format. Supported formats are: 'gz', 'bz2' and 'z.
  [format: <string> | default = ""]

# Describes how to scrape logs from the journal.
[journal: <journal_config>]

# Describes from which encoding a scraped file should be converted.
[encoding: <iana_encoding_name>]

# Describes how to receive logs from syslog.
[syslog: <syslog_config>]

# Describes how to receive logs via the Loki push API, (e.g. from other Promtails or the Docker Logging Driver)
[loki_push_api: <loki_push_api_config>]

# Describes how to scrape logs from the Windows event logs.
[windows_events: <windows_events_config>]

# Configuration describing how to pull/receive Google Cloud Platform (GCP) logs.
[gcplog: <gcplog_config>]

# Configuration describing how to get Azure Event Hubs messages.
[azure_event_hub: <azure_event_hub_config>]

# Describes how to fetch logs from Kafka via a Consumer group.
[kafka: <kafka_config>]

# Describes how to receive logs from gelf client.
[gelf: <gelf_config>]

# Configuration describing how to pull logs from Cloudflare.
[cloudflare: <cloudflare>]

# Configuration describing how to pull logs from a Heroku LogPlex drain.
[heroku_drain: <heroku_drain>]

# Describes how to relabel targets to determine if they should
# be processed.
relabel_configs:
  - [<relabel_config>]

# Static targets to scrape.
static_configs:
  - [<static_config>]

# Files containing targets to scrape.
file_sd_configs:
  - [<file_sd_configs>]

# Describes how to discover Kubernetes services running on the
# same host.
kubernetes_sd_configs:
  - [<kubernetes_sd_config>]

# Describes how to use the Consul Catalog API to discover services registered with the
# consul cluster.
consul_sd_configs:
  [ - <consul_sd_config> ... ]

# Describes how to use the Consul Agent API to discover services registered with the consul agent
# running on the same host as Promtail.
consulagent_sd_configs:
  [ - <consulagent_sd_config> ... ]

# Describes how to use the Docker daemon API to discover containers running on
# the same host as Promtail.
docker_sd_configs:
  [ - <docker_sd_config> ... ]

pipeline_stages

Pipeline stages are used to transform log entries and their labels. The pipeline is executed after the discovery process finishes. The pipeline_stages object consists of a list of stages which correspond to the items listed below.

In most cases, you extract data from logs with regex or json stages. The extracted data is transformed into a temporary map object. The data can then be used by Promtail e.g. as values for labels or as an output. Additionally any other stage aside from docker and cri can access the extracted data.

- [
    <docker> |
    <cri> |
    <regex> |
    <json> |
    <template> |
    <match> |
    <timestamp> |
    <output> |
    <labels> |
    <metrics> |
    <tenant> |
    <replace>
  ]

docker

The Docker stage parses the contents of logs from Docker containers, and is defined by name with an empty object:

docker: {}

The docker stage will match and parse log lines of this format:

`{"log":"level=info ts=2019-04-30T02:12:41.844179Z caller=filetargetmanager.go:180 msg=\"Adding target\"\n","stream":"stderr","time":"2019-04-30T02:12:41.8443515Z"}`

Automatically extracting the time into the logs timestamp, stream into a label, and log field into the output, this can be very helpful as docker is wrapping your application log in this way and this will unwrap it for further pipeline processing of just the log content.

The Docker stage is just a convenience wrapper for this definition:

- json:
    expressions:
      output: log
      stream: stream
      timestamp: time
- labels:
    stream:
- timestamp:
    source: timestamp
    format: RFC3339Nano
- output:
    source: output

cri

The CRI stage parses the contents of logs from CRI containers, and is defined by name with an empty object:

cri: {}

The CRI stage will match and parse log lines of this format:

2019-01-01T01:00:00.000000001Z stderr P some log message

Automatically extracting the time into the logs timestamp, stream into a label, and the remaining message into the output, this can be very helpful as CRI is wrapping your application log in this way and this will unwrap it for further pipeline processing of just the log content.

The CRI stage is just a convenience wrapper for this definition:

- regex:
    expression: "^(?s)(?P<time>\\S+?) (?P<stream>stdout|stderr) (?P<flags>\\S+?) (?P<content>.*)$"
- labels:
    stream:
- timestamp:
    source: time
    format: RFC3339Nano
- output:
    source: content

regex

The Regex stage takes a regular expression and extracts captured named groups to be used in further stages.

regex:
  # The RE2 regular expression. Each capture group must be named.
  expression: <string>

  # Name from extracted data to parse. If empty, uses the log message.
  [source: <string>]

json

The JSON stage parses a log line as JSON and takes JMESPath expressions to extract data from the JSON to be used in further stages.

json:
  # Set of key/value pairs of JMESPath expressions. The key will be
  # the key in the extracted data while the expression will be the value,
  # evaluated as a JMESPath from the source data.
  expressions:
    [ <string>: <string> ... ]

  # Name from extracted data to parse. If empty, uses the log message.
  [source: <string>]

template

The template stage uses Go’s text/template language to manipulate values.

template:
  # Name from extracted data to parse. If key in extract data doesn't exist, an
  # entry for it will be created.
  source: <string>

  # Go template string to use. In additional to normal template
  # functions, ToLower, ToUpper, Replace, Trim, TrimLeft, TrimRight,
  # TrimPrefix, TrimSuffix, and TrimSpace are available as functions.
  template: <string>

Example:

template:
  source: level
  template: '{{ if eq .Value "WARN" }}{{ Replace .Value "WARN" "OK" -1 }}{{ else }}{{ .Value }}{{ end }}'

match

The match stage conditionally executes a set of stages when a log entry matches a configurable LogQL stream selector.

match:
  # LogQL stream selector.
  selector: <string>

  # Names the pipeline. When defined, creates an additional label in
  # the pipeline_duration_seconds histogram, where the value is
  # concatenated with job_name using an underscore.
  [pipeline_name: <string>]

  # Nested set of pipeline stages only if the selector
  # matches the labels of the log entries:
  stages:
    - [
        <docker> |
        <cri> |
        <regex>
        <json> |
        <template> |
        <match> |
        <timestamp> |
        <output> |
        <labels> |
        <metrics>
      ]

timestamp

The timestamp stage parses data from the extracted map and overrides the final time value of the log that is stored by Loki. If this stage isn’t present, Promtail will associate the timestamp of the log entry with the time that log entry was read.

timestamp:
  # Name from extracted data to use for the timestamp.
  source: <string>

  # Determines how to parse the time string. Can use
  # pre-defined formats by name: [ANSIC UnixDate RubyDate RFC822
  # RFC822Z RFC850 RFC1123 RFC1123Z RFC3339 RFC3339Nano Unix
  # UnixMs UnixUs UnixNs].
  format: <string>

  # IANA Timezone Database string.
  [location: <string>]

output

The output stage takes data from the extracted map and sets the contents of the log entry that will be stored by Loki.

output:
  # Name from extracted data to use for the log entry.
  source: <string>

labels

The labels stage takes data from the extracted map and sets additional labels on the log entry that will be sent to Loki.

labels:
  # Key is REQUIRED and the name for the label that will be created.
  # Value is optional and will be the name from extracted data whose value
  # will be used for the value of the label. If empty, the value will be
  # inferred to be the same as the key.
  [ <string>: [<string>] ... ]

metrics

The metrics stage allows for defining metrics from the extracted data.

Created metrics are not pushed to Loki and are instead exposed via Promtail’s /metrics endpoint. Prometheus should be configured to scrape Promtail to be able to retrieve the metrics configured by this stage. If Promtail’s configuration is reloaded, all metrics will be reset.

# A map where the key is the name of the metric and the value is a specific
# metric type.
metrics:
  [<string>: [ <counter> | <gauge> | <histogram> ] ...]

counter

Defines a counter metric whose value only goes up.

# The metric type. Must be Counter.
type: Counter

# Describes the metric.
[description: <string>]

# Key from the extracted data map to use for the metric,
# defaulting to the metric's name if not present.
[source: <string>]

config:
  # Filters down source data and only changes the metric
  # if the targeted value exactly matches the provided string.
  # If not present, all data will match.
  [value: <string>]

  # Must be either "inc" or "add" (case insensitive). If
  # inc is chosen, the metric value will increase by 1 for each
  # log line received that passed the filter. If add is chosen,
  # the extracted value must be convertible to a positive float
  # and its value will be added to the metric.
  action: <string>

gauge

Defines a gauge metric whose value can go up or down.

# The metric type. Must be Gauge.
type: Gauge

# Describes the metric.
[description: <string>]

# Key from the extracted data map to use for the metric,
# defaulting to the metric's name if not present.
[source: <string>]

config:
  # Filters down source data and only changes the metric
  # if the targeted value exactly matches the provided string.
  # If not present, all data will match.
  [value: <string>]

  # Must be either "set", "inc", "dec"," add", or "sub". If
  # add, set, or sub is chosen, the extracted value must be
  # convertible to a positive float. inc and dec will increment
  # or decrement the metric's value by 1 respectively.
  action: <string>

histogram

Defines a histogram metric whose values are bucketed.

# The metric type. Must be Histogram.
type: Histogram

# Describes the metric.
[description: <string>]

# Key from the extracted data map to use for the metric,
# defaulting to the metric's name if not present.
[source: <string>]

config:
  # Filters down source data and only changes the metric
  # if the targeted value exactly matches the provided string.
  # If not present, all data will match.
  [value: <string>]

  # Must be either "inc" or "add" (case insensitive). If
  # inc is chosen, the metric value will increase by 1 for each
  # log line received that passed the filter. If add is chosen,
  # the extracted value must be convertible to a positive float
  # and its value will be added to the metric.
  action: <string>

  # Holds all the numbers in which to bucket the metric.
  buckets:
    - <int>

tenant

The tenant stage is an action stage that sets the tenant ID for the log entry picking it from a field in the extracted data map.

tenant:
  # Either label, source or value config option is required, but not all (they
  # are mutually exclusive).

  # Name from labels to whose value should be set as tenant ID.
  [ label: <string> ]

  # Name from extracted data to whose value should be set as tenant ID.
  [ source: <string> ]

  # Value to use to set the tenant ID when this stage is executed. Useful
  # when this stage is included within a conditional pipeline with "match".
  [ value: <string> ]

replace

The replace stage is a parsing stage that parses a log line using a regular expression and replaces the log line.

replace:
  # The RE2 regular expression. Each named capture group will be added to extracted.
  # Each capture group and named capture group will be replaced with the value given in
  # `replace`
  expression: <string>

  # Name from extracted data to parse. If empty, uses the log message.
  # The replaced value will be assigned back to soure key
  [source: <string>]

  # Value to which the captured group will be replaced. The captured group or the named
  # captured group will be replaced with this value and the log line will be replaced with
  # new replaced values. An empty value will remove the captured group from the log line.
  [replace: <string>]

journal

The journal block configures reading from the systemd journal from Promtail. Requires a build of Promtail that has journal support enabled. If using the AMD64 Docker image, this is enabled by default. On some systems a permission is needed for the user promtail to access journal logs. For Ubuntu (24.04) you need to add promtail to the group systemd-journal with sudo usermod -a -G systemd-journal promtail.

# When true, log messages from the journal are passed through the
# pipeline as a JSON message with all of the journal entries' original
# fields. When false, the log message is the text content of the MESSAGE
# field from the journal entry.
[json: <boolean> | default = false]

# The oldest relative time from process start that will be read
# and sent to Loki.
[max_age: <duration> | default = 7h]

# Label map to add to every log coming out of the journal
labels:
  [ <labelname>: <labelvalue> ... ]

# Get labels from journal, when it is not empty
relabel_configs:
- source_labels: ['__journal__hostname']
  target_label: host
- source_labels: ['__journal__systemd_unit']
  target_label: systemd_unit
  regex: '(.+)'
- source_labels: ['__journal__systemd_user_unit']
  target_label: systemd_user_unit
  regex: '(.+)'
- source_labels: ['__journal__transport']
  target_label: transport
  regex: '(.+)'
- source_labels: ['__journal_priority_keyword']
  target_label: severity
  regex: '(.+)'

# Path to a directory to read entries from. Defaults to system
# paths (/var/log/journal and /run/log/journal) when empty.
[path: <string>]

Available Labels

Labels are imported from systemd-journal fields. The label name is the field name set to lower case with _journal prefix. See the man page systemd.journal-fields for more information.

For example:

journal field	label
_HOSTNAME	__journal__hostname
_SYSTEMD_UNIT	__journal__systemd_unit
_SYSTEMD_USER_UNIT	__journal__systemd_user_unit
ERRNO	__journal_errno

In addition to __journal_priority (imported from PRIORITY journal field, where the value is an integer from 0 to 7), promtail adds __journal_priority_keyword label where the value is generated using the makeJournalPriority mapping function.

journal priority	keyword
0	emerg
1	alert
2	crit
3	error
4	warning
5	notice
6	info
7	debug

syslog

The syslog block configures a syslog listener allowing users to push logs to Promtail with the syslog protocol. Currently supported both BSD syslog Protocol and IETF Syslog (RFC5424) with and without octet counting.

The recommended deployment is to have a dedicated syslog forwarder like syslog-ng or rsyslog in front of Promtail. The forwarder can take care of the various specifications and transports that exist (UDP, BSD syslog, …).

Octet counting is recommended as the message framing method. In a stream with non-transparent framing, Promtail needs to wait for the next message to catch multi-line messages, therefore delays between messages can occur.

See recommended output configurations for syslog-ng and rsyslog. Both configurations enable IETF Syslog with octet-counting.

You may need to increase the open files limit for the Promtail process if many clients are connected. (ulimit -Sn)

# TCP address to listen on. Has the format of "host:port".
listen_address: <string>

# Configure the receiver to use TLS.
tls_config:
  # Certificate and key files sent by the server (required)
  cert_file: <string>
  key_file: <string>

  # CA certificate used to validate client certificate. Enables client certificate verification when specified.
  [ ca_file: <string> ]

# The idle timeout for tcp syslog connections, default is 120 seconds.
idle_timeout: <duration>

# Whether to convert syslog structured data to labels.
# A structured data entry of [example@99999 test="yes"] would become
# the label "__syslog_message_sd_example_99999_test" with the value "yes".
label_structured_data: <bool>

# Label map to add to every log message.
labels:
  [ <labelname>: <labelvalue> ... ]

# Whether Promtail should pass on the timestamp from the incoming syslog message.
# When false, or if no timestamp is present on the syslog message, Promtail will assign the current timestamp to the log when it was processed.
# Default is false
use_incoming_timestamp: <bool>

# Sets the maximum limit to the length of syslog messages
max_message_length: <int>

# Defines used Sylog format at the target. 
syslog_format:
 [type: <string> | default = "rfc5424"]

# Defines whether the full RFC5424 formatted syslog message should be pushed to Loki
use_rfc5424_message: <bool>

Available Labels

__syslog_connection_ip_address: The remote IP address.
__syslog_connection_hostname: The remote hostname.
__syslog_message_severity: The syslog severity parsed from the message. Symbolic name as per syslog_message.go.
__syslog_message_facility: The syslog facility parsed from the message. Symbolic name as per syslog_message.go and syslog(3).
__syslog_message_hostname: The hostname parsed from the message.
__syslog_message_app_name: The app-name field parsed from the message.
__syslog_message_proc_id: The procid field parsed from the message.
__syslog_message_msg_id: The msgid field parsed from the message.
__syslog_message_sd_<sd_id>[_<iana_enterprise_id>]_<sd_name>: The structured-data field parsed from the message. The data field [custom@99770 example="1"] becomes __syslog_message_sd_custom_99770_example.

loki_push_api

The loki_push_api block configures Promtail to expose a Loki push API server.

Each job configured with a loki_push_api will expose this API and will require a separate port.

Note the server configuration is the same as server.

Promtail also exposes a second endpoint on /promtail/api/v1/raw which expects newline-delimited log lines. This can be used to send NDJSON or plaintext logs.

The readiness of the loki_push_api server can be checked using the endpoint /ready.

# The push server configuration options
[server: <server_config>]

# Label map to add to every log line sent to the push API
labels:
  [ <labelname>: <labelvalue> ... ]

# If Promtail should pass on the timestamp from the incoming log or not.
# When false Promtail will assign the current timestamp to the log when it was processed.
# Does not apply to the plaintext endpoint on `/promtail/api/v1/raw`.
[use_incoming_timestamp: <bool> | default = false]

See Example Push Config

windows_events

The windows_events block configures Promtail to scrape windows event logs and send them to Loki.

To subscribe to a specific events stream you need to provide either an eventlog_name or an xpath_query.

Events are scraped periodically every 3 seconds by default but can be changed using poll_interval.

A bookmark path bookmark_path is mandatory and will be used as a position file where Promtail will keep record of the last event processed. This file persists across Promtail restarts.

You can set use_incoming_timestamp if you want to keep incoming event timestamps. By default Promtail will use the timestamp when the event was read from the event log.

Promtail will serialize JSON windows events, adding channel and computer labels from the event received. You can add additional labels with the labels property.

# LCID (Locale ID) for event rendering
# - 1033 to force English language
# -  0 to use default Windows locale
[locale: <int> | default = 0]

# Name of eventlog, used only if xpath_query is empty
# Example: "Application"
[eventlog_name: <string> | default = ""]

# xpath_query can be in defined short form like "Event/System[EventID=999]"
# or you can form a XML Query. Refer to the Consuming Events article:
# https://docs.microsoft.com/en-us/windows/win32/wes/consuming-events
# XML query is the recommended form, because it is most flexible
# You can create or debug XML Query by creating Custom View in Windows Event Viewer
# and then copying resulting XML here
[xpath_query: <string> | default = "*"]

# Sets the bookmark location on the filesystem.
# The bookmark contains the current position of the target in XML.
# When restarting or rolling out Promtail, the target will continue to scrape events where it left off based on the bookmark position.
# The position is updated after each entry processed.
[bookmark_path: <string> | default = ""]

# PollInterval is the interval at which we're looking if new events are available. By default the target will check every 3seconds.
[poll_interval: <duration> | default = 3s]

# Allows to exclude the xml event data.
[exclude_event_data: <bool> | default = false]

# Allows to exclude the human-friendly event message.
[exclude_event_message: <bool> | default = false]

# Allows to exclude the user data of each windows event.
[exclude_user_data: <bool> | default = false]

# Label map to add to every log line read from the windows event log
labels:
  [ <labelname>: <labelvalue> ... ]

# If Promtail should pass on the timestamp from the incoming log or not.
# When false Promtail will assign the current timestamp to the log when it was processed
[use_incoming_timestamp: <bool> | default = false]

GCP Log

The gcplog block configures how Promtail receives GCP logs. There are two strategies, based on the configuration of subscription_type:

Pull: Using GCP Pub/Sub pull subscriptions. Promtail will consume log messages directly from the configured GCP Pub/Sub topic.
Push: Using GCP Pub/Sub push subscriptions. Promtail will expose an HTTP server, and GCP will deliver logs to that server.

When using the push subscription type, keep in mind:

The server configuration is the same as server, since Promtail exposes an HTTP server for target that requires so.
An endpoint at POST /gcp/api/v1/push, which expects requests from GCP PubSub message delivery system.

# Type of subscription used to fetch logs from GCP. Can be either `pull` (default) or `push`.
[subscription_type: <string> | default = "pull"]

# If the subscription_type is pull,  the GCP project ID
[project_id: <string>]

# If the subscription_type is pull, GCP PubSub subscription from where Promtail will pull logs from
[subscription: <string>]

# If the subscription_type is push, the server configuration options
[server: <server_config>]

# Whether Promtail should pass on the timestamp from the incoming GCP Log message.
# When false, or if no timestamp is present in the GCP Log message, Promtail will assign the current
# timestamp to the log when it was processed.
[use_incoming_timestamp: <boolean> | default = false]

# use_full_line to force Promtail to send the full line from Cloud Logging even if `textPayload` is available.
# By default, if `textPayload` is present in the line, then it's used as log line.
[use_full_line: <boolean> | default = false]

# If the subscription_type is push, configures an HTTP handler timeout. If processing the incoming GCP Logs request takes longer
# than the configured duration, that is processing and then sending the entry down the processing pipeline, the server will abort
# and respond with a 503 HTTP status code.
[push_timeout: <duration>|  default = 0 (no timeout)]

# Label map to add to every log message.
labels:
  [ <labelname>: <labelvalue> ... ]

Available Labels

When Promtail receives GCP logs, various internal labels are made available for relabeling. This depends on the subscription type chosen.

Internal labels available for pull

__gcp_logname
__gcp_severity
__gcp_resource_type
__gcp_resource_labels_<NAME>
__gcp_labels_<NAME>

Internal labels available for push

__gcp_message_id
__gcp_subscription_name
__gcp_attributes_<NAME>: All attributes read from .message.attributes in the incoming push message. Each attribute key is conveniently renamed, since it might contain unsupported characters. For example, logging.googleapis.com/timestamp is converted to __gcp_attributes_logging_googleapis_com_timestamp.
__gcp_logname
__gcp_severity
__gcp_resource_type
__gcp_resource_labels_<NAME>
__gcp_labels_<NAME>

Azure Event Hubs

The azure_event_hubs block configures how Promtail receives Azure Event Hubs messages. Promtail uses an Apache Kafka endpoint on Event Hubs to receive messages. For more information, see the Azure Event Hubs documentation.

To learn more about streaming Azure logs to an Azure Event Hubs, you can see this tutorial.

Note that an Apache Kafka endpoint is not available within the Basic pricing plan. For more information, see the Event Hubs pricing page.

# Event Hubs namespace host names (Required). Typically, it looks like <your-namespace>.servicebus.windows.net:9093.
fully_qualified_namespace: <string> | default = ""

# Event Hubs to consume (Required).
event_hubs:
    [ - <string> ... ]

# Event Hubs ConnectionString for authentication on Azure Cloud (Required).
connection_string: <string> | default = "range"

# The consumer group id.
[group_id: <string> | default = "promtail"]

# If Promtail should pass on the timestamp from the incoming message or not.
# When false Promtail will assign the current timestamp to the log when it was processed.
[use_incoming_timestamp: <bool> | default = false]

# If Promtail should ignore messages that don't match the schema for Azure resource logs.
# Schema is described here https://learn.microsoft.com/en-us/azure/azure-monitor/essentials/resource-logs-schema.
[disallow_custom_messages: <bool> | default = false]

# Labels optionally hold labels to associate with each log line.
[labels]:
  [ <labelname>: <labelvalue> ... ]

Available Labels

When Promtail receives Azure Event Hubs messages, various internal labels are made available for relabeling.

__azure_event_hubs_category: The log category of the message when a message is an application log.

The following list of labels is discovered using the Kafka endpoint in Event Hubs.

__meta_kafka_topic: The current topic for where the message has been read.
__meta_kafka_partition: The partition id where the message has been read.
__meta_kafka_member_id: The consumer group member id.
__meta_kafka_group_id: The consumer group id.
__meta_kafka_message_key: The message key. If it is empty, this value will be ’none'.

kafka

The kafka block configures Promtail to scrape logs from Kafka using a group consumer.

The brokers should list available brokers to communicate with the Kafka cluster. Use multiple brokers when you want to increase availability.

The topics is the list of topics Promtail will subscribe to. If a topic starts with ^ then a regular expression (RE2) is used to match topics. For instance ^promtail-.* will match the topic promtail-dev and promtail-prod. Topics are refreshed every 30 seconds, so if a new topic matches, it will be automatically added without requiring a Promtail restart.

The group_id defined the unique consumer group id to use for consuming logs. Each log record published to a topic is delivered to one consumer instance within each subscribing consumer group.

If all promtail instances have the same consumer group, then the records will effectively be load balanced over the promtail instances.
If all promtail instances have different consumer groups, then each record will be broadcast to all promtail instances.

The group_id is useful if you want to effectively send the data to multiple Loki instances and/or other sinks.

The assignor configuration allow you to select the rebalancing strategy to use for the consumer group. Rebalancing is the process where a group of consumer instances (belonging to the same group) co-ordinate to own a mutually exclusive set of partitions of topics that the group is subscribed to.

range the default, assigns partitions as ranges to consumer group members.
sticky assigns partitions to members with an attempt to preserve earlier assignments
roundrobin assigns partitions to members in alternating order.

The version allows to select the kafka version required to connect to the cluster.(default to 2.2.1)

By default, timestamps are assigned by Promtail when the message is read, if you want to keep the actual message timestamp from Kafka you can set the use_incoming_timestamp to true.

# The list of brokers to connect to kafka (Required).
[brokers: <strings> | default = [""]]

# The list of Kafka topics to consume (Required).
[topics: <strings> | default = [""]]

# The Kafka consumer group id.
[group_id: <string> | default = "promtail"]

# The consumer group rebalancing strategy to use. (e.g `sticky`, `roundrobin` or `range`)
[assignor: <string> | default = "range"]

# Kafka version to connect to.
[version: <string> | default = "2.2.1"]

# Optional authentication configuration with Kafka brokers
authentication:
  # Type is authentication type. Supported values [none, ssl, sasl]
  [type: <string> | default = "none"]

  # TLS configuration for authentication and encryption. It is used only when authentication type is ssl.
  tls_config:
    [ <tls_config> ]

  # SASL configuration for authentication. It is used only when authentication type is sasl.
  sasl_config:
    # SASL mechanism. Supported values [PLAIN, SCRAM-SHA-256, SCRAM-SHA-512]
    [mechanism: <string> | default = "PLAIN"]

    # The user name to use for SASL authentication
    [user: <string>]

    # The password to use for SASL authentication
    [password: <secret>]

    # If true, SASL authentication is executed over TLS
    [use_tls: <boolean> | default = false]

    # The CA file to use to verify the server
    [ca_file: <string>]

    # Validates that the server name in the server's certificate
    # is this value.
    [server_name: <string>]

    # If true, ignores the server certificate being signed by an
    # unknown CA.
    [insecure_skip_verify: <boolean> | default = false]


# Label map to add to every log line read from kafka
labels:
  [ <labelname>: <labelvalue> ... ]

# If Promtail should pass on the timestamp from the incoming log or not.
# When false Promtail will assign the current timestamp to the log when it was processed
[use_incoming_timestamp: <bool> | default = false]

Available Labels

The list of labels below are discovered when consuming kafka:

__meta_kafka_topic: The current topic for where the message has been read.
__meta_kafka_partition: The partition id where the message has been read.
__meta_kafka_member_id: The consumer group member id.
__meta_kafka_group_id: The consumer group id.
__meta_kafka_message_key: The message key. If it is empty, this value will be ’none'.

To keep discovered labels to your logs use the relabel_configs section.

GELF

The gelf block configures a GELF UDP listener allowing users to push logs to Promtail with the GELF protocol. Currently only UDP is supported, submit a feature request if you’re interested into TCP support.

Each GELF message received will be encoded in JSON as the log line. For example:

{"version":"1.1","host":"example.org","short_message":"A short message","timestamp":1231231123,"level":5,"_some_extra":"extra"}

You can leverage pipeline stages with the GELF target, if for example, you want to parse the log line and extract more labels or change the log line format.

# UDP address to listen on. Has the format of "host:port". Default to 0.0.0.0:12201
listen_address: <string>

# Label map to add to every log message.
labels:
  [ <labelname>: <labelvalue> ... ]

# Whether Promtail should pass on the timestamp from the incoming gelf message.
# When false, or if no timestamp is present on the gelf message, Promtail will assign the current timestamp to the log when it was processed.
# Default is false
use_incoming_timestamp: <bool>

Available Labels

__gelf_message_level: The GELF level as string.
__gelf_message_host: The host sending the GELF message.
__gelf_message_version: The GELF level message version set by the client.
__gelf_message_facility: The GELF facility.

To keep discovered labels to your logs use the relabel_configs section.

Cloudflare

The cloudflare block configures Promtail to pull logs from the Cloudflare Logpull API.

These logs contain data related to the connecting client, the request path through the Cloudflare network, and the response from the origin web server. This data is useful for enriching existing logs on an origin server.

# The Cloudflare API token to use. (Required)
# You can create a new token by visiting your [Cloudflare profile](https://dash.cloudflare.com/profile/api-tokens).
api_token: <string>

# The Cloudflare zone id to pull logs for. (Required)
zone_id: <string>

# The time range to pull logs for.
[pull_range: <duration> | default = 1m]

# The quantity of workers that will pull logs.
[workers: <int> | default = 3]

# The type list of fields to fetch for logs.
# Supported values: default, minimal, extended, all.
[fields_type: <string> | default = default]

# The additional list of fields to supplement those provided via `fields_type`.
[additional_fields: <string> ... ]

# Label map to add to every log message.
labels:
  [ <labelname>: <labelvalue> ... ]

By default Promtail fetches logs with the default set of fields. Here are the different set of fields type available and the fields they include :

default includes "ClientIP", "ClientRequestHost", "ClientRequestMethod", "ClientRequestURI", "EdgeEndTimestamp", "EdgeResponseBytes", "EdgeRequestHost", "EdgeResponseStatus", "EdgeStartTimestamp", "RayID" plus any extra fields provided via additional_fields argument.
minimal includes all default fields and adds "ZoneID", "ClientSSLProtocol", "ClientRequestProtocol", "ClientRequestPath", "ClientRequestUserAgent", "ClientRequestReferer", "EdgeColoCode", "ClientCountry", "CacheCacheStatus", "CacheResponseStatus", "EdgeResponseContentType" plus any extra fields provided via additional_fields argument.
extended includes all minimalfields and adds "ClientSSLCipher", "ClientASN", "ClientIPClass", "CacheResponseBytes", "EdgePathingOp", "EdgePathingSrc", "EdgePathingStatus", "ParentRayID", "WorkerCPUTime", "WorkerStatus", "WorkerSubrequest", "WorkerSubrequestCount", "OriginIP", "OriginResponseStatus", "OriginSSLProtocol", "OriginResponseHTTPExpires", "OriginResponseHTTPLastModified" plus any extra fields provided via additional_fields argument.
all includes all extended fields and adds "BotScore", "BotScoreSrc", "BotTags", "ClientRequestBytes", "ClientSrcPort", "ClientXRequestedWith", "CacheTieredFill", "EdgeResponseCompressionRatio", "EdgeServerIP", "FirewallMatchesSources", "FirewallMatchesActions", "FirewallMatchesRuleIDs", "OriginResponseBytes", "OriginResponseTime", "ClientDeviceType", "WAFFlags", "WAFMatchedVar", "EdgeColoID", "RequestHeaders", "ResponseHeaders", "ClientRequestSource" plus any extra fields provided via additional_fields argument (this is still relevant when new fields are made available via Cloudflare API but are not yet included in all).
custom includes only the fields defined in additional_fields.

To learn more about each field and its value, refer to the Cloudflare documentation.

Promtail saves the last successfully-fetched timestamp in the position file. If a position is found in the file for a given zone ID, Promtail will restart pulling logs from that position. When no position is found, Promtail will start pulling logs from the current time.

Promtail fetches logs using multiple workers (configurable via workers) which request the last available pull range (configured via pull_range) repeatedly. Verify the last timestamp fetched by Promtail using the cloudflare_target_last_requested_end_timestamp metric. It is possible for Promtail to fall behind due to having too many log lines to process for each pull. Adding more workers, decreasing the pull range, or decreasing the quantity of fields fetched can mitigate this performance issue.

All Cloudflare logs are in JSON. Here is an example:

{
	"CacheCacheStatus": "miss",
	"CacheResponseBytes": 8377,
	"CacheResponseStatus": 200,
	"CacheTieredFill": false,
	"ClientASN": 786,
	"ClientCountry": "gb",
	"ClientDeviceType": "desktop",
	"ClientIP": "100.100.5.5",
	"ClientIPClass": "noRecord",
	"ClientRequestBytes": 2691,
	"ClientRequestHost": "www.foo.com",
	"ClientRequestMethod": "GET",
	"ClientRequestPath": "/comments/foo/",
	"ClientRequestProtocol": "HTTP/1.0",
	"ClientRequestReferer": "https://www.foo.com/foo/168855/?offset=8625",
	"ClientRequestURI": "/foo/15248108/",
	"ClientRequestUserAgent": "some bot",
        "ClientRequestSource": "1"
	"ClientSSLCipher": "ECDHE-ECDSA-AES128-GCM-SHA256",
	"ClientSSLProtocol": "TLSv1.2",
	"ClientSrcPort": 39816,
	"ClientXRequestedWith": "",
	"EdgeColoCode": "MAN",
	"EdgeColoID": 341,
	"EdgeEndTimestamp": 1637336610671000000,
	"EdgePathingOp": "wl",
	"EdgePathingSrc": "macro",
	"EdgePathingStatus": "nr",
	"EdgeRateLimitAction": "",
	"EdgeRateLimitID": 0,
	"EdgeRequestHost": "www.foo.com",
	"EdgeResponseBytes": 14878,
	"EdgeResponseCompressionRatio": 1,
	"EdgeResponseContentType": "text/html",
	"EdgeResponseStatus": 200,
	"EdgeServerIP": "8.8.8.8",
	"EdgeStartTimestamp": 1637336610517000000,
	"FirewallMatchesActions": [],
	"FirewallMatchesRuleIDs": [],
	"FirewallMatchesSources": [],
	"OriginIP": "8.8.8.8",
	"OriginResponseBytes": 0,
	"OriginResponseHTTPExpires": "",
	"OriginResponseHTTPLastModified": "",
	"OriginResponseStatus": 200,
	"OriginResponseTime": 123000000,
	"OriginSSLProtocol": "TLSv1.2",
	"ParentRayID": "00",
	"RayID": "6b0a...",
        "RequestHeaders": [],
        "ResponseHeaders": [
          "x-foo": "bar"
        ],
	"SecurityLevel": "med",
	"WAFAction": "unknown",
	"WAFFlags": "0",
	"WAFMatchedVar": "",
	"WAFProfile": "unknown",
	"WAFRuleID": "",
	"WAFRuleMessage": "",
	"WorkerCPUTime": 0,
	"WorkerStatus": "unknown",
	"WorkerSubrequest": false,
	"WorkerSubrequestCount": 0,
	"ZoneID": 1234
}

You can leverage pipeline stages if, for example, you want to parse the JSON log line and extract more labels or change the log line format.

heroku_drain

The heroku_drain block configures Promtail to expose a Heroku HTTPS Drain.

Each job configured with a Heroku Drain will expose a Drain and will require a separate port.

The server configuration is the same as server, since Promtail exposes an HTTP server for each new drain.

Promtail exposes an endpoint at /heroku/api/v1/drain, which expects requests from Heroku’s log delivery.

# The Heroku drain server configuration options
[server: <server_config>]

# Label map to add to every log message.
labels:
  [ <labelname>: <labelvalue> ... ]

# Whether Promtail should pass on the timestamp from the incoming Heroku drain message.
# When false, or if no timestamp is present in the syslog message, Promtail will assign the current
# timestamp to the log when it was processed.
[use_incoming_timestamp: <boolean> | default = false]

Available Labels

Heroku Log drains send logs in Syslog-formatted messages (with some minor tweaks; they are not RFC-compatible).

The Heroku Drain target exposes for each log entry the received syslog fields with the following labels:

__heroku_drain_host: The HOSTNAME field parsed from the message.
__heroku_drain_app: The APP-NAME field parsed from the message.
__heroku_drain_proc: The PROCID field parsed from the message.
__heroku_drain_log_id: The MSGID field parsed from the message.

Additionally, the Heroku drain target will read all url query parameters from the configured drain target url and make them available as __heroku_drain_param_<name> labels, multiple instances of the same parameter will appear as comma separated strings

relabel_configs

Relabeling is a powerful tool to dynamically rewrite the label set of a target before it gets scraped. Multiple relabeling steps can be configured per scrape configuration. They are applied to the label set of each target in order of their appearance in the configuration file.

After relabeling, the instance label is set to the value of __address__ by default if it was not set during relabeling. The __scheme__ and __metrics_path__ labels are set to the scheme and metrics path of the target respectively. The __param_<name> label is set to the value of the first passed URL parameter called <name>.

Additional labels prefixed with __meta_ may be available during the relabeling phase. They are set by the service discovery mechanism that provided the target and vary between mechanisms.

Labels starting with __ will be removed from the label set after target relabeling is completed.

If a relabeling step needs to store a label value only temporarily (as the input to a subsequent relabeling step), use the __tmp label name prefix. This prefix is guaranteed to never be used by Prometheus itself.

# The source labels select values from existing labels. Their content is concatenated
# using the configured separator and matched against the configured regular expression
# for the replace, keep, and drop actions.
[ source_labels: '[' <labelname> [, ...] ']' ]

# Separator placed between concatenated source label values.
[ separator: <string> | default = ; ]

# Label to which the resulting value is written in a replace action.
# It is mandatory for replace actions. Regex capture groups are available.
[ target_label: <labelname> ]

# Regular expression against which the extracted value is matched.
[ regex: <regex> | default = (.*) ]

# Modulus to take of the hash of the source label values.
[ modulus: <uint64> ]

# Replacement value against which a regex replace is performed if the
# regular expression matches. Regex capture groups are available.
[ replacement: <string> | default = $1 ]

# Action to perform based on regex matching.
[ action: <relabel_action> | default = replace ]

<regex> is any valid RE2 regular expression. It is required for the replace, keep, drop, labelmap,labeldrop and labelkeep actions. The regex is anchored on both ends. To un-anchor the regex, use .*<regex>.*.

<relabel_action> determines the relabeling action to take:

replace: Match regex against the concatenated source_labels. Then, set target_label to replacement, with match group references (${1}, ${2}, …) in replacement substituted by their value. If regex does not match, no replacement takes place.
keep: Drop targets for which regex does not match the concatenated source_labels.
drop: Drop targets for which regex matches the concatenated source_labels.
hashmod: Set target_label to the modulus of a hash of the concatenated source_labels.
labelmap: Match regex against all label names. Then copy the values of the matching labels to label names given by replacement with match group references (${1}, ${2}, …) in replacement substituted by their value.
labeldrop: Match regex against all label names. Any label that matches will be removed from the set of labels.
labelkeep: Match regex against all label names. Any label that does not match will be removed from the set of labels.

Care must be taken with labeldrop and labelkeep to ensure that logs are still uniquely labeled once the labels are removed.

static_configs

A static_configs allows specifying a list of targets and a common label set for them. It is the canonical way to specify static targets in a scrape configuration.

# Configures the discovery to look on the current machine.
# This is required by the prometheus service discovery code but doesn't
# really apply to Promtail which can ONLY look at files on the local machine
# As such it should only have the value of localhost, OR it can be excluded
# entirely and a default value of localhost will be applied by Promtail.
targets:
  - localhost

# Defines a file to scrape and an optional set of additional labels to apply to
# all streams defined by the files from __path__.
labels:
  # The path to load logs from. Can use glob patterns (e.g., /var/log/*.log).
  __path__: <string>

  # Used to exclude files from being loaded. Can also use glob patterns.
  __path_exclude__: <string>

  # Additional labels to assign to the logs
  [ <labelname>: <labelvalue> ... ]

file_sd_config

File-based service discovery provides a more generic way to configure static targets and serves as an interface to plug in custom service discovery mechanisms.

It reads a set of files containing a list of zero or more <static_config>s. Changes to all defined files are detected via disk watches and applied immediately. Files may be provided in YAML or JSON format. Only changes resulting in well-formed target groups are applied.

The JSON file must contain a list of static configs, using this format:

[
  {
    "targets": [ "localhost" ],
    "labels": {
      "__path__": "<string>", ...
      "<labelname>": "<labelvalue>", ...
    }
  },
  ...
]

As a fallback, the file contents are also re-read periodically at the specified refresh interval.

Each target has a meta label __meta_filepath during the relabeling phase. Its value is set to the filepath from which the target was extracted.

# Patterns for files from which target groups are extracted.
files:
  [ - <filename_pattern> ... ]

# Refresh interval to re-read the files.
[ refresh_interval: <duration> | default = 5m ]

Where <filename_pattern> may be a path ending in .json, .yml or .yaml. The last path segment may contain a single * that matches any character sequence, e.g. my/path/tg_*.json.

kubernetes_sd_config

Kubernetes SD configurations allow retrieving scrape targets from Kubernetes’ REST API and always staying synchronized with the cluster state.

One of the following role types can be configured to discover targets:

`node`

The node role discovers one target per cluster node with the address defaulting to the Kubelet’s HTTP port.

The target address defaults to the first existing address of the Kubernetes node object in the address type order of NodeInternalIP, NodeExternalIP, NodeLegacyHostIP, and NodeHostName.

Available meta labels:

__meta_kubernetes_node_name: The name of the node object.
__meta_kubernetes_node_label_<labelname>: Each label from the node object.
__meta_kubernetes_node_labelpresent_<labelname>: true for each label from the node object.
__meta_kubernetes_node_annotation_<annotationname>: Each annotation from the node object.
__meta_kubernetes_node_annotationpresent_<annotationname>: true for each annotation from the node object.
__meta_kubernetes_node_address_<address_type>: The first address for each node address type, if it exists.

In addition, the instance label for the node will be set to the node name as retrieved from the API server.

`service`

The service role discovers a target for each service port of each service. This is generally useful for blackbox monitoring of a service. The address will be set to the Kubernetes DNS name of the service and respective service port.

Available meta labels:

__meta_kubernetes_namespace: The namespace of the service object.
__meta_kubernetes_service_annotation_<annotationname>: Each annotation from the service object.
__meta_kubernetes_service_annotationpresent_<annotationname>: “true” for each annotation of the service object.
__meta_kubernetes_service_cluster_ip: The cluster IP address of the service. (Does not apply to services of type ExternalName)
__meta_kubernetes_service_external_name: The DNS name of the service. (Applies to services of type ExternalName)
__meta_kubernetes_service_label_<labelname>: Each label from the service object.
__meta_kubernetes_service_labelpresent_<labelname>: true for each label of the service object.
__meta_kubernetes_service_name: The name of the service object.
__meta_kubernetes_service_port_name: Name of the service port for the target.
__meta_kubernetes_service_port_protocol: Protocol of the service port for the target.

`pod`

The pod role discovers all pods and exposes their containers as targets. For each declared port of a container, a single target is generated. If a container has no specified ports, a port-free target per container is created for manually adding a port via relabeling.

Available meta labels:

__meta_kubernetes_namespace: The namespace of the pod object.
__meta_kubernetes_pod_name: The name of the pod object.
__meta_kubernetes_pod_ip: The pod IP of the pod object.
__meta_kubernetes_pod_label_<labelname>: Each label from the pod object.
__meta_kubernetes_pod_labelpresent_<labelname>: truefor each label from the pod object.
__meta_kubernetes_pod_annotation_<annotationname>: Each annotation from the pod object.
__meta_kubernetes_pod_annotationpresent_<annotationname>: true for each annotation from the pod object.
__meta_kubernetes_pod_container_init: true if the container is an InitContainer
__meta_kubernetes_pod_container_name: Name of the container the target address points to.
__meta_kubernetes_pod_container_port_name: Name of the container port.
__meta_kubernetes_pod_container_port_number: Number of the container port.
__meta_kubernetes_pod_container_port_protocol: Protocol of the container port.
__meta_kubernetes_pod_ready: Set to true or false for the pod’s ready state.
__meta_kubernetes_pod_phase: Set to Pending, Running, Succeeded, Failed or Unknown in the lifecycle.
__meta_kubernetes_pod_node_name: The name of the node the pod is scheduled onto.
__meta_kubernetes_pod_host_ip: The current host IP of the pod object.
__meta_kubernetes_pod_uid: The UID of the pod object.
__meta_kubernetes_pod_controller_kind: Object kind of the pod controller.
__meta_kubernetes_pod_controller_name: Name of the pod controller.

`endpoints`

The endpoints role discovers targets from listed endpoints of a service. For each endpoint address one target is discovered per port. If the endpoint is backed by a pod, all additional container ports of the pod, not bound to an endpoint port, are discovered as targets as well.

Available meta labels:

__meta_kubernetes_namespace: The namespace of the endpoints object.
__meta_kubernetes_endpoints_name: The names of the endpoints object.
For all targets discovered directly from the endpoints list (those not additionally inferred from underlying pods), the following labels are attached:
- __meta_kubernetes_endpoint_hostname: Hostname of the endpoint.
- __meta_kubernetes_endpoint_node_name: Name of the node hosting the endpoint.
- __meta_kubernetes_endpoint_ready: Set to true or false for the endpoint’s ready state.
- __meta_kubernetes_endpoint_port_name: Name of the endpoint port.
- __meta_kubernetes_endpoint_port_protocol: Protocol of the endpoint port.
- __meta_kubernetes_endpoint_address_target_kind: Kind of the endpoint address target.
- __meta_kubernetes_endpoint_address_target_name: Name of the endpoint address target.
If the endpoints belong to a service, all labels of the role: service discovery are attached.
For all targets backed by a pod, all labels of the role: pod discovery are attached.

`ingress`

The ingress role discovers a target for each path of each ingress. This is generally useful for blackbox monitoring of an ingress. The address will be set to the host specified in the ingress spec.

Available meta labels:

__meta_kubernetes_namespace: The namespace of the ingress object.
__meta_kubernetes_ingress_name: The name of the ingress object.
__meta_kubernetes_ingress_label_<labelname>: Each label from the ingress object.
__meta_kubernetes_ingress_labelpresent_<labelname>: true for each label from the ingress object.
__meta_kubernetes_ingress_annotation_<annotationname>: Each annotation from the ingress object.
__meta_kubernetes_ingress_annotationpresent_<annotationname>: true for each annotation from the ingress object.
__meta_kubernetes_ingress_scheme: Protocol scheme of ingress, https if TLS config is set. Defaults to http.
__meta_kubernetes_ingress_path: Path from ingress spec. Defaults to /.

See below for the configuration options for Kubernetes discovery:

# The information to access the Kubernetes API.

# The API server addresses. If left empty, Prometheus is assumed to run inside
# of the cluster and will discover API servers automatically and use the pod's
# CA certificate and bearer token file at /var/run/secrets/kubernetes.io/serviceaccount/.
[ api_server: <host> ]

# The Kubernetes role of entities that should be discovered.
role: <role>

# Optional authentication information used to authenticate to the API server.
# Note that `basic_auth`, `bearer_token` and `bearer_token_file` options are
# mutually exclusive.
# password and password_file are mutually exclusive.

# Optional HTTP basic authentication information.
basic_auth:
  [ username: <string> ]
  [ password: <secret> ]
  [ password_file: <string> ]

# Optional bearer token authentication information.
[ bearer_token: <secret> ]

# Optional bearer token file authentication information.
[ bearer_token_file: <filename> ]

# Optional proxy URL.
[ proxy_url: <string> ]

# TLS configuration.
tls_config:
  [ <tls_config> ]

# Optional namespace discovery. If omitted, all namespaces are used.
namespaces:
  names:
    [ - <string> ]

# Optional label and field selectors to limit the discovery process to a subset of available
#  resources. See
# https://kubernetes.io/docs/concepts/overview/working-with-objects/field-selectors/
# and https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/ to learn
# more about the possible filters that can be used. The endpoints role supports pod,
# service, and endpoint selectors. Roles only support selectors matching the role itself;
# for example, the node role can only contain node selectors.
# Note: When making decisions about using field/label selectors, make sure that this
# is the best approach. It will prevent Promtail from reusing single list/watch
# for all scrape configurations. This might result in a bigger load on the Kubernetes API,
# because for each selector combination, there will be additional LIST/WATCH.
# On the other hand, if you want to monitor a small subset of pods of a large cluster,
# we recommend using selectors. The decision on the use of selectors or not depends
# on the particular situation.
[ selectors:
          [ - role: <string>
                  [ label: <string> ]
                  [ field: <string> ] ]]

Where <role> must be endpoints, service, pod, node, or ingress.

See this example Prometheus configuration file for a detailed example of configuring Prometheus for Kubernetes.

You may wish to check out the 3rd party Prometheus Operator, which automates the Prometheus setup on top of Kubernetes.

consul_sd_config

Consul SD configurations allow retrieving scrape targets from the Consul Catalog API. When using the Catalog API, each running Promtail will get a list of all services known to the whole consul cluster when discovering new targets.

The following meta labels are available on targets during relabeling:

__meta_consul_address: the address of the target
__meta_consul_dc: the datacenter name for the target
__meta_consul_health: the health status of the service
__meta_consul_metadata_<key>: each node metadata key value of the target
__meta_consul_node: the node name defined for the target
__meta_consul_service_address: the service address of the target
__meta_consul_service_id: the service ID of the target
__meta_consul_service_metadata_<key>: each service metadata key value of the target
__meta_consul_service_port: the service port of the target
__meta_consul_service: the name of the service the target belongs to
__meta_consul_tagged_address_<key>: each node tagged address key value of the target
__meta_consul_tags: the list of tags of the target joined by the tag separator

# The information to access the Consul Catalog API. It is to be defined
# as the Consul documentation requires.
[ server: <host> | default = "localhost:8500" ]
[ token: <secret> ]
[ datacenter: <string> ]
[ scheme: <string> | default = "http" ]
[ username: <string> ]
[ password: <secret> ]

tls_config:
  [ <tls_config> ]

# A list of services for which targets are retrieved. If omitted, all services
# are scraped.
services:
  [ - <string> ]

# See https://www.consul.io/api/catalog.html#list-nodes-for-service to know more
# about the possible filters that can be used.

# An optional list of tags used to filter nodes for a given service. Services must contain all tags in the list.
tags:
  [ - <string> ]

# Node metadata key/value pairs to filter nodes for a given service.
[ node_meta:
  [ <string>: <string> ... ] ]

# The string by which Consul tags are joined into the tag label.
[ tag_separator: <string> | default = , ]

# Allow stale Consul results (see https://www.consul.io/api/features/consistency.html). Will reduce load on Consul.
[ allow_stale: <boolean> | default = true ]

# The time after which the provided names are refreshed.
# On large setup it might be a good idea to increase this value because the catalog will change all the time.
[ refresh_interval: <duration> | default = 30s ]

Note that the IP number and port used to scrape the targets is assembled as <__meta_consul_address>:<__meta_consul_service_port>. However, in some Consul setups, the relevant address is in __meta_consul_service_address. In those cases, you can use the relabel feature to replace the special __address__ label.

The relabeling phase is the preferred and more powerful way to filter services or nodes for a service based on arbitrary labels. For users with thousands of services it can be more efficient to use the Consul API directly which has basic support for filtering nodes (currently by node metadata and a single tag).

consulagent_sd_config

Consul Agent SD configurations allow retrieving scrape targets from Consul’s Agent API. When using the Agent API, each running Promtail will only get services registered with the local agent running on the same host when discovering new targets. This is suitable for very large Consul clusters for which using the Catalog API would be too slow or resource intensive.

The following meta labels are available on targets during relabeling:

__meta_consulagent_address: the address of the target
__meta_consulagent_dc: the datacenter name for the target
__meta_consulagent_health: the health status of the service
__meta_consulagent_metadata_<key>: each node metadata key value of the target
__meta_consulagent_node: the node name defined for the target
__meta_consulagent_service_address: the service address of the target
__meta_consulagent_service_id: the service ID of the target
__meta_consulagent_service_metadata_<key>: each service metadata key value of the target
__meta_consulagent_service_port: the service port of the target
__meta_consulagent_service: the name of the service the target belongs to
__meta_consulagent_tagged_address_<key>: each node tagged address key value of the target
__meta_consulagent_tags: the list of tags of the target joined by the tag separator

# The information to access the Consul Agent API. It is to be defined
# as the Consul documentation requires.
[ server: <host> | default = "localhost:8500" ]
[ token: <secret> ]
[ datacenter: <string> ]
[ scheme: <string> | default = "http" ]
[ username: <string> ]
[ password: <secret> ]

tls_config:
  [ <tls_config> ]

# A list of services for which targets are retrieved. If omitted, all services
# are scraped.
services:
  [ - <string> ]

# See https://www.consul.io/api-docs/agent/service#filtering to know more
# about the possible filters that can be used.

# An optional list of tags used to filter nodes for a given service. Services must contain all tags in the list.
tags:
  [ - <string> ]

# Node metadata key/value pairs to filter nodes for a given service.
[ node_meta:
  [ <string>: <string> ... ] ]

# The string by which Consul tags are joined into the tag label.
[ tag_separator: <string> | default = , ]

Note that the IP address and port number used to scrape the targets is assembled as <__meta_consul_address>:<__meta_consul_service_port>. However, in some Consul setups, the relevant address is in __meta_consul_service_address. In those cases, you can use the relabel feature to replace the special __address__ label.

docker_sd_configs

Docker service discovery allows retrieving targets from a Docker daemon. It will only watch containers of the Docker daemon referenced with the host parameter. Docker service discovery should run on each node in a distributed setup. The containers must run with either the json-file or journald logging driver.

Note that the discovery will not pick up finished containers. That means Promtail will not scrape the remaining logs from finished containers after a restart.

The Docker target correctly joins log segments if a long line was split into different frames by Docker. To avoid hypothetically unlimited line size and out-of-memory errors in Promtail, this target applies a default soft line size limit of 256 kiB corresponding to the default max line size in Loki. If the buffer increases above this size, then the line will be sent to output immediately, and the rest of the line discarded. To change this behaviour, set limits_config.max_line_size to a non-zero value to apply a hard limit.

The configuration is inherited from Prometheus’ Docker service discovery.

# Address of the Docker daemon.  Use unix:///var/run/docker.sock for a local setup.
host: <string>

# Optional proxy URL.
[ proxy_url: <string> ]

# TLS configuration.
tls_config:
  [ <tls_config> ]

# The port to scrape metrics from, when `role` is nodes, and for discovered
# tasks and services that don't have published ports.
[ port: <int> | default = 80 ]

# The host to use if the container is in host networking mode.
[ host_networking_host: <string> | default = "localhost" ]

# Sort all non-nil networks in ascending order based on network name and
# get the first network if the container has multiple networks defined, 
# thus avoiding collecting duplicate targets.
[ match_first_network: <bool> | default = true ]

# Optional filters to limit the discovery process to a subset of available
# resources.
# The available filters are listed in the Docker documentation:
# Containers: https://docs.docker.com/engine/api/v1.41/#operation/ContainerList
[ filters:
  [ - name: <string>
      values: <string>, [...] ]
]

# The time after which the containers are refreshed.
[ refresh_interval: <duration> | default = 60s ]

# Authentication information used by Promtail to authenticate itself to the
# Docker daemon.
# Note that `basic_auth` and `authorization` options are mutually exclusive.
# `password` and `password_file` are mutually exclusive.

# Optional HTTP basic authentication information.
basic_auth:
  [ username: <string> ]
  [ password: <secret> ]
  [ password_file: <string> ]

# Optional `Authorization` header configuration.
authorization:
  # Sets the authentication type.
  [ type: <string> | default: Bearer ]
  # Sets the credentials. It is mutually exclusive with
  # `credentials_file`.
  [ credentials: <secret> ]
  # Sets the credentials to the credentials read from the configured file.
  # It is mutually exclusive with `credentials`.
  [ credentials_file: <filename> ]

# Optional OAuth 2.0 configuration.
# Cannot be used at the same time as basic_auth or authorization.
oauth2:
  [ <oauth2> ]

# Configure whether HTTP requests follow HTTP 3xx redirects.
[ follow_redirects: <bool> | default = true ]

Available meta labels:

__meta_docker_container_id: the ID of the container
__meta_docker_container_name: the name of the container
__meta_docker_container_network_mode: the network mode of the container
__meta_docker_container_label_<labelname>: each label of the container
__meta_docker_container_log_stream: the log stream type stdout or stderr
__meta_docker_network_id: the ID of the network
__meta_docker_network_name: the name of the network
__meta_docker_network_ingress: whether the network is ingress
__meta_docker_network_internal: whether the network is internal
__meta_docker_network_label_<labelname>: each label of the network
__meta_docker_network_scope: the scope of the network
__meta_docker_network_ip: the IP of the container in this network
__meta_docker_port_private: the port on the container
__meta_docker_port_public: the external port if a port-mapping exists
__meta_docker_port_public_ip: the public IP if a port-mapping exists

These labels can be used during relabeling. For instance, the following configuration scrapes the container named flog and removes the leading slash (/) from the container name.

scrape_configs:
  - job_name: flog_scrape
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
        filters:
          - name: name
            values: [flog]
    relabel_configs:
      - source_labels: ['__meta_docker_container_name']
        regex: '/(.*)'
        target_label: 'container'

limits_config

The optional limits_config block configures global limits for this instance of Promtail.

# When true, enforces rate limiting on this instance of Promtail.
[readline_rate_enabled: <bool> | default = false]

# The rate limit in log lines per second that this instance of Promtail may push to Loki.
[readline_rate: <int> | default = 10000]

# The cap in the quantity of burst lines that this instance of Promtail may push
# to Loki.
[readline_burst: <int> | default = 10000]

# When true, exceeding the rate limit causes this instance of Promtail to discard
# log lines, rather than sending them to Loki. When false, exceeding the rate limit
# causes this instance of Promtail to temporarily hold off on sending the log lines and retry later.
[readline_rate_drop: <bool> | default = true]

# Limits the max number of active streams.
# Limiting the number of streams is useful as a mechanism to limit memory usage by Promtail, which helps
# to avoid OOM scenarios.
# 0 means it is disabled.
[max_streams: <int> | default = 0]

# Maximum log line byte size allowed without dropping. Example: 256kb, 2M. 0 to disable.
# If disabled, targets may apply default buffer size safety limits. If a target implements
# a default limit, this will be documented under the `scrape_configs` entry.
[max_line_size: <int> | default = 0]
# Whether to truncate lines that exceed max_line_size. No effect if max_line_size is disabled
[max_line_size_truncate: <bool> | default = false]

target_config

The target_config block controls the behavior of reading files from discovered targets.

# Period to resync directories being watched and files being tailed to discover
# new ones or stop watching removed ones.
sync_period: "10s"

options_config

tracing_config

The tracing block configures tracing for Jaeger. Currently, limited to configuration per environment variables only.

# When true,
[enabled: <boolean> | default = false]

Example Docker Config

It’s fairly difficult to tail Docker files on a standalone machine because they are in different locations for every OS. We recommend the Docker logging driver for local Docker installs or Docker Compose.

If running in a Kubernetes environment, you should look at the defined configs which are in helm and jsonnet, these leverage the prometheus service discovery libraries (and give Promtail its name) for automatically finding and tailing pods. The jsonnet config explains with comments what each section is for.

Example Static Config

While Promtail may have been named for the prometheus service discovery code, that same code works very well for tailing logs without containers or container environments directly on virtual machines or bare metal.

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /var/log/positions.yaml # This location needs to be writeable by Promtail.

clients:
  - url: http://ip_or_hostname_where_Loki_run:3100/loki/api/v1/push

scrape_configs:
 - job_name: system
   pipeline_stages:
   static_configs:
   - targets:
      - localhost
     labels:
      job: varlogs  # A `job` label is fairly standard in prometheus and useful for linking metrics and logs.
      host: yourhost # A `host` label will help identify logs from this machine vs others
      __path__: /var/log/*.log  # The path matching uses a third party library: https://github.com/bmatcuk/doublestar

If you are rotating logs, be careful when using a wildcard pattern like *.log, and make sure it doesn’t match the rotated log file. For example, if you move your logs from server.log to server.01-01-1970.log in the same directory every night, a static config with a wildcard search pattern like *.log will pick up that new file and read it, effectively causing the entire days logs to be re-ingested.

Example Static Config without targets

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /var/log/positions.yaml # This location needs to be writeable by Promtail.

clients:
  - url: http://ip_or_hostname_where_Loki_run:3100/loki/api/v1/push

scrape_configs:
 - job_name: system
   pipeline_stages:
   static_configs:
   - labels:
      job: varlogs  # A `job` label is fairly standard in prometheus and useful for linking metrics and logs.
      host: yourhost # A `host` label will help identify logs from this machine vs others
      __path__: /var/log/*.log  # The path matching uses a third party library: https://github.com/bmatcuk/doublestar

Example Journal Config

This example reads entries from a systemd journal:

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://ip_or_hostname_where_loki_runs:3100/loki/api/v1/push

scrape_configs:
  - job_name: journal
    journal:
      max_age: 12h
      labels:
        job: systemd-journal
    relabel_configs:
      - source_labels: ['__journal__systemd_unit']
        target_label: 'unit'

Example Syslog Config

This example starts Promtail as a syslog receiver and can accept syslog entries in Promtail over TCP:

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki_addr:3100/loki/api/v1/push

scrape_configs:
  - job_name: syslog
    syslog:
      listen_address: 0.0.0.0:1514
      labels:
        job: "syslog"
    relabel_configs:
      - source_labels: ['__syslog_message_hostname']
        target_label: 'host'

Example Push Config

The example starts Promtail as a Push receiver and will accept logs from other Promtail instances or the Docker Logging Driver:

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://ip_or_hostname_where_Loki_run:3100/loki/api/v1/push

scrape_configs:
- job_name: push1
  loki_push_api:
    server:
      http_listen_port: 3500
      grpc_listen_port: 3600
    labels:
      pushserver: push1

Note the job_name must be provided and must be unique between multiple loki_push_api scrape_configs, it will be used to register metrics.

A new server instance is created so the http_listen_port and grpc_listen_port must be different from the Promtail server config section (unless it’s disabled)

You can set grpc_listen_port to 0 to have a random port assigned if not using httpgrpc.

Ingesting logs to Loki using OpenTelemetry Collector

Sat, 11 Apr 2026 00:53:11 +0000

Ingesting logs to Loki using OpenTelemetry Collector

Loki natively supports ingesting OpenTelemetry logs over HTTP. For ingesting logs to Loki using the OpenTelemetry Collector, you must use the otlphttp exporter.

For more information about using OpenTelemetry with Grafana products, refer to the Grafana OpenTelemetry documentation.

Loki configuration

When logs are ingested by Loki using an OpenTelemetry protocol (OTLP) ingestion endpoint, some of the data is stored as Structured Metadata.

You must set allow_structured_metadata to true within your Loki config file. Otherwise, Loki will reject the log payload as malformed. Note that Structured Metadata is enabled by default in Loki 3.0 and later.

limits_config:
  allow_structured_metadata: true

Configure the OpenTelemetry Collector to write logs into Loki

You need to make the following changes to the OpenTelemetry Collector config to write logs to Loki on its OTLP ingestion endpoint.

exporters:
  otlphttp:
    endpoint: http://<loki-addr>/otlp

And enable it in service.pipelines:

service:
  pipelines:
    logs:
      receivers: [...]
      processors: [...]
      exporters: [..., otlphttp]

If you want to authenticate using basic auth, we recommend the basicauth extension.

extensions:
  basicauth/otlp:
    client_auth:
      username: username
      password: password

exporters:
  otlphttp:
    auth:
      authenticator: basicauth/otlp
    endpoint: http://<loki-addr>/otlp

service:
  extensions: [basicauth/otlp]
  pipelines:
    logs:
      receivers: [...]
      processors: [...]
      exporters: [..., otlphttp]

Format considerations

Since the OpenTelemetry protocol differs from the Loki storage model, here is how data in the OpenTelemetry format will be mapped by default to the Loki data model during ingestion, which can be changed as explained later:

Index labels: Resource attributes map well to index labels in Loki, since both usually identify the source of the logs. The default list of Resource Attributes to store as Index labels can be configured using default_resource_attributes_as_index_labels under distributor’s otlp_config. By default, the following resource attributes will be stored as index labels, while the remaining attributes are stored as Structured Metadata with each log entry:
- cloud.availability_zone
- cloud.region
- container.name
- deployment.environment.name
- k8s.cluster.name
- k8s.container.name
- k8s.cronjob.name
- k8s.daemonset.name
- k8s.deployment.name
- k8s.job.name
- k8s.namespace.name
- k8s.pod.name
- k8s.replicaset.name
- k8s.statefulset.name
- service.instance.id
- service.name
- service.namespace
  
  Note
  Because Loki has a default limit of 15 index labels, we recommend storing only select resource attributes as index labels. Although the default config selects more than 15 Resource Attributes, it should be fine since a few are mutually exclusive.
  
  Tip
  For Grafana Cloud Logs, see the current OpenTelemetry guidance.
Timestamp: One of LogRecord.TimeUnixNano or LogRecord.ObservedTimestamp, based on which one is set. If both are not set, the ingestion timestamp will be used.
LogLine: LogRecord.Body holds the body of the log. However, since Loki only supports Log body in string format, we will stringify non-string values using the AsString method from the OTel collector lib.
Structured Metadata: Anything which can’t be stored in Index labels and LogLine would be stored as Structured Metadata. Here is a non-exhaustive list of what will be stored in Structured Metadata to give a sense of what it will hold:
- Resource Attributes not stored as Index labels is replicated and stored with each log entry.
- Everything under InstrumentationScope is replicated and stored with each log entry.
- Everything under LogRecord except LogRecord.Body, LogRecord.TimeUnixNano and sometimes LogRecord.ObservedTimestamp.

The default list of resource attributes to store as labels can be configured using default_resource_attributes_as_index_labels under the distributor’s otlp_config. You can set global limits using limits_config.otlp_config. If you are using Grafana Cloud, contact support to configure this setting.

Things to note before ingesting OpenTelemetry logs to Loki:

Dots (.) are converted to underscores (_).

Loki does not support . or any other special characters other than _ in label names. The unsupported characters are replaced with an _ while converting Attributes to Index Labels or Structured Metadata. Also, please note that while writing the queries, you must use the normalized format, i.e. use _ instead of special characters while querying data using OTel Attributes.

For example, service.name in OTLP would become service_name in Loki.
Flattening of nested Attributes

While converting Attributes in OTLP to Index labels or Structured Metadata, any nested attribute values are flattened out using _ as a separator. It is done in a similar way as to how it is done in the LogQL json parser.
Stringification of non-string Attribute values

While converting Attribute values in OTLP to Index label values or Structured Metadata, any non-string values are converted to string using AsString method from the OTel collector lib.

Changing the default mapping of OTLP to Loki Format

Loki supports per tenant OTLP config which lets you change the default mapping of OTLP to Loki format for each tenant. It currently only supports changing the storage of Attributes. Here is what the config looks like:

# OTLP log ingestion configurations
limits_config:
  otlp_config:
    # Configuration for Resource Attributes to store them as index labels or
    # Structured Metadata or drop them altogether
    resource_attributes:
      # Configure whether to ignore the default list of resource attributes set in
      # 'distributor.otlp.default_resource_attributes_as_index_labels' to be
      # stored as index labels and only use the given resource attributes config
      [ignore_defaults: <boolean>]
  
      [attributes_config: <list of attributes_configs>]
  
    # Configuration for Scope Attributes to store them as Structured Metadata or
    # drop them altogether
    [scope_attributes: <list of attributes_configs>]
  
    # Configuration for Log Attributes to store them as Structured Metadata or
    # drop them altogether
    [log_attributes: <list of attributes_configs>]
  
  attributes_config:
    # Configures action to take on matching Attributes. It allows one of
    # [structured_metadata, drop] for all Attribute types. It additionally allows
    # index_label action for Resource Attributes
    [action: <string> | default = ""]
  
    # List of attributes to configure how to store them or drop them altogether
    [attributes: <list of strings>]
  
    # Regex to choose attributes to configure how to store them or drop them
    # altogether
    [regex: <Regexp>]

Here are some example configs to change the default mapping of OTLP to Loki format:

Example 1

limits_config:
  otlp_config:
    resource_attributes:
      attributes_config:
        - action: index_label
          attributes:
            - service.group

With the example config, here is how various kinds of Attributes would be stored:

Store all 17 Resource Attributes mentioned earlier and service.group Resource Attribute as index labels.
Store remaining Resource Attributes as Structured Metadata.
Store all the Scope and Log Attributes as Structured Metadata.

Example 2

limits_config:
  otlp_config:
    resource_attributes:
      ignore_defaults: true
      attributes_config:
        - action: index_label
          regex: service.group

With the example config, here is how various kinds of Attributes would be stored:

Only store service.group Resource Attribute as index labels.
Store remaining Resource Attributes as Structured Metadata.
Store all the Scope and Log Attributes as Structured Metadata.

Example 3

limits_config:
  otlp_config:
    resource_attributes:
      attributes_config:
        - action: index_label
          regex: service.group
    scope_attributes:
      - action: drop
        attributes:
          - method.name
    log_attributes:
      - action: structured_metadata
        attributes:
          - user.id
      - action: drop
        regex: .*

With the example config, here is how various kinds of Attributes would be stored:

Store all 17 Resource Attributes mentioned earlier and service.group Resource Attribute as index labels.
Store remaining Resource Attributes as Structured Metadata.
Drop Scope Attribute named method.name and store all other Scope Attributes as Structured Metadata.
Store Log Attribute named user.id as Structured Metadata and drop all other Log Attributes.

Sending logs from the cloud

Sat, 11 Apr 2026 00:53:11 +0000

Sending logs from the cloud

Sending logs from cloud services to Grafana Loki is a little different depending on the AWS service you are using. The following tutorials walk you through configuring cloud services to send logs to Loki.

Configuring Promtail for service discovery

Sat, 11 Apr 2026 00:53:11 +0000

Configuring Promtail for service discovery

Promtail currently supports scraping from the following sources:

Azure Event Hubs

Promtail supports reading messages from Azure Event Hubs. Targets can be configured using the azure_event_hubs stanza:

- job_name: azure_event_hubs
  azure_event_hubs:
    group_id: "mygroup"
    fully_qualified_namespace: my-namespace.servicebus.windows.net:9093
    connection_string: "my-connection-string"
    event_hubs:
      - event-hub-name
    labels:
      job: azure_event_hub
  relabel_configs:
    - action: replace
      source_labels:
        - __azure_event_hubs_category
      target_label: category

Only fully_qualified_namespace, connection_string and event_hubs are required fields. Read the configuration section for more information.

Cloudflare

Promtail supports pulling HTTP log messages from Cloudflare using the Logpull API. The Cloudflare targets can be configured with a cloudflare block:

scrape_configs:
- job_name: cloudflare
  cloudflare:
    api_token: REDACTED
    zone_id: REDACTED
    fields_type: all
    labels:
      job: cloudflare-foo.com

Only api_token and zone_id are required. Refer to the Cloudfare configuration section for details.

File Target Discovery

Promtail discovers locations of log files and extract labels from them through the scrape_configs section in the config YAML. The syntax is identical to what Prometheus uses.

scrape_configs contains one or more entries which are executed for each discovered target (i.e., each container in each new pod running in the instance):

scrape_configs:
  - job_name: local
    static_configs:
      - ...

  - job_name: kubernetes
    kubernetes_sd_config:
      - ...

If more than one scrape config section matches your logs, you will get duplicate entries as the logs are sent in different streams likely with slightly different labels.

There are different types of labels present in Promtail:

Labels starting with __ (two underscores) are internal labels. They usually come from dynamic sources like service discovery. Once relabeling is done, they are removed from the label set. To persist internal labels so they’re sent to Grafana Loki, rename them so they don’t start with __. See Relabeling for more information.
Labels starting with __meta_kubernetes_pod_label_* are “meta labels” which are generated based on your Kubernetes pod’s labels.

For example, if your Kubernetes pod has a label name set to foobar, then the scrape_configs section will receive an internal label __meta_kubernetes_pod_label_name with a value set to foobar.
Other labels starting with __meta_kubernetes_* exist based on other Kubernetes metadata, such as the namespace of the pod (__meta_kubernetes_namespace) or the name of the container inside the pod (__meta_kubernetes_pod_container_name). Refer to the Prometheus docs for the full list of Kubernetes meta labels.
The __path__ label is a special label which Promtail uses after discovery to figure out where the file to read is located. Wildcards are allowed, for example /var/log/*.log to get all files with a log extension in the specified directory, and /var/log/**/*.log for matching files and directories recursively. For a full list of options check out the docs for the library Promtail uses.
The __path_exclude__ label is another special label Promtail uses after discovery, to exclude a subset of the files discovered using __path__ from being read in the current scrape_config block. It uses the same library to enable usage of wildcards and glob patterns.
The label filename is added for every file found in __path__ to ensure the uniqueness of the streams. It is set to the absolute path of the file the line was read from.

Example of File Discovery

To scrape a set of log files defined manually on a machine, one can use static_configs or file_sd_configs. Using static_configs, one must reload Promtail to apply modifications. Using file_sd_configs that reload is not needed. Promtail reloads discovery files when they are updated.

The below excerpt of Promtail’s configuration show a file_sd_configs that is used to scrape apt and dpkg’s logs.

scrape_configs:
 - job_name: apt-dpkg
    file_sd_configs:
      - files:
        - /etc/promtail/dpkg-apt.yaml
    refresh_interval: 5m

The targets to be scraped by Promtail are defined in /etc/promtail/dpkg-apt.yaml. In fact, Promtail read the target to scrape in the list of file provided under files.

Below is the content of /etc/promtail/dpkg-apt.yaml.

- targets: ["localhost"]
  labels:
    job: dpkg
    __path__: /var/log/dpkg.log
- targets: ["localhost"]
  labels:
    job: apt
    __path__: /var/log/apt/*.log

As one can realize, /etc/promtail/dpkg-apt.yaml contains the list of targets we would have defined under static_configs.
It defines two targets. The first one with label job set to dpkg and __path__ specifying dpkg’s log file: /var/log/dpkg.log. The second has two labels: the label job and again __path__ specifying the path to APT’s log files. This __path__ contains a glob. Every log file matching that regular expression will be scrapped under that target.
To summarize, the above /etc/promtail/dpkg-apt.yaml showcase YAML format of file_sd_config discovery file. The JSON format can be seen here,

Kubernetes Discovery

While Promtail can use the Kubernetes API to discover pods as targets, it can only read log files from pods that are running on the same node as the one Promtail is running on. Promtail looks for a __host__ label on each target and validates that it is set to the same hostname as Promtail’s (using either $HOSTNAME or the hostname reported by the kernel if the environment variable is not set).

This means that any time Kubernetes service discovery is used, there must be a relabel_config that creates the intermediate label __host__ from __meta_kubernetes_pod_node_name:

relabel_configs:
  - source_labels: ['__meta_kubernetes_pod_node_name']
    target_label: '__host__'

See Relabeling for more information. For more information on how to configure the service discovery see the Kubernetes Service Discovery configuration.

GCP Log scraping

Promtail supports scraping cloud resource logs such as GCS bucket logs, load balancer logs, and Kubernetes cluster logs from GCP. Configuration is specified in the gcplog section, within scrape_config.

There are two kind of scraping strategies: pull and push.

Pull

  - job_name: gcplog
    gcplog:
      subscription_type: "pull" # If the `subscription_type` field is empty, defaults to `pull`
      project_id: "my-gcp-project"
      subscription: "my-pubsub-subscription"
      use_incoming_timestamp: false # default rewrite timestamps.
      use_full_line: false # default use textPayload as log line.
      labels:
        job: "gcplog"
    relabel_configs:
      - source_labels: ['__gcp_resource_type']
        target_label: 'resource_type'
      - source_labels: ['__gcp_resource_labels_project_id']
        target_label: 'project'

Here project_id and subscription are the only required fields.

project_id is the GCP project id.
subscription is the GCP pubsub subscription where Promtail can consume log entries from.

Before using gcplog target, GCP should be configured with pubsub subscription to receive logs from.

It also supports relabeling and pipeline stages just like other targets.

When Promtail receives GCP logs, various internal labels are made available for relabeling:

__gcp_logname
__gcp_severity
__gcp_resource_type
__gcp_resource_labels_<NAME>
__gcp_labels_<NAME> In the example above, the project_id label from a GCP resource was transformed into a label called project through relabel_configs.

Push

  - job_name: gcplog
    gcplog:
      subscription_type: "push"
      use_incoming_timestamp: false
      labels:
        job: "gcplog-push"
      server:
        http_listen_address: 0.0.0.0
        http_listen_port: 8080
    relabel_configs:
      - source_labels: ['__gcp_message_id']
        target_label: 'message_id'
      - source_labels: ['__gcp_attributes_logging_googleapis_com_timestamp']
        target_label: 'incoming_ts'

When configuring the GCP Log push target, Promtail will start an HTTP server listening on port 8080, as configured in the server section. This server exposes the single endpoint POST /gcp/api/v1/push, responsible for receiving logs from GCP.

For Google’s PubSub to be able to send logs, Promtail server must be publicly accessible, and support HTTPS. For that, Promtail can be deployed as part of a larger orchestration service like Kubernetes, which can handle HTTPS traffic through an ingress, or it can be hosted behind a proxy/gateway, offloading the HTTPS to that component and routing the request to Promtail. Once that’s solved, GCP can be configured to send logs to Promtail.

It also supports relabeling and pipeline stages.

When Promtail receives GCP logs, various internal labels are made available for relabeling:

__gcp_message_id
__gcp_subscription_name
__gcp_attributes_<NAME>
__gcp_logname
__gcp_severity
__gcp_resource_type
__gcp_resource_labels_<NAME>
__gcp_labels_<NAME>

In the example above, the __gcp_message_id and the __gcp_attributes_logging_googleapis_com_timestamp labels are transformed to message_id and incoming_ts through relabel_configs. All other internal labels, for example some other attribute, will be dropped by the target if not transformed.

GELF

Promtail supports listening message using the GELF UDP protocol. The GELF targets can be configured using the gelf stanza:

scrape_configs:
- job_name: gelf
  gelf:
    listen_address: "0.0.0.0:12201"
    use_incoming_timestamp: true
    labels:
      job: gelf
  relabel_configs:
      - action: replace
        source_labels:
          - __gelf_message_host
        target_label: host
      - action: replace
        source_labels:
          - __gelf_message_level
        target_label: level
      - action: replace
        source_labels:
          - __gelf_message_facility
        target_label: facility

Heroku Drain

Promtail supports receiving logs from a Heroku application by using a Heroku HTTPS Drain. Configuration is specified in aheroku_drain block within the Promtail scrape_config configuration.

- job_name: heroku_drain
    heroku_drain:
      server:
        http_listen_address: 0.0.0.0
        http_listen_port: 8080
      labels:
        job: heroku_drain_docs
      use_incoming_timestamp: true
    relabel_configs:
      - source_labels: ['__heroku_drain_host']
        target_label: 'host'
      - source_labels: ['__heroku_drain_app']
        target_label: 'source'
      - source_labels: ['__heroku_drain_proc']
        target_label: 'proc'
      - source_labels: ['__heroku_drain_log_id']
        target_label: 'log_id'

Within the scrape_configs configuration for a Heroku Drain target, the job_name must be a Prometheus-compatible metric name.

The server section configures the HTTP server created for receiving logs. labels defines a static set of label values added to each received log entry. use_incoming_timestamp can be used to pass the timestamp received from Heroku.

Before using a heroku_drain target, Heroku should be configured with the URL where the Promtail instance will be listening. Follow the steps in Heroku HTTPS Drain docs for using the Heroku CLI with a command like the following:

heroku drains:add [http|https]://HOSTNAME:8080/heroku/api/v1/drain -a HEROKU_APP_NAME

Getting the Heroku application name

Note that the __heroku_drain_app label will contain the source of the log line, either app or heroku and not the name of the heroku application.

The easiest way to provide the actual application name is to include a query parameter when creating the heroku drain and then relabel that parameter in your scraping config, for example:

heroku drains:add [http|https]://HOSTNAME:8080/heroku/api/v1/drain?app_name=HEROKU_APP_NAME -a HEROKU_APP_NAME

And then in a relabel_config:

    relabel_configs:
      - source_labels: ['__heroku_drain_param_app_name']
        target_label: 'app'

It also supports relabeling and pipeline stages just like other targets.

When Promtail receives Heroku Drain logs, various internal labels are made available for relabeling:

__heroku_drain_host
__heroku_drain_app
__heroku_drain_proc
__heroku_drain_log_id In the example above, the project_id label from a GCP resource was transformed into a label called project through relabel_configs.

HTTP client

Promtail uses the Prometheus HTTP client implementation for all calls to Loki. Therefore it can be configured using the clients stanza, where one or more connections to Loki can be established:

clients:
  - [ <client_option> ]

Refer to client_config from the Promtail Configuration reference for all available options.

Journal Scraping (Linux Only)

On systems with systemd, Promtail also supports reading from the journal. Unlike file scraping which is defined in the static_configs stanza, journal scraping is defined in a journal stanza:

scrape_configs:
  - job_name: journal
    journal:
      json: false
      max_age: 12h
      path: /var/log/journal
      matches: _TRANSPORT=kernel
      labels:
        job: systemd-journal
    relabel_configs:
      - source_labels: ['__journal__systemd_unit']
        target_label: 'unit'

All fields defined in the journal section are optional, and are just provided here for reference. The max_age field ensures that no older entry than the time specified will be sent to Loki; this circumvents “entry too old” errors. The path field tells Promtail where to read journal entries from. The labels map defines a constant list of labels to add to every journal entry that Promtail reads. The matches field adds journal filters. If multiple filters are specified matching different fields, the log entries are filtered by both, if two filters apply to the same field, then they are automatically matched as alternatives.

When the json field is set to true, messages from the journal will be passed through the pipeline as JSON, keeping all of the original fields from the journal entry. This is useful when you don’t want to index some fields but you still want to know what values they contained.

By default, Promtail reads from the journal by looking in the /var/log/journal and /run/log/journal paths. If running Promtail inside of a Docker container, the path appropriate to your distribution should be bind mounted inside of Promtail along with binding /etc/machine-id. Bind mounting /etc/machine-id to the path of the same name is required for the journal reader to know which specific journal to read from. For example:

docker run \
  -v /var/log/journal/:/var/log/journal/ \
  -v /run/log/journal/:/run/log/journal/ \
  -v /etc/machine-id:/etc/machine-id \
  grafana/promtail:latest \
  -config.file=/path/to/config/file.yaml

When Promtail reads from the journal, it brings in all fields prefixed with __journal_ as internal labels. Like in the example above, the _SYSTEMD_UNIT field from the journal was transformed into a label called unit through relabel_configs. See Relabeling for more information, also look at the systemd man pages for a list of fields exposed by the journal.

Here’s an example where the SYSTEMD_UNIT, HOSTNAME, and SYSLOG_IDENTIFIER are relabeled for use in Loki.

Keep in mind that labels prefixed with __ will be dropped, so relabeling is required to keep these labels.

- job_name: systemd-journal
  journal:
    labels:
      cluster: ops-tools1
      job: default/systemd-journal
    path: /var/log/journal
  relabel_configs:
  - source_labels:
    - __journal__systemd_unit
    target_label: systemd_unit
  - source_labels:
    - __journal__hostname
    target_label: nodename
  - source_labels:
    - __journal_syslog_identifier
    target_label: syslog_identifier

Kafka

Promtail supports reading message from Kafka using a consumer group. The Kafka targets can be configured using the kafka stanza:

scrape_configs:
- job_name: kafka
  kafka:
    brokers:
    - my-kafka-0.org:50705
    - my-kafka-1.org:50705
    topics:
    - ^promtail.*
    - some_fixed_topic
    labels:
      job: kafka
  relabel_configs:
      - action: replace
        source_labels:
          - __meta_kafka_topic
        target_label: topic
      - action: replace
        source_labels:
          - __meta_kafka_partition
        target_label: partition
      - action: replace
        source_labels:
          - __meta_kafka_group_id
        target_label: group
      - action: replace
        source_labels:
          - __meta_kafka_message_key
        target_label: message_key

Only the brokers and topics are required. Read the configuration section for more information.

Relabeling

Each scrape_configs entry can contain a relabel_configs stanza. relabel_configs is a list of operations to transform the labels from discovery into another form.

A single entry in relabel_configs can also reject targets by doing an action: drop if a label value matches a specified regex. When a target is dropped, the owning scrape_config will not process logs from that particular source. Other scrape_configs without the drop action reading from the same target may still use and forward logs from it to Loki.

A common use case of relabel_configs is to transform an internal label such as __meta_kubernetes_* into an intermediate internal label such as __service__. The intermediate internal label may then be dropped based on value or transformed to a final external label, such as __job__.

Examples

Drop the target if a label (__service__ in the example) is empty:

  - action: drop
    regex: ''
    source_labels:
    - __service__

Drop the target if any of the source_labels contain a value:

  - action: drop
    regex: .+
    separator: ''
    source_labels:
    - __meta_kubernetes_pod_label_name
    - __meta_kubernetes_pod_label_app

Persist an internal label by renaming it so it will be sent to Loki:

  - action: replace
    source_labels:
    - __meta_kubernetes_namespace
    target_label: namespace

Persist all Kubernetes pod labels by mapping them, like by mapping __meta_kube__meta_kubernetes_pod_label_foo to foo.

  - action: labelmap
    regex: __meta_kubernetes_pod_label_(.+)

Additional reading:

Julien Pivotto’s slides from PromConf Munich, 2017

Syslog Receiver

Promtail supports receiving IETF Syslog (RFC5424) messages from a TCP or UDP stream. Receiving syslog messages is defined in a syslog stanza:

scrape_configs:
  - job_name: syslog
    syslog:
      listen_address: 0.0.0.0:1514
      listen_protocol: tcp
      idle_timeout: 60s
      label_structured_data: yes
      labels:
        job: "syslog"
    relabel_configs:
      - source_labels: ['__syslog_message_hostname']
        target_label: 'host'

The only required field in the syslog section is the listen_address field, where a valid network address must be provided. The default protocol for receiving messages is TCP. To change the protocol, the listen_protocol field can be changed to udp. Note, that UDP does not support TLS. The idle_timeout can help with cleaning up stale syslog connections. If label_structured_data is set, structured data in the syslog header will be translated to internal labels in the form of __syslog_message_sd_<ID>_<KEY>. The labels map defines a constant list of labels to add to every journal entry that Promtail reads.

Note that it is recommended to deploy a dedicated syslog forwarder like syslog-ng or rsyslog in front of Promtail. The forwarder can take care of the various specifications and transports that exist (UDP, BSD syslog, …). See recommended output configurations for syslog-ng and rsyslog.

When Promtail receives syslog messages, it brings in all header fields, parsed from the received message, prefixed with __syslog_ as internal labels. Like in the example above, the __syslog_message_hostname field from the journal was transformed into a label called host through relabel_configs. See Relabeling for more information.

Syslog-NG Output Configuration

destination d_loki {
  syslog("localhost" transport("tcp") port(<promtail_port>));
};

Rsyslog Output Configuration

For sending messages via TCP:

*.* action(type="omfwd" protocol="tcp" target="<promtail_host>" port="<promtail_port>" Template="RSYSLOG_SyslogProtocol23Format" TCP_Framing="octet-counted" KeepAlive="on")

For sending messages via UDP:

*.* action(type="omfwd" protocol="udp" target="<promtail_host>" port="<promtail_port>" Template="RSYSLOG_SyslogProtocol23Format")

Windows Event Log

On Windows Promtail supports reading from the event log. Windows event targets can be configured using the windows_events stanza:

scrape_configs:
- job_name: windows
  windows_events:
    use_incoming_timestamp: false
    bookmark_path: "./bookmark.xml"
    eventlog_name: "Application"
    xpath_query: '*'
    labels:
      job: windows
  relabel_configs:
    - source_labels: ['computer']
      target_label: 'host'

When Promtail receives an event it will attach the channel and computer labels and serialize the event in json. You can relabel default labels via Relabeling if required.

Providing a path to a bookmark is mandatory, it will be used to persist the last event processed and allow resuming the target without skipping logs.

Read the configuration section for more information.

See the eventlogmessage stage for extracting data from the message.

Promtail and Log Rotation

Sat, 11 Apr 2026 00:53:11 +0000

Promtail and Log Rotation

Why does log rotation matter?

At any point in time, there may be three processes working on a log file as shown in the image below.

Appender - A writer that keeps appending to a log file. This can be your application or some system daemons like Syslog, Docker log driver or Kubelet, etc.
Tailer - A reader that reads log lines as they are appended, for example, agents like Promtail.
Log Rotator - A process that rotates the log file either based on time (for example, scheduled every day) or size (for example, a log file reached its maximum size).

One of the critical components here is the log rotator. Let’s understand how it impacts other components like the appender and tailer.

In general, when a software program rotates a log, it can be done in two ways internally.

Given a log file named error.log

Copy the log file with a different name, for example, error.log.1 and truncate the original log file error.log.
Rename the log file with a different name, for example, error.log.1 and create a new log file with the original name error.log.

In both cases, after log rotation, all new log lines are written to the original error.log file.

These two methods of log rotation are shown in the following images.

Copy and Truncate

Rename and Create

Both types of log rotation produce the same result. However, there are some subtle differences.

Copy and Truncate(1) favors the appender as its descriptor for the original log file error.log doesn’t change. Therefore, it can keep writing to the same file descriptor. In other words, re-opening the file is not needed.

However, (1) has a serious problem when considering the tailer. There is a race between truncating the file and the tailer finishing reading that log file. Meaning, there is a high chance of the log rotation mechanism truncating the file error.log before the tailer reads everything from it.

This is where Rename and Create(2) can help. Here, when the log file error.log is renamed to error.log.1, the tailer still holds the file descriptor of error.log.1. Therefore, it can continue reading the log file until it is completed. But that comes with the tradeoff: with (2), you have to signal the appender to reopen error.log (and appender should be able to reopen it). Otherwise, it would keep writing to error.log.1 as the file descriptor won’t change. The good news is that most of the popular appender solutions (for example, Syslog, Kubelet, Docker log driver) support reopening log files when they are renamed.

We recommend Rename and Create(2) as that is the method which works well with Promtail (or any similar log scraping agent) without any data loss. Now let’s understand exactly how we configure log rotation in different platforms.

Configure log rotation

Your logs can be rotated by different components depending on where you are running your applications or services. If you are running them on Linux machines, there is a high chance you are using the logrotate utility. However, if you are running in Kubernetes, it’s not that obvious what rotates the logs and interestingly it may depend on what container runtime your Kubernetes cluster is using.

Non-Kubernetes

As mentioned above, in Linux machines, log rotation is often handled by the logrotate utility.

The configuration for logrotate is usually located in /etc/logrotate/.

It has a wide range of options for compression, mailing, running scripts pre- and post-rotation, etc.

It supports both methods of log rotation described previously.

Copy and Truncate

/var/log/apache2/*.log {
        weekly
        maxsize 1G
        copytruncate
}

Here copytruncate mode works exactly like (1) explained above.

/var/log/apache2/*.log {
        weekly
        maxsize 1G
        create
}

Here, the create mode works as explained in (2) above. The create mode is optional because it’s the default mode in logrotate.

Kubernetes

Kubernetes Service Discovery in Promtail also uses file-based scraping. Meaning, logs from your pods are stored on the nodes and Promtail scrapes the pod logs from the node files.

You can configure the kubelet process running on each node to manage log rotation via two configuration settings.

containerLogMaxSize - It defines the maximum size of the container log file before it is rotated. For example: “5Mi” or “256Ki”. Default: “10Mi”.
containerLogMaxFiles - It specifies the maximum number of container log files that can be present per container. Default: 5

Both should be part of the kubelet config. If you run a managed version of Kubernetes in Cloud, refer to your cloud provider documentation for configuring kubelet. Examples GKE, AKS and EKS.

If kubelet is not configured to manage the log rotation, then it’s up to the Container Runtime Interface (CRI) the cluster uses. Alternatively, log rotation can be managed by the logrotate utility in the Kubernetes node itself.

Check your container runtime (CRI) on your nodes by running:

$ kubectl get nodes -o wide

Two of the commonly used CRI implementations are containerd and docker.

containerd CRI

At the time of writing this guide, containerd doesn’t support any method of log rotation. In this case, rotation is often handled by kubelet itself. Managed Kubernetes clusters like GKE and AKS use containerd as runtime and log rotation is handled by kubelet. EKS after version 1.24 also uses containerd as its default container runtime.

docker CRI

When using docker as runtime (EKS before 1.24 uses it by default), log rotation is managed by its logging driver (if supported). Docker has support for several logging drivers.

You can determine which logging driver docker is using by running the following command:

 docker info --format '{{.LoggingDriver}}'

Out of all these logging drivers only the local (default) and the json-file drivers support log rotation. You can configure the following log-opts under /etc/docker/daemon.json

max-size - The maximum size of the log file before it is rotated. A positive integer plus a modifier representing the unit of measure (k, m, or g). Defaults to 20m (20 megabytes).
max-file - The maximum number of log files that can be present. If rolling the logs creates excess files, the oldest file is removed. A positive integer. Defaults to 5.

Example /etc/docker/daemon.json:

{
  "log-driver": "local",
  "log-opts": {
    "max-size": "10m",
    "max-file": "10"
  }
}

If neither kubelet nor CRI is configured for rotating logs, then the logrotate utility can be used on the Kubernetes nodes as explained previously.

Configure Promtail

Promtail uses polling to watch for file changes. A polling mechanism combined with a copy and truncate log rotation may result in losing some logs. As explained earlier in this topic, this happens when the file is truncated before Promtail reads all the log lines from such a file.

Therefore, for a long-term solution, we strongly recommend changing the log rotation strategy to rename and create. Alternatively, as a workaround in the short term, you can tweak the promtail client’s batchsize config to set higher values (like 5M or 8M). This gives Promtail more room to read loglines without frequently waiting for push responses from the Loki server.

Pipelines

Sat, 11 Apr 2026 00:53:11 +0000

Pipelines

A detailed look at how to set up Promtail to process your log lines, including extracting metrics and labels.

Pipeline

A pipeline is used to transform a single log line, its labels, and its timestamp. A pipeline is comprised of a set of stages. There are 4 types of stages:

Parsing stages parse the current log line and extract data out of it. The extracted data is then available for use by other stages.
Transform stages transform extracted data from previous stages.
Action stages take extracted data from previous stages and do something with them. Actions can:
1. Add or modify existing labels to the log line
2. Change the timestamp of the log line
3. Change the content of the log line
4. Create a metric based on the extracted data
Filtering stages optionally apply a subset of stages or drop entries based on some condition.

Typical pipelines will start with a parsing stage (such as a regex or json stage) to extract data from the log line. Then, a series of action stages will be present to do something with that extracted data. The most common action stage will be a labels stage to turn extracted data into a label.

A common stage will also be the match stage to selectively apply stages or drop entries based on a LogQL stream selector and filter expressions.

Note that pipelines can not currently be used to deduplicate logs; Grafana Loki will receive the same log line multiple times if, for example:

Two scrape configs read from the same file
Duplicate log lines in a file are sent through a pipeline. Deduplication is not done.

However, Loki will perform some deduplication at query time for logs that have the exact same nanosecond timestamp, labels, and log contents.

This documented example gives a good glimpse of what you can achieve with a pipeline:

scrape_configs:
- job_name: kubernetes-pods-name
  kubernetes_sd_configs: ....
  pipeline_stages:

  # This stage is only going to run if the scraped target has a label
  # of "name" with value "promtail".
  - match:
      selector: '{name="promtail"}'
      stages:
      # The regex stage parses out a level, timestamp, and component. At the end
      # of the stage, the values for level, timestamp, and component are only
      # set internally for the pipeline. Future stages can use these values and
      # decide what to do with them.
      - regex:
          expression: '.*level=(?P<level>[a-zA-Z]+).*ts=(?P<timestamp>[T\d-:.Z]*).*component=(?P<component>[a-zA-Z]+)'

      # The labels stage takes the level and component entries from the previous
      # regex stage and promotes them to a label. For example, level=error may
      # be a label added by this stage.
      - labels:
          level:
          component:

      # Finally, the timestamp stage takes the timestamp extracted from the
      # regex stage and promotes it to be the new timestamp of the log entry,
      # parsing it as an RFC3339Nano-formatted value.
      - timestamp:
          format: RFC3339Nano
          source: timestamp

  # This stage is only going to run if the scraped target has a label of
  # "name" with a value of "nginx" and if the log line contains the word "GET"
  - match:
      selector: '{name="nginx"} |= "GET"'
      stages:
      # This regex stage extracts a new output by matching against some
      # values and capturing the rest.
      - regex:
          expression: \w{1,3}.\w{1,3}.\w{1,3}.\w{1,3}(?P<output>.*)

      # The output stage changes the content of the captured log line by
      # setting it to the value of output from the regex stage.
      - output:
          source: output

  # This stage is only going to run if the scraped target has a label of
  # "name" with a value of "jaeger-agent".
  - match:
      selector: '{name="jaeger-agent"}'
      stages:
      # The JSON stage reads the log line as a JSON string and extracts
      # the "level" field from the object for use in further stages.
      - json:
          expressions:
            level: level

      # The labels stage pulls the value from "level" that was extracted
      # from the previous stage and promotes it to a label.
      - labels:
          level:
- job_name: kubernetes-pods-app
  kubernetes_sd_configs: ....
  pipeline_stages:
  # This stage will only run if the scraped target has a label of "app"
  # with a name of *either* grafana or prometheus.
  - match:
      selector: '{app=~"grafana|prometheus"}'
      stages:
      # The regex stage will extract a level and component for use in further
      # stages, allowing the level to be defined as either lvl=<level> or
      # level=<level> and the component to be defined as either
      # logger=<component> or component=<component>
      - regex:
          expression: ".*(lvl|level)=(?P<level>[a-zA-Z]+).*(logger|component)=(?P<component>[a-zA-Z]+)"

      # The labels stage then promotes the level and component extracted from
      # the regex stage to labels.
      - labels:
          level:
          component:

  # This stage will only run if the scraped target has a label "app"
  # with a value of "some-app" and the log line doesn't contain the word "info"
  - match:
      selector: '{app="some-app"} != "info"'
      stages:
      # The regex stage tries to extract a Go panic by looking for panic:
      # in the log message.
      - regex:
          expression: ".*(?P<panic>panic: .*)"

      # The metrics stage is going to increment a panic_total metric counter
      # which Promtail exposes. The counter is only incremented when panic
      # was extracted from the regex stage.
      - metrics:
          panic_total:
            type: Counter
            description: "total count of panic"
            source: panic
            config:
              action: inc

Data Accessible to Stages

The following sections further describe the types that are accessible to each stage (although not all may be used):

Label Set

The current set of labels for the log line. Initialized to be the set of labels that were scraped along with the log line. The label set is only modified by an action stage, but filtering stages read from it.

The final label set will be index by Loki and can be used for queries.

Extracted Map

A collection of key-value pairs extracted during a parsing stage. Subsequent stages operate on the extracted map, either transforming them or taking action with them. At the end of a pipeline, the extracted map is discarded; for a parsing stage to be useful, it must always be paired with at least one action stage.

The extracted map is initialized with the same set of initial labels that were scraped along with the log line. This initial data allows for taking action on the values of labels inside pipeline stages that only manipulate the extracted map. For example, log entries tailed from files have the label filename whose value is the file path that was tailed. When a pipeline executes for that log entry, the initial extracted map would contain filename using the same value as the label.

Log Timestamp

The current timestamp for the log line. Action stages can modify this value. If left unset, it defaults to the time when the log was scraped.

The final value for the timestamp is sent to Loki.

Log Line

The current log line, represented as text. Initialized to be the text that Promtail scraped. Action stages can modify this value.

The final value for the log line is sent to Loki as the text content for the given log entry.

Stages

Refer to the Promtail Stages Configuration Reference for the schema on the various supported stages supported.

Promtail pipeline stages

Sat, 11 Apr 2026 00:53:11 +0000

Promtail pipeline stages

This section is a collection of all stages Promtail supports in a Pipeline.

Parsing stages:

docker: Extract data by parsing the log line using the standard Docker format.
cri: Extract data by parsing the log line using the standard CRI format.
regex: Extract data using a regular expression.
json: Extract data by parsing the log line as JSON.
logfmt: Extract data by parsing the log line as logfmt.
replace: Replace data using a regular expression.
multiline: Merge multiple lines into a multiline block.
geoip: Extract geoip data from extracted labels.

Transform stages:

template: Use Go templates to modify extracted data.
pack: Packs a log line in a JSON object allowing extracted values and labels to be placed inside the log line.
decolorize: Strips ANSI color sequences from the log line.

Action stages:

timestamp: Set the timestamp value for the log entry.
output: Set the log line text.
labeldrop: Drop label set for the log entry.
labelallow: Allow label set for the log entry.
labels: Update the label set for the log entry.
limit: Limit the rate lines will be sent to Loki.
sampling: Sampling the lines will be sent to Loki.
static_labels: Add static-labels to the log entry.
metrics: Calculate metrics based on extracted data.
tenant: Set the tenant ID value to use for the log entry.
structured_metadata: Add structured metadata to the log entry.

Filtering stages:

match: Conditionally run stages based on the label set.
drop: Conditionally drop log lines based on several options.

Troubleshooting Promtail

Sat, 11 Apr 2026 00:53:11 +0000

Troubleshooting Promtail

This document describes known failure modes of Promtail on edge cases and the adopted trade-offs.

Dry running

Promtail can be configured to print log stream entries instead of sending them to Loki. This can be used in combination with piping data to debug or troubleshoot Promtail log parsing.

In dry run mode, Promtail still support reading from a positions file however no update will be made to the targeted file, this is to ensure you can easily retry the same set of lines.

To start Promtail in dry run mode use the flag --dry-run as shown in the example below:

cat my.log | promtail --stdin --dry-run --client.url http://127.0.0.1:3100/loki/api/v1/push

Inspecting a config file

Promtail can validate and syntatically check your config file for a valid configuration. This can be used to check for errors and inconsistency in your config file and help prevent deploying invalid configurations.

In check syntax mode, promtail will just validate the config file and then exit:

promtail -config.file=myConfigFile.yaml -check-syntax

Inspecting pipeline stages

Promtail can output all changes to log entries as each pipeline stage is executed. Each log entry contains four fields:

line
timestamp
labels
extracted fields

Enable the inspection output using the --inspect command-line option. The --inspect option can be used in combination with --stdin and --dry-run.

cat my.log | promtail --stdin --dry-run --inspect --client.url http://127.0.0.1:3100/loki/api/v1/push

The output uses color to highlight changes. Additions are in green, modifications in yellow, and removals in red.

If no changes are applied during a stage, that is usually an indication of a misconfiguration or undesired behavior.

The --inspect flag should not be used in production, as the calculation of changes between pipeline stages negatively impacts Promtail’s performance.

Pipe data to Promtail

Promtail supports piping data for sending logs to Loki (via the flag --stdin). This is a very useful way to troubleshooting your configuration. Once you have Promtail installed you can for instance use the following command to send logs to a local Loki instance:

cat my.log | promtail --stdin  --client.url http://127.0.0.1:3100/loki/api/v1/push

You can also add additional labels from command line using:

cat my.log | promtail --stdin  --client.url http://127.0.0.1:3100/loki/api/v1/push --client.external-labels=k1=v1,k2=v2

This will add labels k1 and k2 with respective values v1 and v2.

In pipe mode Promtail also support file configuration using --config.file, however do note that positions config is not used and only the first scrape config is used.

static_configs: can be used to provide static labels, although the targets property is ignored.

If you don’t provide any scrape_config: a default one is used which will automatically adds the following default labels: {job="stdin",hostname="<detected_hostname>"}.

For example you could use this config below to parse and add the label level on all your piped logs:

clients:
  - url: http://localhost:3100/loki/api/v1/push

scrape_configs:
- job_name: system
  pipeline_stages:
  - regex:
      expression: '(level|lvl|severity)=(?P<level>\\w+)'
  - labels:
      level:
  static_configs:
  - labels:
      job: my-stdin-logs

cat my.log | promtail --config.file promtail.yaml

A tailed file is truncated while Promtail is not running

Given the following order of events:

Promtail is tailing /app.log
Promtail current position for /app.log is 100 (byte offset)
Promtail is stopped
/app.log is truncated and new logs are appended to it
Promtail is restarted

When Promtail is restarted, it reads the previous position (100) from the positions file. Two scenarios are then possible:

/app.log size is less than the position before truncating
/app.log size is greater than or equal to the position before truncating

If the /app.log file size is less than the previous position, then the file is detected as truncated and logs will be tailed starting from position 0. Otherwise, if the /app.log file size is greater than or equal to the previous position, Promtail can’t detect it was truncated while not running and will continue tailing the file from position 100.

Generally speaking, Promtail uses only the path to the file as key in the positions file. Whenever Promtail is started, for each file path referenced in the positions file, Promtail will read the file from the beginning if the file size is less than the offset stored in the position file, otherwise it will continue from the offset, regardless the file has been truncated or rolled multiple times while Promtail was not running.

Loki is unavailable

For each tailing file, Promtail reads a line, process it through the configured pipeline_stages and push the log entry to Loki. Log entries are batched together before getting pushed to Loki, based on the max batch duration client.batch-wait and size client.batch-size-bytes, whichever comes first.

In case of any error while sending a log entries batch, Promtail adopts a “retry then discard” strategy:

Promtail retries to send log entry to the ingester up to max_retries times
If all retries fail, Promtail discards the batch of log entries (which will be lost) and proceeds with the next one

You can configure the max_retries and the delay between two retries via the backoff_config in the Promtail config file:

clients:
  - url: INGESTER-URL
    backoff_config:
      min_period: 100ms
      max_period: 10s
      max_retries: 10

The following table shows an example of the total delay applied by the backoff algorithm with min_period: 100ms and max_period: 10s:

Retry	Min delay	Max delay	Total min delay	Total max delay
1	100ms	200ms	100ms	200ms
2	200ms	400ms	300ms	600ms
3	400ms	800ms	700ms	1.4s
4	800ms	1.6s	1.5s	3s
5	1.6s	3.2s	3.1s	6.2s
6	3.2s	6.4s	6.3s	12.6s
7	6.4s	10s	12.7s	22.6s
8	6.4s	10s	19.1s	32.6s
9	6.4s	10s	25.5s	42.6s
10	6.4s	10s	31.9s	52.6s
11	6.4s	10s	38.3s	62.6s
12	6.4s	10s	44.7s	72.6s
13	6.4s	10s	51.1s	82.6s
14	6.4s	10s	57.5s	92.6s
15	6.4s	10s	63.9s	102.6s
16	6.4s	10s	70.3s	112.6s
17	6.4s	10s	76.7s	122.6s
18	6.4s	10s	83.1s	132.6s
19	6.4s	10s	89.5s	142.6s
20	6.4s	10s	95.9s	152.6s

Log entries pushed after a Promtail crash / panic / abruptly termination

When Promtail shuts down gracefully, it saves the last read offsets in the positions file, so that on a subsequent restart it will continue tailing logs without duplicates neither losses.

In the event of a crash or abruptly termination, Promtail can’t save the last read offsets in the positions file. When restarted, Promtail will read the positions file saved at the last sync period and will continue tailing the files from there. This means that if new log entries have been read and pushed to the ingester between the last sync period and the crash, these log entries will be sent again to the ingester on Promtail restart.

If Loki is not configured to accept out-of-order writes, Loki will reject all log lines received in what it perceives is out of order. If Promtail happens to crash, it may re-send log lines that were sent prior to the crash. The default behavior of Promtail is to assign a timestamp to logs at the time it read the entry from the tailed file. This would result in duplicate log lines being sent to Loki; to avoid this issue, if your tailed file has a timestamp embedded in the log lines, a timestamp stage should be added to your pipeline.

Using k6 for load testing

Sat, 11 Apr 2026 00:53:11 +0000

Using k6 for load testing

Grafana k6 is a modern load-testing tool. Its clean and approachable scripting API works locally or in the cloud. Its configuration makes it flexible.

The xk6-loki extension permits pushing logs to and querying logs from a Loki instance. It acts as a Loki client, simulating real-world load to test the scalability, reliability, and performance of your Loki installation.

Before you begin

k6 is written in Golang. Download and install a Go environment.

Installation

xk6-loki is an extension to the k6 binary. Build a custom k6 binary that includes the xk6-loki extension.

Install the xk6 extension bundler:
Bash
```
go install go.k6.io/xk6/cmd/xk6@latest
```

Check out the grafana/xk6-loki repository:

git clone https://github.com/grafana/xk6-loki
cd xk6-loki

Build k6 with the extension:
Bash
```
make k6
```

Usage

Use the custom-built k6 binary in the same way as a non-custom k6 binary:

./k6 run test.js

test.js is a Javascript load test. Refer to the k6 documentation to get started.

Scripting API

The custom-built k6 binary provides a Javascript loki module.

Your Javascript load test imports the module:

import loki from 'k6/x/loki';

Classes of this module are:

class	description
`Config`	configuration for the `Client` class
`Client`	client for writing and reading logs from Loki

Config and Client must be called on the k6 init context (see Test life cycle) outside of the default function so the client is only configured once and shared between all VU iterations.

The Client class exposes the following instance methods:

method	description
`push()`	shortcut for `pushParameterized(5, 8001024, 10241024)`
`pushParameterized(streams, minSize, maxSize)`	execute push request ( POST /loki/api/v1/push
`instantQuery(query, limit)`	execute instant query ( GET /loki/api/v1/query
`client.rangeQuery(query, duration, limit)`	execute range query ( GET /loki/api/v1/query_range
`client.labelsQuery(duration)`	execute labels query ( GET /loki/api/v1/labels
`client.labelValuesQuery(label, duration)`	execute label values query ( GET /loki/api/v1/label/<name>/values
`client.seriesQuery(matchers, duration)`	execute series query ( GET /loki/api/v1/series

Javascript load test example:

import loki from 'k6/x/loki';

const timeout = 5000; // ms
const conf = loki.Config("http://localhost:3100", timeout);
const client = loki.Client(conf);

export default () => {
   client.pushParameterized(2, 512*1024, 1024*1024);
};

Refer to grafana/xk6-loki for the complete k6/x/loki module API reference.

Send log data to Grafana Enterprise Logs on Grafana Labs

Configure Promtail for Grafana Enterprise Logs

Configure Promtail for Grafana Enterprise Logs

Verify that Promtail is running

Configure Promtail to write to GEL

Run this Promtail configuration using Docker

Verify that Promtail is writing to GEL

Ingesting logs to Loki using Alloy

Ingesting logs to Loki using Alloy

Installing Alloy

Components of Alloy for logs

Log components in Alloy

Learning path

Interactive Tutorials

Install Promtail

Install Promtail

Install the binary

Install using APT or RPM package manager

Install using Docker

Install on MacOS with Homebrew

Install using Helm

Install as Kubernetes daemonSet (recommended)

Configure Promtail

Configure Promtail

Printing Promtail Config At Runtime

Configuration File Reference

Reload at runtime

Use environment variables in the configuration

Generic placeholders

Supported contents and default values of config.yaml:

global

file_watch_config

server

clients

positions

scrape_configs

pipeline_stages

docker

cri

regex

json

template

match

timestamp

output

labels

metrics

counter

gauge

histogram

tenant

replace

journal

Available Labels

syslog

Available Labels

loki_push_api

windows_events

GCP Log

Available Labels

Azure Event Hubs

Available Labels

kafka

Available Labels

GELF

Available Labels

Cloudflare

heroku_drain

Available Labels

relabel_configs

static_configs

file_sd_config

kubernetes_sd_config

node

service

pod

endpoints

ingress

consul_sd_config

consulagent_sd_config

Supported contents and default values of `config.yaml`:

`node`

`service`

`pod`

`endpoints`

`ingress`