Send log data to Loki on Grafana Labs

Ingesting logs to Loki using Alloy

Thu, 09 Apr 2026 02:28:18 +0000

Ingesting logs to Loki using Alloy

Grafana Alloy is a versatile observability collector that can ingest logs in various formats and send them to Loki. We recommend Alloy as the primary method for sending logs to Loki, as it provides a more robust and feature-rich solution for building a highly scalable and reliable observability pipeline.

Installing Alloy

To get started with Grafana Alloy and send logs to Loki, you need to install and configure Alloy. You can follow the Alloy documentation to install Alloy on your preferred platform.

Components of Alloy for logs

Alloy pipelines are built using components that perform specific functions. For logs these can be broken down into three categories:

Collector: These components collect/receive logs from various sources. This can be scraping logs from a file, receiving logs over HTTP, gRPC or ingesting logs from a message queue.
Transformer: These components can be used to manipulate logs before they are sent to a writer. This can be used to add additional metadata, filter logs, or batch logs before sending them to a writer.
Writer: These components send logs to the desired destination. Our documentation will focus on sending logs to Loki, but Alloy supports sending logs to various destinations.

Log components in Alloy

Here is a non-exhaustive list of components that can be used to build a log pipeline in Alloy. For a complete list of components, refer to the components list.

Type	Component
Collector	loki.source.api
Collector	loki.source.awsfirehose
Collector	loki.source.azure_event_hubs
Collector	loki.source.cloudflare
Collector	loki.source.docker
Collector	loki.source.file
Collector	loki.source.gcplog
Collector	loki.source.gelf
Collector	loki.source.heroku
Collector	loki.source.journal
Collector	loki.source.kafka
Collector	loki.source.kubernetes
Collector	loki.source.kubernetes_events
Collector	loki.source.podlogs
Collector	loki.source.syslog
Collector	loki.source.windowsevent
Collector	otelcol.receiver.loki
Transformer	loki.relabel
Transformer	loki.process
Writer	loki.write
Writer	otelcol.exporter.loki

Learning path

Start your learning experience with Grafana Learning Paths

Grafana Learning Paths provide a clear, structured path that leads you from beginner concepts to advanced use cases. Learn about this Grafana feature on Send logs to Grafana Cloud using Alloy.

Start learning

Interactive Tutorials

To learn more about how to configure Alloy to send logs to Loki within different scenarios, follow these interactive tutorials:

Ingesting logs to Loki using OpenTelemetry Collector

Thu, 09 Apr 2026 02:28:18 +0000

Ingesting logs to Loki using OpenTelemetry Collector

Loki natively supports ingesting OpenTelemetry logs over HTTP. For ingesting logs to Loki using the OpenTelemetry Collector, you must use the otlphttp exporter.

For more information about using OpenTelemetry with Grafana products, refer to the Grafana OpenTelemetry documentation.

Loki configuration

When logs are ingested by Loki using an OpenTelemetry protocol (OTLP) ingestion endpoint, some of the data is stored as Structured Metadata.

You must set allow_structured_metadata to true within your Loki config file. Otherwise, Loki will reject the log payload as malformed. Note that Structured Metadata is enabled by default in Loki 3.0 and later.

limits_config:
  allow_structured_metadata: true

Configure the OpenTelemetry Collector to write logs into Loki

You need to make the following changes to the OpenTelemetry Collector config to write logs to Loki on its OTLP ingestion endpoint.

exporters:
  otlphttp:
    endpoint: http://<loki-addr>/otlp

And enable it in service.pipelines:

service:
  pipelines:
    logs:
      receivers: [...]
      processors: [...]
      exporters: [..., otlphttp]

If you want to authenticate using basic auth, we recommend the basicauth extension.

extensions:
  basicauth/otlp:
    client_auth:
      username: username
      password: password

exporters:
  otlphttp:
    auth:
      authenticator: basicauth/otlp
    endpoint: http://<loki-addr>/otlp

service:
  extensions: [basicauth/otlp]
  pipelines:
    logs:
      receivers: [...]
      processors: [...]
      exporters: [..., otlphttp]

Format considerations

Since the OpenTelemetry protocol differs from the Loki storage model, here is how data in the OpenTelemetry format will be mapped by default to the Loki data model during ingestion, which can be changed as explained later:

Index labels: Resource attributes map well to index labels in Loki, since both usually identify the source of the logs. The default list of Resource Attributes to store as Index labels can be configured using default_resource_attributes_as_index_labels under distributor’s otlp_config. By default, the following resource attributes will be stored as index labels, while the remaining attributes are stored as Structured Metadata with each log entry:
- cloud.availability_zone
- cloud.region
- container.name
- deployment.environment.name
- k8s.cluster.name
- k8s.container.name
- k8s.cronjob.name
- k8s.daemonset.name
- k8s.deployment.name
- k8s.job.name
- k8s.namespace.name
- k8s.pod.name
- k8s.replicaset.name
- k8s.statefulset.name
- service.instance.id
- service.name
- service.namespace
  
  Note
  Because Loki has a default limit of 15 index labels, we recommend storing only select resource attributes as index labels. Although the default config selects more than 15 Resource Attributes, it should be fine since a few are mutually exclusive.
  
  Tip
  For Grafana Cloud Logs, see the current OpenTelemetry guidance.
Timestamp: One of LogRecord.TimeUnixNano or LogRecord.ObservedTimestamp, based on which one is set. If both are not set, the ingestion timestamp will be used.
LogLine: LogRecord.Body holds the body of the log. However, since Loki only supports Log body in string format, we will stringify non-string values using the AsString method from the OTel collector lib.
Structured Metadata: Anything which can’t be stored in Index labels and LogLine would be stored as Structured Metadata. Here is a non-exhaustive list of what will be stored in Structured Metadata to give a sense of what it will hold:
- Resource Attributes not stored as Index labels is replicated and stored with each log entry.
- Everything under InstrumentationScope is replicated and stored with each log entry.
- Everything under LogRecord except LogRecord.Body, LogRecord.TimeUnixNano and sometimes LogRecord.ObservedTimestamp.

The default list of resource attributes to store as labels can be configured using default_resource_attributes_as_index_labels under the distributor’s otlp_config. You can set global limits using limits_config.otlp_config. If you are using Grafana Cloud, contact support to configure this setting.

Caution
Because of the potential for high cardinality, k8s.pod.name and service.instance.id are no longer recommended as default labels. But because removing these resource attributes from the default labels would be a breaking change for existing users, they have not yet been deprecated as default labels. If you are a new user of Grafana Loki, we recommend that you modify your Alloy or OpenTelemetry Collector configuration to convert k8s.pod.name and service.instance.id from index labels to structured metadata. For sample configurations, refer to Remove default labels.

Things to note before ingesting OpenTelemetry logs to Loki:

Dots (.) are converted to underscores (_).

Loki does not support . or any other special characters other than _ in label names. The unsupported characters are replaced with an _ while converting Attributes to Index Labels or Structured Metadata. Also, please note that while writing the queries, you must use the normalized format, i.e. use _ instead of special characters while querying data using OTel Attributes.

For example, service.name in OTLP would become service_name in Loki.
Flattening of nested Attributes

While converting Attributes in OTLP to Index labels or Structured Metadata, any nested attribute values are flattened out using _ as a separator. It is done in a similar way as to how it is done in the LogQL json parser.
Stringification of non-string Attribute values

While converting Attribute values in OTLP to Index label values or Structured Metadata, any non-string values are converted to string using AsString method from the OTel collector lib.

Changing the default mapping of OTLP to Loki Format

Loki supports per tenant OTLP config which lets you change the default mapping of OTLP to Loki format for each tenant. It currently only supports changing the storage of Attributes. Here is what the config looks like:

# OTLP log ingestion configurations
limits_config:
  otlp_config:
    # Configuration for Resource Attributes to store them as index labels or
    # Structured Metadata or drop them altogether
    resource_attributes:
      # Configure whether to ignore the default list of resource attributes set in
      # 'distributor.otlp.default_resource_attributes_as_index_labels' to be
      # stored as index labels and only use the given resource attributes config
      [ignore_defaults: <boolean>]
  
      [attributes_config: <list of attributes_configs>]
  
    # Configuration for Scope Attributes to store them as Structured Metadata or
    # drop them altogether
    [scope_attributes: <list of attributes_configs>]
  
    # Configuration for Log Attributes to store them as Structured Metadata or
    # drop them altogether
    [log_attributes: <list of attributes_configs>]
  
  attributes_config:
    # Configures action to take on matching Attributes. It allows one of
    # [structured_metadata, drop] for all Attribute types. It additionally allows
    # index_label action for Resource Attributes
    [action: <string> | default = ""]
  
    # List of attributes to configure how to store them or drop them altogether
    [attributes: <list of strings>]
  
    # Regex to choose attributes to configure how to store them or drop them
    # altogether
    [regex: <Regexp>]

Note
If you are a Grafana Cloud customer, you can use the config self-serve API to configure your OTLP config.

Here are some example configs to change the default mapping of OTLP to Loki format:

Example 1

limits_config:
  otlp_config:
    resource_attributes:
      attributes_config:
        - action: index_label
          attributes:
            - service.group

With the example config, here is how various kinds of Attributes would be stored:

Store all 17 Resource Attributes mentioned earlier and service.group Resource Attribute as index labels.
Store remaining Resource Attributes as Structured Metadata.
Store all the Scope and Log Attributes as Structured Metadata.

Example 2

limits_config:
  otlp_config:
    resource_attributes:
      ignore_defaults: true
      attributes_config:
        - action: index_label
          regex: service.group

With the example config, here is how various kinds of Attributes would be stored:

Only store service.group Resource Attribute as index labels.
Store remaining Resource Attributes as Structured Metadata.
Store all the Scope and Log Attributes as Structured Metadata.

Example 3

limits_config:
  otlp_config:
    resource_attributes:
      attributes_config:
        - action: index_label
          regex: service.group
    scope_attributes:
      - action: drop
        attributes:
          - method.name
    log_attributes:
      - action: structured_metadata
        attributes:
          - user.id
      - action: drop
        regex: .*

With the example config, here is how various kinds of Attributes would be stored:

Store all 17 Resource Attributes mentioned earlier and service.group Resource Attribute as index labels.
Store remaining Resource Attributes as Structured Metadata.
Drop Scope Attribute named method.name and store all other Scope Attributes as Structured Metadata.
Store Log Attribute named user.id as Structured Metadata and drop all other Log Attributes.

Kubernetes Monitoring Helm tutorial

Thu, 09 Apr 2026 02:28:18 +0000

Kubernetes Monitoring Helm tutorial

One of the primary use cases for Loki is to collect and store logs from your Kubernetes cluster. These logs fall into three categories:

Pod logs: Logs generated by pods running in your cluster.
Kubernetes Events: Logs generated by the Kubernetes API server.
Node logs: Logs generated by the nodes in your cluster.

In this tutorial, we will deploy Loki and the Kubernetes Monitoring Helm chart to collect two of these log types: Pod logs and Kubernetes events. We will also deploy Grafana to visualize these logs.

Things to know

Before you begin, here are some things you should know:

Loki: Loki can run in a single binary mode or as a distributed system. In this tutorial, we will deploy Loki as a single binary, otherwise known as monolithic mode. Loki can be vertically scaled in this mode depending on the number of logs you are collecting. Grafana Labs recommends running Loki in a distributed/microservice mode for production use cases to monitor high volumes of logs.
Deployment: You will deploy Loki, Grafana, and Alloy (as part of the Kubernetes Monitoring Helm chart) in the meta namespace of your Kubernetes cluster. Make sure you have the necessary permissions to create resources in this namespace. These pods will also require resources to run, so consider the amount of capacity your nodes have available. It also possible to just deploy the Kubernetes monitoring Helm chart (since it has a minimal resource footprint) within your cluster and write logs to an external Loki instance or Grafana Cloud.
Storage: In this tutorial, Loki will use the default object storage backend provided in the Loki Helm chart; MinIO. You should migrate to a more production-ready storage backend like S3, GCS, Azure Blob Storage or a MinIO Cluster for production use cases.

Prerequisites

Before you begin, you will need the following:

A Kubernetes cluster running version 1.23 or later.
kubectl installed on your local machine.
Helm installed on your local machine.

Tip
Alternatively, you can try out this example in our interactive learning environment: Kubernetes Monitoring with Loki.

It’s a fully configured environment with all the dependencies already installed.

Provide feedback, report bugs, and raise issues in the Grafana Killercoda repository.

Create the `meta` and `prod` namespaces

The K8s Monitoring Helm chart will monitor two namespaces: meta and prod:

meta namespace: This namespace will be used to deploy Loki, Grafana, and Alloy.
prod namespace: This namespace will be used to deploy the sample application that will generate logs.

Create the meta and prod namespaces by running the following command:

kubectl create namespace meta && kubectl create namespace prod

Add the Grafana Helm repository

All three Helm charts (Loki, Grafana, and the Kubernetes Monitoring Helm) are available in the Grafana Helm repository. Add the Grafana Helm repository by running the following command:

helm repo add grafana https://grafana.github.io/helm-charts && helm repo update

As well as adding the repo to your local helm list, you should also run helm repo update to ensure you have the latest version of the charts.

Clone the tutorial repository

Clone the tutorial repository by running the following command:

git clone https://github.com/grafana/alloy-scenarios.git

Then change directories to the alloy-scenarios/k8s/logs directory:

cd alloy-scenarios/k8s/logs

The rest of this tutorial assumes you are in the alloy-scenarios/k8s/logs directory.

Deploy Loki

Grafana Loki will be used to store our collected logs. In this tutorial we will deploy Loki with a minimal footprint and use the default storage backend provided by the Loki Helm chart, MinIO.

To deploy Loki run the following command:

helm install --values loki-values.yml loki grafana/loki -n meta

This command will deploy Loki in the meta namespace. The command also includes a values file that specifies the configuration for Loki. For more details on how to configure the Loki Helm chart refer to the Loki Helm documentation.

Deploy Grafana

Next we will deploy Grafana to the meta namespace. You will use Grafana to visualize the logs stored in Loki. To deploy Grafana run the following command:

helm install --values grafana-values.yml grafana grafana/grafana --namespace meta

As before, the command also includes a values file that specifies the configuration for Grafana. There are two important configuration attributes to take note of:

adminUser and adminPassword: These are the credentials you will use to log in to Grafana. The values are admin and adminadminadmin respectively. The recommended practice is to either use a Kubernetes secret or allow Grafana to generate a password for you. For more details on how to configure the Grafana Helm chart, refer to the Grafana Helm documentation.

datasources: This section of the configuration lets you define the data sources that Grafana should use. In this tutorial, you will define a Loki data source. The data source is defined as follows:

 datasources:
   datasources.yaml:
     apiVersion: 1
     datasources:
     - name: Loki
       type: loki
       access: proxy
       orgId: 1
       url: http://loki-gateway.meta.svc.cluster.local:80
       basicAuth: false
       isDefault: false
       version: 1
       editable: false

This configuration defines a data source named Loki that Grafana will use to query logs stored in Loki. The url attribute specifies the URL of the Loki gateway. The Loki gateway is a service that sits in front of the Loki API and provides a single endpoint for ingesting and querying logs. The URL is in the format http://loki-gateway.<NAMESPACE>.svc.cluster.local:80. The loki-gateway service is created by the Loki Helm chart and is used to query logs stored in Loki. If you choose to deploy Loki in a different namespace or with a different name, you will need to update the url attribute accordingly.

Deploy the Kubernetes Monitoring Helm chart

The Kubernetes Monitoring Helm chart is used for gathering, scraping, and forwarding Kubernetes telemetry data to a Grafana stack. This includes the ability to collect metrics, logs, traces, and continuous profiling data. The scope of this tutorial is to deploy the Kubernetes Monitoring Helm chart to collect pod logs and Kubernetes events.

To deploy the Kubernetes Monitoring Helm chart run the following command:

helm install --values ./k8s-monitoring-values.yml k8s grafana/k8s-monitoring -n meta

Within the configuration file k8s-monitoring-values.yml we have defined the following:

---
cluster:
  name: meta-monitoring-tutorial

destinations:
  - name: loki
    type: loki
    url: http://loki-gateway.meta.svc.cluster.local/loki/api/v1/push


clusterEvents:
  enabled: true
  collector: alloy-logs
  namespaces:
    - meta
    - prod

nodeLogs:
  enabled: false

podLogs:
  enabled: true
  gatherMethod: kubernetesApi
  collector: alloy-logs
  labelsToKeep: ["app_kubernetes_io_name","container","instance","job","level","namespace","service_name","service_namespace","deployment_environment","deployment_environment_name"]
  structuredMetadata:
    pod: pod  # Set structured metadata "pod" from label "pod"
  namespaces:
    - meta
    - prod

# Collectors
alloy-singleton:
  enabled: false

alloy-metrics:
  enabled: false

alloy-logs:
  enabled: true
  # Required when using the Kubernetes API to pod logs
  alloy:
    mounts:
      varlog: false
    clustering:
      enabled: true

alloy-profiles:
  enabled: false

alloy-receiver:
  enabled: false

To break down the configuration file:

Define the cluster name as meta-monitoring-tutorial. This a static label that will be attached to all logs collected by the Kubernetes Monitoring Helm chart.
Define a destination named loki that will be used to forward logs to Loki. The url attribute specifies the URL of the Loki gateway. If you choose to deploy Loki in a different namespace or in a different location entirely, you will need to update the url attribute accordingly.
Enable the collection of cluster events and pod logs:
- collector: specifies which collector to use to collect logs. In this case, we are using the alloy-logs collector.
- labelsToKeep: specifies the labels to keep when collecting logs. Note this does not drop logs. This is useful when you do not want to apply a high cardanility label. In this case we have removed pod from the labels to keep.
- structuredMetadata: specifies the structured metadata to collect. In this case, we are setting the structured metadata pod so we can retain the pod name for querying. Though it does not need to be indexed as a label.zw
- namespaces: specifies the namespaces to collect logs from. In this case, we are collecting logs from the meta and prod namespaces.
Disable the collection of node logs for the purpose of this tutorial as it requires the mounting of /var/log/journal. This is out of scope for this tutorial.
Lastly, define the role of the collector. The Kubernetes Monitoring Helm chart will deploy only what you need and nothing more. In this case, we are telling the Helm chart to only deploy Alloy with the capability to collect logs. If you need to collect K8s metrics, traces, or continuous profiling data, you can enable the respective collectors.

Accessing Grafana

To access Grafana, you will need to port-forward the Grafana service to your local machine. To do this, run the following command:

export POD_NAME=$(kubectl get pods --namespace meta -l "app.kubernetes.io/name=grafana,app.kubernetes.io/instance=grafana" -o jsonpath="{.items[0].metadata.name}") && \
kubectl --namespace meta port-forward $POD_NAME 3000 --address 0.0.0.0

Tip
This will make your terminal unusable until you stop the port-forwarding process. To stop the process, press Ctrl + C.

This command will port-forward the Grafana service to your local machine on port 3000.

You can now access Grafana by navigating to http://localhost:3000 in your browser. The default credentials are admin and adminadminadmin.

One of the first places you should visit is Logs Drilldown which lets you automatically visualize and explore your logs without having to write queries: http://localhost:3000/a/grafana-lokiexplore-app

(Optional) View the Alloy UI

The Kubernetes Monitoring Helm chart deploys Grafana Alloy to collect and forward telemetry data from the Kubernetes cluster. The Helm chart is designed to abstract you away from creating an Alloy configuration file. However if you would like to understand the pipeline you can view the Alloy UI. To access the Alloy UI, you will need to port-forward the Alloy service to your local machine. To do this, run the following command:

export POD_NAME=$(kubectl get pods --namespace meta -l "app.kubernetes.io/name=alloy-logs,app.kubernetes.io/instance=k8s-alloy-logs" -o jsonpath="{.items[0].metadata.name}") && \
kubectl --namespace meta port-forward $POD_NAME 12345 --address 0.0.0.0

Tip
This will make your terminal unusable until you stop the port-forwarding process. To stop the process, press Ctrl + C.

This command will port-forward the Alloy service to your local machine on port 12345. You can access the Alloy UI by navigating to http://localhost:12345 in your browser.

Adding a sample application to `prod`

Finally, lets deploy a sample application to the prod namespace that will generate some logs. To deploy the sample application run the following command:

helm install tempo grafana/tempo-distributed -n prod

This will deploy a default version of Grafana Tempo to the prod namespace. Tempo is a distributed tracing backend that is used to store and query traces. Normally Tempo would sit alongside Loki and Grafana in the meta namespace, but for the purpose of this tutorial, we will pretend this is the primary application generating logs.

Once deployed lets expose Grafana once more:

export POD_NAME=$(kubectl get pods --namespace meta -l "app.kubernetes.io/name=grafana,app.kubernetes.io/instance=grafana" -o jsonpath="{.items[0].metadata.name}") && \
kubectl --namespace meta port-forward $POD_NAME 3000 --address 0.0.0.0

and navigate to http://localhost:3000/a/grafana-lokiexplore-app to view Grafana Tempo logs.

Conclusion

In this tutorial, you learned how to deploy Loki, Grafana, and the Kubernetes Monitoring Helm chart to collect and store logs from a Kubernetes cluster. We have deployed a minimal test version of each of these Helm charts to demonstrate how quickly you can get started with Loki. It is now worth exploring each of these Helm charts in more detail to understand how to scale them to meet your production needs:

Promtail agent

Thu, 09 Apr 2026 02:28:18 +0000

Promtail agent

Warning
Promtail is end of life (EOL) as of March 2, 2026. Commercial support has ended. No future support or updates will be provided. All future feature development will occur in Grafana Alloy.
Note that the deprecation of Promtail does NOT include the lambda-promtail client.

If you are currently using Promtail, you must migrate to Alloy or another supported client.

The Alloy migration documentation includes a migration tool for converting your Promtail configuration to an Alloy configuration with a single command.

Resources

Links to Migrate to Alloy documentation.
Medium post with migration instructions written by a Grafana Developer Advocate.
Video for Docker users with Promtail migration guide filmed by a Grafana Developer Advocate.

Docker driver client

Thu, 09 Apr 2026 02:28:18 +0000

Docker driver client

Grafana Loki officially supports a Docker plugin that will read logs from Docker containers and ship them to Loki. The plugin can be configured to send the logs to a private Loki instance or Grafana Cloud.

Note
Docker plugins are not supported on Windows; see the Docker Engine managed plugin system documentation for more information.

Documentation on configuring the Loki Docker Driver can be found on the configuration page. If you have any questions or issues using the Docker plugin, open an issue in the Loki repository.

Install the Docker driver client

The Docker plugin must be installed on each Docker host that will be running containers you want to collect logs from.

Run the following command to install the plugin, updating the release version, or changing the architecture (arm64 and amd64 are currently supported), if needed:

docker plugin install grafana/loki-docker-driver:3.7.0-arm64 --alias loki --grant-all-permissions

Note
Add -arm64 to the image tag for ARM64 hosts.

To check installed plugins, use the docker plugin ls command. Plugins that have started successfully are listed as enabled:

docker plugin ls

You should see output similar to the following:

ID                  NAME         DESCRIPTION           ENABLED
ac720b8fcfdb        loki         Loki Logging Driver   true

Once you have successfully installed the plugin you can configure it.

Upgrade the Docker driver client

The upgrade process involves disabling the existing plugin, upgrading (chaning version and architecture as needed), then re-enabling and restarting Docker:

docker plugin disable loki --force
docker plugin upgrade loki grafana/loki-docker-driver:3.7.0-arm64 --grant-all-permissions
docker plugin enable loki
systemctl restart docker

Note
Update the version number to the appropriate version.

Uninstall the Docker driver client

To cleanly uninstall the plugin, disable and remove it:

docker plugin disable loki --force
docker plugin rm loki

Known Issue: Deadlocked Docker Daemon

The driver keeps all logs in memory and will drop log entries if Loki is not reachable and if the quantity of max_retries has been exceeded. To avoid the dropping of log entries, setting max_retries to zero allows unlimited retries; the driver will continue trying forever until Loki is again reachable. Trying forever may have undesired consequences, because the Docker daemon will wait for the Loki driver to process all logs of a container, until the container is removed. Thus, the Docker daemon might wait forever if the container is stuck.

The wait time can be lowered by setting loki-retries=2, loki-max-backoff=800ms, loki-timeout=1s and keep-file=true. This way the daemon will be locked only for a short time and the logs will be persisted locally when the Loki client is unable to re-connect.

Also you can use non-blocking mode by setting services.logger.logging.options.mode=non-blocking in your docker-compose file. Non-blocking means that the process of writing logs to Loki will not block the main flow of an application or service if Loki is temporarily unavailable or unable to process log messages. In non-blocking mode, log messages will be buffered and sent to Loki asynchronously, which allows the main thread to continue working without delay. If Loki is unavailable, log messages will be stored in a buffer and sent when Loki becomes available again. However, this setting is useful to prevent blocking the main flow of an application or service due to logging issues, but it can also lead to loss of log messages if the buffer overflows or if Loki is unavailable for a long time.

To avoid this issue, use the Alloy loki.source.docker component or discovery.docker for Docker service discovery.

Fluent Bit

Thu, 09 Apr 2026 02:28:18 +0000

Fluent Bit

Fluent Bit is a fast, lightweight logs and metrics agent. It is a CNCF graduated sub-project under the umbrella of Fluentd. Fluent Bit is licensed under the terms of the Apache License v2.0.

When using Fluent Bit to ship logs to Loki, you can define which log files you want to collect using the Tail or Stdin data pipeline input. Additionally, Fluent Bit supports multiple Filter and Parser plugins (Kubernetes, JSON, etc.) to structure and alter log lines.

There are two Fluent Bit plugins for Loki:

The integrated loki plugin, which is officially maintained by the Fluent Bit project.
The grafana-loki plugin, an alternative community plugin by Grafana Labs.

We recommend using the loki plugin as this provides the most complete feature set and is actively maintained by the Fluent Bit project.

Tutorial

To get started with the loki plugin, follow the Sending logs to Loki using Fluent Bit tutorial.

Fluentd client

Thu, 09 Apr 2026 02:28:18 +0000

Fluentd client

The Fluentd output plugin for Grafana Loki is called fluent-plugin-grafana-loki that enables shipping logs to a private Loki instance or Grafana Cloud.

The plugin source code is in the fluentd directory of the repository.

Installation

Local

To install the plugin use fluent-gem:

fluent-gem install fluent-plugin-grafana-loki

Docker Image

The Docker image grafana/fluent-plugin-loki:main contains default configuration files. By default, fluentd containers use that default configuration. You can instead specify your fluentd.conf configuration file with a FLUENTD_CONF environment variable.

This image also uses LOKI_URL, LOKI_USERNAME, and LOKI_PASSWORD environment variables to specify the Loki endpoint, user, and password (you can leave the USERNAME and PASSWORD blank if they’re not used).

This image starts an instance of Fluentd that forwards incoming logs to the specified Loki URL. As an alternate, containerized applications can also use docker driver plugin to ship logs without needing Fluentd.

Example

A Docker Compose configuration that will work looks like:

services:
  fluentd:
    image: grafana/fluent-plugin-loki:main
    command:
      - "fluentd"
      - "-v"
      - "-p"
      - "/fluentd/plugins"
    environment:
      LOKI_URL: http://loki:3100
      LOKI_USERNAME:
      LOKI_PASSWORD:
    deploy:
      mode: global
    configs:
      - source: loki_config
        target: /fluentd/etc/loki/loki.conf
    networks:
      - loki
    volumes:
      - host_logs:/var/log
      # Needed for journald log ingestion:
      - /etc/machine-id:/etc/machine-id
      - /dev/log:/dev/log
      - /var/run/systemd/journal/:/var/run/systemd/journal/
    logging:
      options:
        tag: infra.monitoring

Usage

Note
Use either <label>...</label> or extra_labels to set at least one label.

In your Fluentd configuration, add @type loki. Additional configuration is optional. Default values would look like this:

<match **>
  @type loki
  url "https://logs-prod-us-central1.grafana.net"
  username "#{ENV['LOKI_USERNAME']}"
  password "#{ENV['LOKI_PASSWORD']}"
  extra_labels {"env":"dev"}
  <buffer>
    flush_interval 10s
    flush_at_shutdown true
  </buffer>
  buffer_chunk_limit 1m
</match>

Adding labels

Simple label from top level attribute

<match mytag>
  @type loki
  # ...
  <label>
    fluentd_worker
  </label>
  # ...
</match>

You can rewrite the label keys as well as the following

<match mytag>
  @type loki
  # ...
  <label>
    worker fluentd_worker
  </label>
  # ...
</match>

You can use record accessor syntax for nested field.

<match mytag>
  @type loki
  # ...
  <label>
    container $.kubernetes.container
  </label>
  # ...
</match>

Extracting Kubernetes labels

Since Kubernetes labels are a list of nested key-value pairs, a separate option is available to extract them. Note that special characters like “. - /” will be overwritten with _. Use with the remove_keys kubernetes option to eliminate metadata from the log.

<match mytag>
  @type loki
  # ...
  extract_kubernetes_labels true
  remove_keys kubernetes
  <label>
    container $.kubernetes.container
  </label>
  # ...
</match>

You can also include automatically all kubernetes labels by using extract_kubernetes_labels true in your configuration.

Multi-worker usage

Loki enables out-of-order inserts by default; refer to accept out-of-order writes. If out-of-order inserts are disabled, attempting to insert a log entry with an earlier timestamp after a log entry with identical labels but a later timestamp, the insert will fail with HTTP status code: 500, message: rpc error: code = Unknown desc = Entry out of order. Therefore, in order to use this plugin in a multi worker Fluentd setup, you’ll need to include the worker ID in the labels or otherwise ensure log streams are always sent to the same worker.

For example, using fluent-plugin-record-modifier:

<filter mytag>
    @type record_modifier
    <record>
        fluentd_worker "#{worker_id}"
    </record>
</filter>

<match mytag>
  @type loki
  # ...
  <label>
    fluentd_worker
  </label>
  # ...
</match>

Using multiple buffer flush threads

Similarly, when using flush_thread_count > 1 in the buffer section, a thread identifier must be added as a label to ensure that log chunks flushed in parallel to loki by fluentd always have increasing times for their unique label sets.

This plugin automatically adds a fluentd_thread label with the name of the buffer flush thread when flush_thread_count > 1.

Configuration

`url`

The URL of the Loki server to send logs to. When sending data, the publish path (../reference/api/loki-http-api/v1/push/) will automatically be appended. By default the url is set to https://logs-prod-us-central1.grafana.net, the url of the Grafana Labs hosted Loki service.

Proxy Support

Starting with version 0.8.0, this gem uses excon, which supports proxy with environment variables.

`username` / `password`

If the Loki server requires authentication, specify a username and password. If using the GrafanaLab’s hosted Loki, the username needs to be set to your instanceId and the password should be a Grafana.com api key.

tenant

All requests sent to Loki, a multi-tenant log storage platform, must include a tenant. For some installations the tenant will be set automatically by an authenticating proxy. Otherwise you can define a tenant to be passed through. The tenant can be any string value.

The tenant field also supports placeholders, allowing it to dynamically change based on tag and record fields. Each placeholder must be added as a buffer chunk key. The following is an example of setting the tenant based on a Kubernetes pod label:

<match **>
  @type loki
  url "https://logs-prod-us-central1.grafana.net"
  tenant ${$.kubernetes.labels.tenant}
  # ...
  <buffer $.kubernetes.labels.tenant>
    @type memory
    flush_interval 5s
  </buffer>
</match>

Client certificate verification

If a reverse proxy with client certificate verification is configured in front of Loki, specify a pair of client certificate and private key with cert and key. ca_cert can also be specified if the server uses custom certificate authority.

<match **>
  @type loki

  url "https://loki"

  cert /path/to/certificate.pem
  key /path/to/key.key
  ca_cert /path/to/ca.pem

  ...
</match>

Server certificate verification

A flag to disable server certificate verification. By default the insecure_tls is set to false.

<match **>
  @type loki

  url "https://loki"

  insecure_tls true

  ...
</match>

compress

Enable compression for the HTTP request body. Supported values: gzip. Not compressed by default.

<match **>
  @type loki

  url "https://loki"
  compress gzip

  ...
</match>

Output format

Loki is intended to index and group log streams using only a small set of labels. It is not intended for full-text indexing. When sending logs to Loki the majority of log message will be sent as a single log “line”.

Several configuration settings are available to control the output format.

extra_labels: (default: nil) set of labels to include with every Loki stream. eg {"env":"dev", "datacenter": "dc1"}
remove_keys: (default: nil) comma separated list of needless record keys to remove. All other keys will be placed into the log line. You can use record_accessor syntax.
line_format (default:key_value): format to use when flattening the record to a log line. Valid values are “json” or “key_value”. If set to “json” the log line sent to Loki will be the fluentd record (excluding any keys extracted out as labels) dumped as json. If set to “key_value”, the log line will be each item in the record concatenated together (separated by a single space) in the format <key>=<value>.
drop_single_key: if set to true and a record only has 1 key after extracting <label></label> blocks, set the log line to the value and discard the key.
include_thread_label (default: true): whether or not to include the fluentd_thread label when multiple threads are used for flushing.

Buffer options

fluentd-plugin-loki extends Fluentd’s builtin Output plugin and use compat_parameters plugin helper. It adds the following options:

buffer_type memory
flush_interval 10s
retry_limit 17
retry_wait 1.0
num_threads 1

Lambda Promtail client

Thu, 09 Apr 2026 02:28:18 +0000

Lambda Promtail client

Grafana Loki includes Terraform and CloudFormation for shipping Cloudwatch, Cloudtrail, VPC Flow Logs and loadbalancer logs to Loki via a lambda function. This is done via lambda-promtail which processes cloudwatch events and propagates them to Loki (or a Promtail instance) via the push-api scrape config.

Deployment

lambda-promtail can easily be deployed via provided Terraform and CloudFormation files. The Terraform deployment also pulls variable values defined from variables.tf.

For both deployment types there are a few values that must be defined:

the write address, a Loki Write API compatible endpoint (Loki or Promtail)
basic auth username/password if the write address is a Loki endpoint and has authentication
the lambda-promtail image, full ECR repo path:tag

The Terraform deployment also takes in an array of log group and bucket names, and can take arrays for VPC subnets and security groups.

There’s also a flag to keep the log stream label when propagating the logs from Cloudwatch, which defaults to false. This can be helpful when the cardinality is too large, such as the case of a log stream per lambda invocation.

Additionally, an environment variable can be configured to add extra labels to the logs streamed by lambda-promtail. These extra labels will take the form __extra_<name>=<value>.

An optional environment variable can be configured to add the tenant ID to the logs streamed by lambda-promtail.

In an effort to make deployment of lambda-promtail as simple as possible, we’ve created a public ECR repo to publish our builds of lambda-promtail. Users may clone this repo, make their own modifications to the Go code, and upload their own image to their own ECR repo.

Examples

Terraform:

## use cloudwatch log group
terraform apply -var "lambda_promtail_image=<repo:tag>" -var "write_address=https://logs-prod-us-central1.grafana.net/loki/api/v1/push" -var "password=<password>" -var "username=<user>" -var 'log_group_names=["/aws/lambda/log-group-1", "/aws/lambda/log-group-2"]' -var 'bucket_names=["bucket-a", "bucket-b"]' -var 'batch_size=131072'

## use kinesis data stream
terraform apply -var "<ecr-repo>:<tag>" -var "write_address=https://your-loki-url/loki/api/v1/push" -var "password=<basic-auth-pw>" -var "username=<basic-auth-username>" -var 'kinesis_stream_name=["kinesis-stream-01", "kinesis-stream-02"]' -var 'extra_labels="name1,value1,name2,value2"' -var "tenant_id=<value>"

The first few lines of main.tf define the AWS region to deploy to. Modify as desired, or remove and deploy to

provider "aws" {
  region = "us-east-2"
}

To keep the log group label add -var "keep_stream=true".

To add extra labels add -var 'extra_labels="name1,value1,name2,value2"'.

To add tenant id add -var "tenant_id=value".

Note that the creation of a subscription filter on Cloudwatch in the provided Terraform file only accepts an array of log group names. It does not accept strings for regex filtering on the logs contents via the subscription filters. We suggest extending the Terraform file to do so. Or, have lambda-promtail write to Promtail and use pipeline stages.

CloudFormation:

aws cloudformation create-stack --stack-name lambda-promtail --template-body file://template.yaml --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM --region us-east-2 --parameters ParameterKey=WriteAddress,ParameterValue=https://logs-prod-us-central1.grafana.net/loki/api/v1/push ParameterKey=Username,ParameterValue=<user> ParameterKey=Password,ParameterValue=<password> ParameterKey=LambdaPromtailImage,ParameterValue=<repo:tag>

Within the CloudFormation template file, copy, paste, and modify the subscription filter section as needed for each log group:

MainLambdaPromtailSubscriptionFilter:
  Type: AWS::Logs::SubscriptionFilter
  Properties:
    DestinationArn: !GetAtt LambdaPromtailFunction.Arn
    FilterPattern: ""
    LogGroupName: "/aws/lambda/some-lambda-log-group"

To keep the log group label, add ParameterKey=KeepStream,ParameterValue=true.

To add extra labels, include ParameterKey=ExtraLabels,ParameterValue="name1,value1,name2,value2".

To add a tenant ID, add ParameterKey=TenantID,ParameterValue=value.

To modify an existing CloudFormation stack, use update-stack.

If using CloudFormation to write your infrastructure code, you should consider the EventBridge based solution for easier deployment.

Uses

Ephemeral Jobs

This workflow is intended to be an effective approach for monitoring ephemeral jobs such as those run on AWS Lambda which are otherwise hard/impossible to monitor via one of the other Loki clients.

Ephemeral jobs can quite easily run afoul of cardinality best practices. During high request load, an AWS lambda function might balloon in concurrency, creating many log streams in Cloudwatch. For this reason lambda-promtail defaults to not keeping the log stream value as a label when propagating the logs to Loki. This is only possible because new versions of Loki no longer have an ingestion ordering constraint on logs within a single stream.

Proof of concept Loki deployments

For those using Cloudwatch and wishing to test out Loki in a low-risk way, this workflow allows piping Cloudwatch logs to Loki regardless of the event source (EC2, Kubernetes, Lambda, ECS, etc) without setting up a set of Promtail daemons across their infrastructure. However, running Promtail as a daemon on your infrastructure is the best-practice deployment strategy in the long term for flexibility, reliability, performance, and cost.

Note
Propagating logs from Cloudwatch to Loki means you’ll still need to pay for Cloudwatch.

VPC Flow logs

This workflow allows ingesting AWS VPC Flow logs from s3.

One thing to be aware of with this is that the default flow log format doesn’t have a timestamp, so the log timestamp will be set to the time the lambda starts processing the log file.

Loadbalancer logs

This workflow allows ingesting AWS Application/Network Load Balancer logs stored on S3 to Loki.

Cloudtrail logs

This workflow allows ingesting AWS Cloudtrail logs stored on S3 to Loki.

Cloudfront logs

Cloudfront logs can be either batched or streamed in real time to Loki:

Logging can be activated on a Cloudfront distribution with an S3 bucket as the destination. In this case, the workflow is the same as for other services (VPC Flow logs, Loadbalancer logs, Cloudtrail logs).
Cloudfront real-time logs can be sent to a Kinesis data stream. The data stream can be mapped to be an event source for lambda-promtail to deliver the logs to Loki.

Triggering Lambda-Promtail via SQS

For AWS services supporting sending messages to SQS (for example, S3 with an S3 Notification to SQS), events can be processed through an SQS queue using a lambda trigger instead of directly configuring the source service to trigger lambda. Lambda-promtail will retrieve the nested events from the SQS messages’ body and process them as if them came directly from the source service.

On-Failure log recovery using SQS

Triggering lambda-promtail through SQS allows handling on-failure recovery of the logs using a secondary SQS queue as a dead-letter-queue (DLQ). You can configure lambda so that unsuccessfully processed messages will be sent to the DLQ. After fixing the issue, operators will be able to reprocess the messages by sending back messages from the DLQ to the source queue using the SQS DLQ redrive feature.

S3 based logging and CloudFormation

Lambda-promtail lets you send logs from different services that use S3 as their logs destination (ALB, VPC Flow, CloudFront access logs, etc.). For this, you need to configure S3 bucket notifications to trigger the lambda-promtail deployment. However, when using CloudFormation to encode infrastructure, there is a known issue when configuring AWS::S3::BucketNotification and the resource that will be triggered by the notification in the same stack.

To manage this issue, AWS introduced S3 event notifications with Event Bridge. When an object gets created in a S3 bucket, this sends an event to an EventBridge bus, and you can create a rule to send those events to Lambda-promtail.

The diagram below shows how notifications logs will be written from the source service into an S3 bucket. From there on, the S3 bucket will send an Object created notification into the EventBridge default bus, where we can configure a rule to trigger Lambda Promtail.

The template-eventbridge.yaml CloudFormation template configures Lambda-promtail with EventBridge to address this known issue. To deploy the template, use the snippet below, completing appropriately the ParameterValue arguments.

aws cloudformation create-stack \
  --stack-name lambda-promtail-stack \
  --template-body file://template-eventbridge.yaml \
  --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM \
  --region us-east-2 \
  --parameters ParameterKey=WriteAddress,ParameterValue=https://your-loki-url/loki/api/v1/push ParameterKey=Username,ParameterValue=<basic-auth-username> ParameterKey=Password,ParameterValue=<basic-auth-pw> ParameterKey=BearerToken,ParameterValue=<bearer-token> ParameterKey=LambdaPromtailImage,ParameterValue=<ecr-repo>:<tag> ParameterKey=ExtraLabels,ParameterValue="name1,value1,name2,value2" ParameterKey=TenantID,ParameterValue=<value> ParameterKey=SkipTlsVerify,ParameterValue="false" ParameterKey=EventSourceS3Bucket,ParameterValue=<S3 where target logs are stored>

Propagated Labels

Incoming logs can have seven special labels assigned to them which can be used in relabeling or later stages in a Promtail pipeline:

__aws_log_type: Where this log came from (Cloudwatch, Kinesis or S3).
__aws_cloudwatch_log_group: The associated Cloudwatch Log Group for this log.
__aws_cloudwatch_log_stream: The associated Cloudwatch Log Stream for this log (if KEEP_STREAM=true).
__aws_cloudwatch_owner: The AWS ID of the owner of this event.
__aws_kinesis_event_source_arn: The Kinesis event source ARN.
__aws_s3_log_lb: The name of the loadbalancer.
__aws_s3_log_lb_owner: The Account ID of the loadbalancer owner.

Relabeling Configuration

Lambda-promtail supports Prometheus-style relabeling through the RELABEL_CONFIGS environment variable. This allows you to modify, keep, or drop labels before sending logs to Loki. The configuration is provided as a JSON array of relabel configurations. The relabeling functionality follows the same principles as Prometheus relabeling - for a detailed explanation of how relabeling works, see How relabeling in Prometheus works.

Example configurations:

Rename a label and capture regex groups:

{
  "RELABEL_CONFIGS": [
    {
      "source_labels": ["__aws_log_type"],
      "target_label": "log_type",
      "action": "replace",
      "regex": "(.*)",
      "replacement": "${1}"
    }
  ]
}

Keep only specific log types (useful for filtering):

{
  "RELABEL_CONFIGS": [
    {
      "source_labels": ["__aws_log_type"],
      "regex": "s3_.*",
      "action": "keep"
    }
  ]
}

Drop internal AWS labels (cleanup):

{
  "RELABEL_CONFIGS": [
    {
      "regex": "__aws_.*",
      "action": "labeldrop"
    }
  ]
}

Multiple relabeling rules (combining different actions):

{
  "RELABEL_CONFIGS": [
    {
      "source_labels": ["__aws_log_type"],
      "target_label": "log_type",
      "action": "replace",
      "regex": "(.*)",
      "replacement": "${1}"
    },
    {
      "source_labels": ["__aws_s3_log_lb"],
      "target_label": "loadbalancer",
      "action": "replace"
    },
    {
      "regex": "__aws_.*",
      "action": "labeldrop"
    }
  ]
}

Supported Actions

The following actions are supported, matching Prometheus relabeling capabilities:

replace: Replace a label value with a new value using regex capture groups
keep: Keep entries where labels match the regex (useful for filtering)
drop: Drop entries where labels match the regex (useful for excluding)
hashmod: Set a label to the modulus of a hash of labels (useful for sharding)
labelmap: Copy labels to other labels based on regex matching
labeldrop: Remove labels matching the regex pattern
labelkeep: Keep only labels matching the regex pattern
lowercase: Convert label values to lowercase
uppercase: Convert label values to uppercase

Configuration Fields

Each relabel configuration supports these fields (all fields are optional except for action):

source_labels: List of label names to use as input for the action
separator: String to join source label values (default: “;”)
target_label: Label to modify (required for replace and hashmod actions)
regex: Regular expression to match against (defaults to “(.+)” for most actions)
replacement: Replacement pattern for matched regex, supports ${1}, ${2}, etc. for capture groups
modulus: Modulus for hashmod action
action: One of the supported actions listed above

Important Notes

Relabeling is applied after merging extra labels and dropping labels specified by DROP_LABELS.
If all labels are removed after relabeling, the log entry will be dropped entirely.
The relabeling configuration follows the same format as Prometheus’s relabel_configs, making it familiar for users of Prometheus.
Relabeling rules are processed in order, and each rule can affect the input of subsequent rules.
Regular expressions in the regex field support full RE2 syntax.
For the replace action, if the regex doesn’t match, the target label remains unchanged.

For more details about how relabeling works and advanced use cases, refer to the Prometheus relabeling blog post.

Limitations

Promtail labels

Note
This section is relevant if running Promtail between lambda-promtail and the end Loki deployment and was used to circumvent out of order problems prior to the v2.4 Loki release which removed the ordering constraint.

As stated earlier, this workflow moves the worst case stream cardinality from number_of_log_streams -> number_of_log_groups * number_of_promtails. For this reason, each Promtail must have a unique label attached to logs it processes (ideally via something like --client.external-labels=promtail=${HOSTNAME}) and it’s advised to run a small number of Promtails behind a load balancer according to your throughput and redundancy needs.

This trade-off is very effective when you have a large number of log streams but want to aggregate them by the log group. This is very common in AWS Lambda, where log groups are the “application” and log streams are the individual application containers which are spun up and down at a whim, possibly just for a single function invocation.

Data Persistence

Availability

For availability concerns, run a set of Promtails behind a load balancer.

Batching

Relevant if lambda-promtail is configured to write to Promtail. Since Promtail batches writes to Loki for performance, it’s possible that Promtail will receive a log, issue a successful 204 http status code for the write, then be killed at a later time before it writes upstream to Loki. This should be rare, but is a downside this workflow has.

This lambda will flush logs when the batch size hits the default value of 131072 (128KB), this can be changed with BATCH_SIZE environment variable, which is set to the number of bytes to use.

Templating/Deployment

The current CloudFormation template is rudimentary. If you need to add vpc configs, extra log groups to monitor, subnet declarations, etc, you’ll need to edit the template manually. If you need to subscribe to more than one Cloudwatch Log Group you’ll also need to copy paste that section of the template for each group.

The Terraform file is a bit more fleshed out, and can be configured to take in an array of log group and bucket names, as well as vpc configuration.

The provided Terraform and CloudFormation files are meant to cover the default use case, and more complex deployments will likely require some modification and extension of the provided files.

Example Promtail Config

Note
This should be run in conjunction with a Promtail-specific label attached, ideally via a flag argument like --client.external-labels=promtail=${HOSTNAME}. It will receive writes via the push-api on ports 3500 (http) and 3600 (grpc).

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://ip_or_hostname_where_Loki_run:3100/loki/api/v1/push

scrape_configs:
  - job_name: push1
    loki_push_api:
      server:
        http_listen_port: 3500
        grpc_listen_port: 3600
      labels:
        # Adds a label on all streams indicating it was processed by the lambda-promtail workflow.
        promtail: 'lambda-promtail'
    relabel_configs:
      - source_labels: ['__aws_log_type']
        target_label: 'log_type'
      # Maps the cloudwatch log group into a label called `log_group` for use in Loki.
      - source_labels: ['__aws_cloudwatch_log_group']
        target_label: 'log_group'
      # Maps the loadbalancer name into a label called `loadbalancer_name` for use in Loki.
      - source_labels: ['__aws_s3_log_lb']
        target_label: 'loadbalancer_name'

Multiple Promtail Deployment

Disclaimer: The following section is only relevant for older versions of Loki that cannot accept out of order logs.

However, these may only be active for a very short while. This creates a problem for combining these short-lived log streams in Loki because timestamps may not strictly increase across multiple log streams. The other obvious route is creating labels based on log streams, which is also undesirable because it leads to cardinality problems via many low-throughput log streams.

Instead we can pipeline Cloudwatch logs to a set of Promtails, which can mitigate these problem in two ways:

Using Promtail’s push api along with the use_incoming_timestamp: false config, we let Promtail determine the timestamp based on when it ingests the logs, not the timestamp assigned by cloudwatch. Obviously, this means that we lose the origin timestamp because Promtail now assigns it, but this is a relatively small difference in a real time ingestion system like this.
In conjunction with (1), Promtail can coalesce logs across Cloudwatch log streams because it’s no longer susceptible to out-of-order errors when combining multiple sources (lambda invocations).

One important aspect to keep in mind when running with a set of Promtails behind a load balancer is that we’re effectively moving the cardinality problems from the number of log streams -> number of Promtails. If you have not configured Loki to accept out-of-order writes, you’ll need to assign a Promtail-specific label on each Promtail so that you don’t run into out-of-order errors when the Promtails send data for the same log groups to Loki. This can easily be done via a configuration like --client.external-labels=promtail=${HOSTNAME} passed to Promtail.

Logstash plugin

Thu, 09 Apr 2026 02:28:18 +0000

Logstash plugin

Grafana Loki has a Logstash output plugin called logstash-output-loki that enables shipping logs to a Loki instance or Grafana Cloud.

Warning
Grafana Labs does not recommend using the Logstash plugin for new deployments. Even as a mechanism for quickly testing Loki with your existing Beats/Logstash infrastructure we highly discourage the use of this plugin.

Our experience over the years has found numerous significant challenges using Logstash and this plugin:

It is very difficult to configure labels correctly. Conceptually Elasticsearch is a very different database from Loki and users almost always end up sending too many high cardinality labels to Loki, which makes getting started with Loki unnecessarily complicated and confusing vs. using other clients.

Logstash and the upstream Beats components implement backoff and flow control which we’ve found hard to observe, leading to ingestion delays into Loki which are extremely difficult to address.

We at Grafana Labs have no expertise at configuring Logstash or understanding of its configuration language, so we cannot provide support for it.

It’s very hard to troubleshoot and debug. Our experience has shown that in nearly every case where it was assumed this would be the fast path to getting logs to Loki, that was not the case and it ended up taking far longer than anticipated.

Please strongly consider using any alternative mechanism to sending logs to Loki. We recommend using Grafana Alloy. This is the tool we build and where we can offer the best experience and most support.

Installation

Local

If you need to install the Logstash output plugin manually you can do simply so by using the command below:

$ bin/logstash-plugin install logstash-output-loki

This will download the latest gem for the output plugin and install it in logstash.

Docker

We also provide a docker image on docker hub. The image contains logstash and the Loki output plugin already pre-installed.

For example if you want to run logstash in docker with the loki.conf as pipeline configuration you can use the command bellow :

docker run -v `pwd`/loki-test.conf:/home/logstash/ --rm grafana/logstash-output-loki:1.0.1 -f loki-test.conf

Kubernetes

We also provide default helm values for scraping logs with Filebeat and forward them to Loki with logstash in our loki-stack umbrella chart. You can switch to logstash by using the following command:

helm upgrade --install loki loki/loki-stack \
    --set filebeat.enabled=true,logstash.enabled=true,promtail.enabled=false \
    --set loki.fullnameOverride=loki,logstash.fullnameOverride=logstash-loki

This will automatically scrape all pods logs in the cluster and send them to Loki with Kubernetes metadata attached as labels. You can use the values.yaml file as a starting point for your own configuration.

Usage and Configuration

To configure Logstash to forward logs to Loki, simply add the loki output to your Logstash configuration file as documented below :

output {
  loki {
    [url => "" | default = none | required=true]

    [tenant_id => string | default = nil | required=false]

    [message_field => string | default = "message" | required=false]

    [include_fields => array | default = [] | required=false]

    [metadata_fields => array | default = [] | required=false]

    [batch_wait => number | default = 1(s) | required=false]

    [batch_size => number | default = 102400(bytes) | required=false]

    [min_delay => number | default = 1(s) | required=false]

    [max_delay => number | default = 300(s) | required=false]

    [retries => number | default = 10 | required=false]

    [username => string | default = nil | required=false]

    [password => secret | default = nil | required=false]

    [cert => path | default = nil | required=false]

    [key => path | default = nil| required=false]

    [ca_cert => path | default = nil | required=false]

    [insecure_skip_verify => boolean | default = false | required=false]
  }
}

By default Loki will create entry from event fields it receives. A logstash event as shown below.

{
  "@timestamp" => 2017-04-26T19:33:39.257Z,
  "src"        => "localhost",
  "@version"   => "1",
  "host"       => "localhost.localdomain",
  "pid"        => "1",
  "message"    => "Apr 26 12:20:02 localhost systemd[1]: Starting system activity accounting tool...",
  "type"       => "stdin",
  "prog"       => "systemd",
}

Contains a message and @timestamp fields, which are respectively used to form the Loki entry log line and timestamp.

You can use a different property for the log line by using the configuration property message_field. If you also need to change the timestamp value use the Logstash date filter to change the @timestamp field.

All other fields (except nested fields) will form the label set (key value pairs) attached to the log line. This means you’re responsible for mutating and dropping high cardinality labels such as client IPs. You can usually do so by using a mutate filter.

For example the configuration below :

input {
  ...
}

filter {
  mutate {
    add_field => {
      "cluster" => "us-central1"
      "job" => "logstash"
    }
    replace => { "type" => "stream"}
    remove_field => ["src"]
  }
}
output {
  loki {
    url => "http://myloki.domain:3100/loki/api/v1/push"
  }
}

Will add cluster and job static labels, remove src fields and replace type to be named stream.

If you want to include nested fields or metadata fields (starting with @) you need to rename them.

For example when using Filebeat with the add_kubernetes_metadata processor, it will attach Kubernetes metadata to your events like below:

{
  "kubernetes" : {
    "labels" : {
      "app" : "MY-APP",
      "pod-template-hash" : "959f54cd",
      "serving" : "true",
      "version" : "1.0",
      "visualize" : "true"
    },
    "pod" : {
      "uid" : "e20173cb-3c5f-11ea-836e-02c1ee65b375",
      "name" : "MY-APP-959f54cd-lhd5p"
    },
    "node" : {
      "name" : "ip-xxx-xx-xx-xxx.ec2.internal"
    },
    "container" : {
      "name" : "istio"
    },
    "namespace" : "production",
    "replicaset" : {
      "name" : "MY-APP-959f54cd"
    }
  },
  "message": "Failed to parse configuration",
  "@timestamp": "2017-04-26T19:33:39.257Z",
}

The filter below show you how to extract those Kubernetes fields into labels (container_name,namespace,pod and host):

filter {
  if [kubernetes] {
    mutate {
      add_field => {
        "container_name" => "%{[kubernetes][container][name]}"
        "namespace" => "%{[kubernetes][namespace]}"
        "pod" => "%{[kubernetes][pod][name]}"
      }
      replace => { "host" => "%{[kubernetes][node][name]}"}
    }
  }
  mutate {
    remove_field => ["tags"]
  }
}

Version Notes

Important notes regarding versions:

Version 1.1.0 and greater of this plugin you can also specify a list of labels to allow list via the include_fields configuration.
Version 1.2.0 and greater of this plugin you can also specify structured metadata via the metadata_fields configuration.

Configuration Properties

url

The url of the Loki server to send logs to. When sending data the push path need to also be provided e.g. http://localhost:3100/loki/api/v1/push.

If you want to send to GrafanaCloud you would use https://logs-prod-us-central1.grafana.net/loki/api/v1/push.

username / password

Specify a username and password if the Loki server requires basic authentication. If using the GrafanaLab’s hosted Loki, the username needs to be set to your instance/user id and the password should be a Grafana.com api key.

message_field

Message field to use for log lines. You can use logstash key accessor language to grab nested property, for example : [log][message].

include_fields

An array of fields which will be mapped to labels and sent to Loki, when this list is configured only these fields will be sent, all other fields will be ignored.

metadata_fields

An array of fields which will be mapped to structured metadata and sent to Loki for each log line

batch_wait

Interval in seconds to wait before pushing a batch of records to Loki. This means even if the batch size is not reached after batch_wait a partial batch will be sent, this is to ensure freshness of the data.

batch_size

Maximum batch size to accrue before pushing to Loki. Defaults to 102400 bytes

Backoff config

min_delay => 1(1s)

Initial backoff time between retries

max_delay => 300(5m)

Maximum backoff time between retries

retries => 10

Maximum number of retries to do. Setting it to 0 will retry indefinitely.

tenant_id

Loki is a multi-tenant log storage platform and all requests sent must include a tenant. For some installations the tenant will be set automatically by an authenticating proxy. Otherwise you can define a tenant to be passed through. The tenant can be any string value.

client certificate verification

Specify a pair of client certificate and private key with cert and key if a reverse proxy with client certificate verification is configured in front of Loki. ca_cert can also be specified if the server uses custom certificate authority.

insecure_skip_verify

A flag to disable server certificate verification. By default it is set to false.

Full configuration example

input {
  beats {
    port => 5044
  }
}

filter {
  if [kubernetes] {
    mutate {
      add_field => {
        "container_name" => "%{[kubernetes][container][name]}"
        "namespace" => "%{[kubernetes][namespace]}"
        "pod" => "%{[kubernetes][pod][name]}"
      }
      replace => { "host" => "%{[kubernetes][node][name]}"}
    }
  }
  mutate {
    remove_field => ["tags"]  # Note: with include_fields defined below this wouldn't be necessary
  }
}

output {
  loki {
    url => "https://logs-prod-us-central1.grafana.net/loki/api/v1/push"
    username => "3241"
    password => "REDACTED"
    batch_size => 112640 #112.64 kilobytes
    retries => 5
    min_delay => 3
    max_delay => 500
    message_field => "message"
    include_fields => ["container_name","namespace","pod","host"]
    metadata_fields => ["pod"]
  }
  # stdout { codec => rubydebug }
}

Using k6 for load testing

Thu, 09 Apr 2026 02:28:18 +0000

Using k6 for load testing

Grafana k6 is a modern load-testing tool. Its clean and approachable scripting API works locally or in the cloud. Its configuration makes it flexible.

The xk6-loki extension permits pushing logs to and querying logs from a Loki instance. It acts as a Loki client, simulating real-world load to test the scalability, reliability, and performance of your Loki installation.

Before you begin

k6 is written in Golang. Download and install a Go environment.

Installation

xk6-loki is an extension to the k6 binary. Build a custom k6 binary that includes the xk6-loki extension.

Install the xk6 extension bundler:
Bash
```
go install go.k6.io/xk6/cmd/xk6@latest
```

Check out the grafana/xk6-loki repository:

git clone https://github.com/grafana/xk6-loki
cd xk6-loki

Build k6 with the extension:
Bash
```
make k6
```

Usage

Use the custom-built k6 binary in the same way as a non-custom k6 binary:

./k6 run test.js

test.js is a Javascript load test. Refer to the k6 documentation to get started.

Scripting API

The custom-built k6 binary provides a Javascript loki module.

Your Javascript load test imports the module:

import loki from 'k6/x/loki';

Classes of this module are:

class	description
`Config`	configuration for the `Client` class
`Client`	client for writing and reading logs from Loki

Config and Client must be called on the k6 init context (see Test life cycle) outside of the default function so the client is only configured once and shared between all VU iterations.

The Client class exposes the following instance methods:

method	description
`push()`	shortcut for `pushParameterized(5, 8001024, 10241024)`
`pushParameterized(streams, minSize, maxSize)`	execute push request ( POST /loki/api/v1/push
`instantQuery(query, limit)`	execute instant query ( GET /loki/api/v1/query
`client.rangeQuery(query, duration, limit)`	execute range query ( GET /loki/api/v1/query_range
`client.labelsQuery(duration)`	execute labels query ( GET /loki/api/v1/labels
`client.labelValuesQuery(label, duration)`	execute label values query ( GET /loki/api/v1/label/<name>/values
`client.seriesQuery(matchers, duration)`	execute series query ( GET /loki/api/v1/series

Javascript load test example:

import loki from 'k6/x/loki';

const timeout = 5000; // ms
const conf = loki.Config("http://localhost:3100", timeout);
const client = loki.Client(conf);

export default () => {
   client.pushParameterized(2, 512*1024, 1024*1024);
};

Refer to grafana/xk6-loki for the complete k6/x/loki module API reference.

Send log data to Loki on Grafana Labs

Ingesting logs to Loki using Alloy

Ingesting logs to Loki using Alloy

Installing Alloy

Components of Alloy for logs

Log components in Alloy

Learning path

Start your learning experience with Grafana Learning Paths

Interactive Tutorials

Ingesting logs to Loki using OpenTelemetry Collector

Ingesting logs to Loki using OpenTelemetry Collector

Loki configuration

Configure the OpenTelemetry Collector to write logs into Loki

Format considerations

Changing the default mapping of OTLP to Loki Format

Example 1

Example 2

Example 3

Kubernetes Monitoring Helm tutorial

Kubernetes Monitoring Helm tutorial

Things to know

Prerequisites

Create the meta and prod namespaces

Add the Grafana Helm repository

Clone the tutorial repository

Deploy Loki

Deploy Grafana

Deploy the Kubernetes Monitoring Helm chart

Accessing Grafana

(Optional) View the Alloy UI

Adding a sample application to prod

Conclusion

Promtail agent

Promtail agent

Resources

Docker driver client

Docker driver client

Install the Docker driver client

Upgrade the Docker driver client

Uninstall the Docker driver client

Known Issue: Deadlocked Docker Daemon

Fluent Bit

Fluent Bit

Tutorial

Fluentd client

Fluentd client

Installation

Local

Docker Image

Example

Usage

Adding labels

Extracting Kubernetes labels

Multi-worker usage

Using multiple buffer flush threads

Configuration

url

Proxy Support

username / password

tenant

Client certificate verification

Server certificate verification

compress

Output format

Buffer options

Lambda Promtail client

Lambda Promtail client

Deployment

Examples

Uses

Ephemeral Jobs

Proof of concept Loki deployments

VPC Flow logs

Loadbalancer logs

Cloudtrail logs

Cloudfront logs

Triggering Lambda-Promtail via SQS

On-Failure log recovery using SQS

S3 based logging and CloudFormation

Propagated Labels

Create the `meta` and `prod` namespaces

Adding a sample application to `prod`

`url`

`username` / `password`