Clients on Grafana Labs

Promtail

Tue, 16 Jul 2024 15:42:20 +0000

Promtail

Promtail is an agent which ships the contents of local logs to a private Grafana Loki instance or Grafana Cloud. It is usually deployed to every machine that has applications needed to be monitored.

It primarily:

Discovers targets
Attaches labels to log streams
Pushes them to the Loki instance.

Currently, Promtail can tail logs from two sources: local log files and the systemd journal (on AMD64 machines only).

Log file discovery

Before Promtail can ship any data from log files to Loki, it needs to find out information about its environment. Specifically, this means discovering applications emitting log lines to files that need to be monitored.

Promtail borrows the same service discovery mechanism from Prometheus, although it currently only supports static and kubernetes service discovery. This limitation is due to the fact that Promtail is deployed as a daemon to every local machine and, as such, does not discover label from other machines. kubernetes service discovery fetches required labels from the Kubernetes API server while static usually covers all other use cases.

Just like Prometheus, promtail is configured using a scrape_configs stanza. relabel_configs allows for fine-grained control of what to ingest, what to drop, and the final metadata to attach to the log line. Refer to the docs for configuring Promtail for more details.

Loki Push API

Promtail can also be configured to receive logs from another Promtail or any Loki client by exposing the Loki Push API with the loki_push_api scrape config.

There are a few instances where this might be helpful:

complex network infrastructures where many machines having egress is not desirable.
using the Docker Logging Driver and wanting to provide a complex pipeline or to extract metrics from logs.
serverless setups where many ephemeral log sources want to send to Loki, sending to a Promtail instance with use_incoming_timestamp == false can avoid out-of-order errors and avoid having to use high cardinality labels.

Receiving logs From Syslog

When the Syslog Target is being used, logs can be written with the syslog protocol to the configured port.

AWS

If you need to run Promtail on Amazon Web Services EC2 instances, you can use our detailed tutorial.

Labeling and parsing

During service discovery, metadata is determined (pod name, filename, etc.) that may be attached to the log line as a label for easier identification when querying logs in Loki. Through relabel_configs, discovered labels can be mutated into the desired form.

To allow more sophisticated filtering afterwards, Promtail allows to set labels not only from service discovery, but also based on the contents of each log line. The pipeline_stages can be used to add or update labels, correct the timestamp, or re-write log lines entirely. Refer to the documentation for pipelines for more details.

Shipping

Once Promtail has a set of targets (i.e., things to read from, like files) and all labels are set correctly, it will start tailing (continuously reading) the logs from targets. Once enough data is read into memory or after a configurable timeout, it is flushed as a single batch to Loki.

As Promtail reads data from sources (files and systemd journal, if configured), it will track the last offset it read in a positions file. By default, the positions file is stored at /var/log/positions.yaml. The positions file helps Promtail continue reading from where it left off in the case of the Promtail instance restarting.

API

Promtail features an embedded web server exposing a web console at / and the following API endpoints:

`GET /ready`

This endpoint returns 200 when Promtail is up and running, and there’s at least one working target.

`GET /metrics`

This endpoint returns Promtail metrics for Prometheus. Refer to Observing Grafana Loki for the list of exported metrics.

Promtail web server config

The web server exposed by Promtail can be configured in the Promtail .yaml config file:

server:
  http_listen_address: 127.0.0.1
  http_listen_port: 9080

Lambda Promtail

Tue, 16 Jul 2024 15:42:20 +0000

Lambda Promtail

Grafana Loki includes Terraform and CloudFormation for shipping Cloudwatch and loadbalancer logs to Loki via a lambda function. This is done via lambda-promtail which processes cloudwatch events and propagates them to Loki (or a Promtail instance) via the push-api scrape config.

Deployment

lambda-promtail can easily be deployed via provided Terraform and CloudFormation files. The Terraform deployment also pulls variable values defined from variables.tf.

For both deployment types there are a few values that must be defined:

the write address, a Loki Write API compatible endpoint (Loki or Promtail)
basic auth username/password if the write address is a Loki endpoint and has authentication
the lambda-promtail image, full ECR repo path:tag

The Terraform deployment also takes in an array of log group and bucket names, and can take arrays for VPC subnets and security groups.

There’s also a flag to keep the log stream label when propagating the logs from Cloudwatch, which defaults to false. This can be helpful when the cardinality is too large, such as the case of a log stream per lambda invocation.

Additionally, an environment variable can be configured to add extra lables to the logs streamed by lambda-protmail. These extra labels will take the form __extra_<name>=<value>

Optional environment variable can be configured to add tenant id to the logs streamed by lambda-protmail.

In an effort to make deployment of lambda-promtail as simple as possible, we’ve created a public ECR repo to publish our builds of lambda-promtail. Users are still able to clone this repo, make their own modifications to the Go code, and upload their own image to their own ECR repo if they wish.

Examples

Terraform:

terraform apply -var "lambda_promtail_image=<repo:tag>" -var "write_address=https://logs-prod-us-central1.grafana.net/loki/api/v1/push" -var "password=<password>" -var "username=<user>" -var 'log_group_names=["/aws/lambda/log-group-1", "/aws/lambda/log-group-2"]' -var 'bucket_names=["bucket-a", "bucket-b"]' -var 'batch_size=131072'

The first few lines of main.tf define the AWS region to deploy to, you are free to modify this or remove and deploy to

provider "aws" {
  region = "us-east-2"
}

To keep the log group label add -var "keep_stream=true".

To add extra labels add -var 'extra_labels="name1,value1,name2,value2"'

To add tenant id add -var "tenant_id=value"

Note that the creation of subscription filter on Cloudwatch in the provided Terraform file only accepts an array of log group names, it does not accept strings for regex filtering on the logs contents via the subscription filters. We suggest extending the Terraform file to do so, or having lambda-promtail write to Promtail and using pipeline stages.

CloudFormation:

aws cloudformation create-stack --stack-name lambda-promtail --template-body file://template.yaml --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM --region us-east-2 --parameters ParameterKey=WriteAddress,ParameterValue=https://logs-prod-us-central1.grafana.net/loki/api/v1/push ParameterKey=Username,ParameterValue=<user> ParameterKey=Password,ParameterValue=<password> ParameterKey=LambdaPromtailImage,ParameterValue=<repo:tag>

Within the CloudFormation template file you should copy/paste and modify the subscription filter section as needed for each log group:

MainLambdaPromtailSubscriptionFilter:
  Type: AWS::Logs::SubscriptionFilter
  Properties:
    DestinationArn: !GetAtt LambdaPromtailFunction.Arn
    FilterPattern: ""
    LogGroupName: "/aws/lambda/some-lamda-log-group"

To keep the log group label add ParameterKey=KeepStream,ParameterValue=true.

To add extra labels, include ParameterKey=ExtraLabels,ParameterValue="name1,value1,name2,value2"

To add tenant id add ParameterKey=TenantID,ParameterValue=value.

To modify an already created CloudFormation stack you need to use update-stack.

Uses

Ephemeral Jobs

This workflow is intended to be an effective approach for monitoring ephemeral jobs such as those run on AWS Lambda which are otherwise hard/impossible to monitor via one of the other Loki clients.

Ephemeral jobs can quite easily run afoul of cardinality best practices. During high request load, an AWS lambda function might balloon in concurrency, creating many log streams in Cloudwatch. For this reason lambda-promtail defaults to not keeping the log stream value as a label when propagating the logs to Loki. This is only possible because new versions of Loki no longer have an ingestion ordering constraint on logs within a single stream.

Proof of concept Loki deployments

For those using Cloudwatch and wishing to test out Loki in a low-risk way, this workflow allows piping Cloudwatch logs to Loki regardless of the event source (EC2, Kubernetes, Lambda, ECS, etc) without setting up a set of Promtail daemons across their infrastructure. However, running Promtail as a daemon on your infrastructure is the best-practice deployment strategy in the long term for flexibility, reliability, performance, and cost.

Note: Propagating logs from Cloudwatch to Loki means you’ll still need to pay for Cloudwatch.

Loadbalancer logs

This workflow allows ingesting AWS loadbalancer logs stored on S3 to Loki.

Propagated Labels

Incoming logs can have six special labels assigned to them which can be used in relabeling or later stages in a Promtail pipeline:

__aws_log_type: Where this log came from (Cloudwatch or S3).
__aws_cloudwatch_log_group: The associated Cloudwatch Log Group for this log.
__aws_cloudwatch_log_stream: The associated Cloudwatch Log Stream for this log (if KEEP_STREAM=true).
__aws_cloudwatch_owner: The AWS ID of the owner of this event.
__aws_s3_log_lb: The name of the loadbalancer.
__aws_s3_log_lb_owner: The Account ID of the loadbalancer owner.

Limitations

Promtail labels

Note: This section is relevant if running Promtail between lambda-promtail and the end Loki deployment and was used to circumvent out of order problems prior to the v2.4 Loki release which removed the ordering constraint.

As stated earlier, this workflow moves the worst case stream cardinality from number_of_log_streams -> number_of_log_groups * number_of_promtails. For this reason, each Promtail must have a unique label attached to logs it processes (ideally via something like --client.external-labels=promtail=${HOSTNAME}) and it’s advised to run a small number of Promtails behind a load balancer according to your throughput and redundancy needs.

This trade-off is very effective when you have a large number of log streams but want to aggregate them by the log group. This is very common in AWS Lambda, where log groups are the “application” and log streams are the individual application containers which are spun up and down at a whim, possibly just for a single function invocation.

Data Persistence

Availability

For availability concerns, run a set of Promtails behind a load balancer.

Batching

Relevant if lambda-promtail is configured to write to Promtail. Since Promtail batches writes to Loki for performance, it’s possible that Promtail will receive a log, issue a successful 204 http status code for the write, then be killed at a later time before it writes upstream to Loki. This should be rare, but is a downside this workflow has.

This lambda will flush logs when the batch size hits the default value of 131072 (128KB), this can be changed with BATCH_SIZE environment variable, which is set to the number of bytes to use.

Templating/Deployment

The current CloudFormation template is rudimentary. If you need to add vpc configs, extra log groups to monitor, subnet declarations, etc, you’ll need to edit the template manually. If you need to subscribe to more than one Cloudwatch Log Group you’ll also need to copy paste that section of the template for each group.

The Terraform file is a bit more fleshed out, and can be configured to take in an array of log group and bucket names, as well as vpc configuration.

The provided Terraform and CloudFormation files are meant to cover the default use case, and more complex deployments will likely require some modification and extenstion of the provided files.

Example Promtail Config

Note: this should be run in conjunction with a Promtail-specific label attached, ideally via a flag argument like --client.external-labels=promtail=${HOSTNAME}. It will receive writes via the push-api on ports 3500 (http) and 3600 (grpc).

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://ip_or_hostname_where_Loki_run:3100/loki/api/v1/push

scrape_configs:
  - job_name: push1
    loki_push_api:
      server:
        http_listen_port: 3500
        grpc_listen_port: 3600
      labels:
        # Adds a label on all streams indicating it was processed by the lambda-promtail workflow.
        promtail: 'lambda-promtail'
    relabel_configs:
      - source_labels: ['__aws_log_type']
        target_label: 'log_type'
      # Maps the cloudwatch log group into a label called `log_group` for use in Loki.
      - source_labels: ['__aws_cloudwatch_log_group']
        target_label: 'log_group'
      # Maps the loadbalancer name into a label called `loadbalancer_name` for use in Loki.
      - source_label: ['__aws_s3_log_lb']
        target_label: 'loadbalancer_name'

Multiple Promtail Deployment

Disclaimer: The following section is only relevant for older versions of Loki that cannot accept out of order logs.

However, these may only be active for a very short while. This creates a problem for combining these short-lived log streams in Loki because timestamps may not strictly increase across multiple log streams. The other obvious route is creating labels based on log streams, which is also undesirable because it leads to cardinality problems via many low-throughput log streams.

Instead we can pipeline Cloudwatch logs to a set of Promtails, which can mitigate these problem in two ways:

Using Promtail’s push api along with the use_incoming_timestamp: false config, we let Promtail determine the timestamp based on when it ingests the logs, not the timestamp assigned by cloudwatch. Obviously, this means that we lose the origin timestamp because Promtail now assigns it, but this is a relatively small difference in a real time ingestion system like this.
In conjunction with (1), Promtail can coalesce logs across Cloudwatch log streams because it’s no longer susceptible to out-of-order errors when combining multiple sources (lambda invocations).

One important aspect to keep in mind when running with a set of Promtails behind a load balancer is that we’re effectively moving the cardinality problems from the number of log streams -> number of Promtails. If you have not configured Loki to accept out-of-order writes, you’ll need to assign a Promtail-specific label on each Promtail so that you don’t run into out-of-order errors when the Promtails send data for the same log groups to Loki. This can easily be done via a configuration like --client.external-labels=promtail=${HOSTNAME} passed to Promtail.

AWS

Tue, 16 Jul 2024 15:42:20 +0000

Sending logs from AWS services to Grafana Loki is a little different depending on what AWS service you are using:

Docker driver

Tue, 16 Jul 2024 15:42:20 +0000

Docker Driver Client

Grafana Loki officially supports a Docker plugin that will read logs from Docker containers and ship them to Loki. The plugin can be configured to send the logs to a private Loki instance or Grafana Cloud.

Docker plugins are not yet supported on Windows; see the Docker Engine managed plugin system documentation for more information.

Documentation on configuring the Loki Docker Driver can be found on the configuration page.

If you have any questions or issues using the Docker plugin feel free to open an issue in this repository.

Installing

The Docker plugin must be installed on each Docker host that will be running containers you want to collect logs from.

Run the following command to install the plugin:

docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions

To check installed plugins, use the docker plugin ls command. Plugins that have started successfully are listed as enabled:

$ docker plugin ls
ID                  NAME         DESCRIPTION           ENABLED
ac720b8fcfdb        loki         Loki Logging Driver   true

Once the plugin is installed it can be configured.

Upgrading

The upgrade process involves disabling the existing plugin, upgrading, then re-enabling and restarting Docker:

docker plugin disable loki --force
docker plugin upgrade loki grafana/loki-docker-driver:latest --grant-all-permissions
docker plugin enable loki
systemctl restart docker

Uninstalling

To cleanly uninstall the plugin, disable and remove it:

docker plugin disable loki --force
docker plugin rm loki

Know Issues

The driver keeps all logs in memory and will drop log entries if Loki is not reachable and if the quantity of max_retries has been exceeded. To avoid the dropping of log entries, setting max_retries to zero allows unlimited retries; the drive will continue trying forever until Loki is again reachable. Trying forever may have undesired consequences, because the Docker daemon will wait for the Loki driver to process all logs of a container, until the container is removed. Thus, the Docker daemon might wait forever if the container is stuck.

Use Promtail’s Docker target or Docker service discovery to avoid this issue.

Fluent Bit

Tue, 16 Jul 2024 15:42:20 +0000

Fluent Bit Loki Output

Fluent Bit is a fast and lightweight logs and metrics processor and forwarder that can be configured with the Grafana Loki output plugin to ship logs to Loki. You can define which log files you want to collect using the Tail or Stdin data pipeline input. Additionally, Fluent Bit supports multiple Filter and Parser plugins (Kubernetes, JSON, etc.) to structure and alter log lines.

Usage

Docker

You can run a Fluent Bit container with Loki output plugin pre-installed using our Docker Hub image:

docker run -v /var/log:/var/log \
    -e LOG_PATH="/var/log/*.log" -e LOKI_URL="http://localhost:3100/loki/api/v1/push" \
    grafana/fluent-bit-plugin-loki:latest

Kubernetes

You can run Fluent Bit as a Daemonset to collect all your Kubernetes workload logs.

To do so you can use our Fluent Bit helm chart:

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm upgrade --install fluent-bit grafana/fluent-bit \
    --set loki.serviceName=loki.svc.cluster.local

By default it will collect all containers logs and extract labels from Kubernetes API (container_name, namespace, etc..).

Alternatively you can install the Loki and Fluent Bit all together using:

helm upgrade --install loki-stack grafana/loki-stack \
    --set fluent-bit.enabled=true,promtail.enabled=false

AWS Elastic Container Service (ECS)

You can use fluent-bit Loki Docker image as a Firelens log router in AWS ECS. For more information about this see our AWS documentation

Local

First, you need to follow the instructions in order to build the plugin dynamic library.

The assuming you have Fluent Bit installed in your $PATH you can run the plugin using:

fluent-bit -e /path/to/built/out_grafana_loki.so -c fluent-bit.conf

You can also adapt your plugins.conf, removing the need to change the command line options:

[PLUGINS]
    Path /path/to/built/out_grafana_loki.so

Configuration Options

Key	Description	Default
Url	Url of loki server API endpoint.	http://localhost:3100/loki/api/v1/push
TenantID	The tenant ID used by default to push logs to Loki. If omitted or empty it assumes Loki is running in single-tenant mode and no `X-Scope-OrgID` header is sent.	""
BatchWait	Time to wait before send a log batch to Loki, full or not.	1s
BatchSize	Log batch size to send a log batch to Loki (unit: Bytes).	10 KiB (10 * 1024 Bytes)
Timeout	Maximum time to wait for loki server to respond to a request.	10s
MinBackoff	Initial backoff time between retries.	500ms
MaxBackoff	Maximum backoff time between retries.	5m
MaxRetries	Maximum number of retries when sending batches. Setting it to `0` will retry indefinitely.	10
Labels	labels for API requests.	{job=“fluent-bit”}
LogLevel	LogLevel for plugin logger.	“info”
RemoveKeys	Specify removing keys.	none
AutoKubernetesLabels	If set to true, it will add all Kubernetes labels to Loki labels	false
LabelKeys	Comma separated list of keys to use as stream labels. All other keys will be placed into the log line. LabelKeys is deactivated when using `LabelMapPath` label mapping configuration.	none
LineFormat	Format to use when flattening the record to a log line. Valid values are “json” or “key_value”. If set to “json” the log line sent to Loki will be the fluentd record (excluding any keys extracted out as labels) dumped as json. If set to “key_value”, the log line will be each item in the record concatenated together (separated by a single space) in the format =.	json
DropSingleKey	If set to true and after extracting label_keys a record only has a single key remaining, the log line sent to Loki will just be the value of the record key.	true
LabelMapPath	Path to a json file defining how to transform nested records.	none
Buffer	Enable buffering mechanism	false
BufferType	Specify the buffering mechanism to use (currently only dque is implemented).	dque
DqueDir	Path to the directory for queued logs	/tmp/flb-storage/loki
DqueSegmentSize	Segment size in terms of number of records per segment	500
DqueSync	Whether to fsync each queue change. Specify no fsync with “normal”, and fsync with “full”.	“normal”
DqueName	Queue name, must be uniq per output	dque

Labels

Labels are used to query logs {container_name="nginx", cluster="us-west1"}, they are usually metadata about the workload producing the log stream (instance, container_name, region, cluster, level). In Loki labels are indexed consequently you should be cautious when choosing them (high cardinality label values can have performance drastic impact).

You can use Labels, RemoveKeys , LabelKeys and LabelMapPath to how the output plugin will perform labels extraction.

AutoKubernetesLabels

If set to true, it will add all Kubernetes labels to Loki labels automatically and ignore parameters LabelKeys, LabelMapPath.

LabelMapPath

When using the Parser and Filter plugins Fluent Bit can extract and add data to the current record/log data. While Loki labels are key value pair, record data can be nested structures. You can pass a JSON file that defines how to extract labels from each record. Each json key from the file will be matched with the log record to find label values. Values from the configuration are used as label names.

Considering the record below :

{
  "kubernetes": {
    "container_name": "promtail",
    "pod_name": "promtail-xxx",
    "namespace_name": "prod",
    "labels" : {
        "team": "x-men"
    }
  },
  "HOSTNAME": "docker-desktop",
  "log" : "a log line",
  "time": "20190926T152206Z"
}

and a LabelMap file as follow :

{
  "kubernetes": {
    "container_name": "container",
    "pod_name": "pod",
    "namespace_name": "namespace",
    "labels" : {
        "team": "team"
    }
  }
}

The labels extracted will be {team="x-men", container="promtail", pod="promtail-xxx", namespace="prod"}.

If you don’t want the kubernetes and HOSTNAME fields to appear in the log line you can use the RemoveKeys configuration field. (e.g. RemoveKeys kubernetes,HOSTNAME).

Buffering

Buffering refers to the ability to store the records somewhere, and while they are processed and delivered, still be able to store more. The Loki output plugin can be blocked by the Loki client because of its design:

If the BatchSize is over the limit, the output plugin pauses receiving new records until the pending batch is successfully sent to the server
If the Loki server is unreachable (retry 429s, 500s and connection-level errors), the output plugin blocks new records until the Loki server is available again, and the pending batch is successfully sent to the server or as long as the maximum number of attempts has been reached within configured back-off mechanism

The blocking state with some of the input plugins is not acceptable, because it can have an undesirable side effect on the part that generates the logs. Fluent Bit implements a buffering mechanism that is based on parallel processing. Therefore, it cannot send logs in order. There are two ways of handling the out-of-order logs:

Configure Loki to accept out-of-order writes.

Configure the Loki output plugin to use the buffering mechanism based on dque, which is compatible with the Loki server strict time ordering:

[Output]
    Name grafana-loki
    Match *
    Url http://localhost:3100/loki/api/v1/push
    Buffer true
    DqueSegmentSize 8096
    DqueDir /tmp/flb-storage/buffer
    DqueName loki.0

Configuration examples

To configure the Loki output plugin add this section to fluent-bit.conf

[Output]
    Name grafana-loki
    Match *
    Url http://localhost:3100/loki/api/v1/push
    BatchWait 1s
    BatchSize 30720
    # (30KiB)
    Labels {test="fluent-bit-go", lang="Golang"}
    RemoveKeys key1,key2
    LabelKeys key3,key4
    LineFormat key_value

[Output]
    Name grafana-loki
    Match *
    Url http://localhost:3100/loki/api/v1/push
    BatchWait 1s
    BatchSize 30720 # (30KiB)
    AutoKubernetesLabels true
    RemoveKeys key1,key2

A full example configuration file is also available in the Loki repository.

Running multiple plugin instances

You can run multiple plugin instances in the same fluent-bit process, for example if you want to push to different Loki servers or route logs into different Loki tenant IDs. To do so, add additional [Output] sections.

Fluentd

Tue, 16 Jul 2024 15:42:20 +0000

Fluentd Loki Output Plugin

Grafana Loki has a Fluentd output plugin called fluent-plugin-grafana-loki that enables shipping logs to a private Loki instance or Grafana Cloud.

The plugin source code is in the fluentd directory of the repository.

Installation

Local

To install the plugin use fluent-gem:

fluent-gem install fluent-plugin-grafana-loki

Docker Image

The Docker image grafana/fluent-plugin-loki:master contains default configuration files. By default, fluentd containers use that default configuration. You can instead specify your fluentd.conf configuration file with a FLUENTD_CONF environment variable.

This image also uses LOKI_URL, LOKI_USERNAME, and LOKI_PASSWORD environment variables to specify the Loki’s endpoint, user, and password (you can leave the USERNAME and PASSWORD blank if they’re not used).

This image will start an instance of Fluentd to forward incoming logs to the specified Loki URL. As an alternate, containerized applications can also use docker driver plugin to ship logs without needing Fluentd.

Example

A Docker Compose configuration that will work looks like:

services:
  fluentd:
    image: grafana/fluent-plugin-loki:master
    command:
      - "fluentd"
      - "-v"
      - "-p"
      - "/fluentd/plugins"
    environment:
      LOKI_URL: http://loki:3100
      LOKI_USERNAME:
      LOKI_PASSWORD:
    deploy:
      mode: global
    configs:
      - source: loki_config
        target: /fluentd/etc/loki/loki.conf
    networks:
      - loki
    volumes:
      - host_logs:/var/log
      # Needed for journald log ingestion:
      - /etc/machine-id:/etc/machine-id
      - /dev/log:/dev/log
      - /var/run/systemd/journal/:/var/run/systemd/journal/
    logging:
      options:
         tag: infra.monitoring

Usage

Note: use either <label>...</label> or extra_labels to set at least one label.

In your Fluentd configuration, use @type loki. Additional configuration is optional. Default values would look like this:

<match **>
  @type loki
  url "https://logs-prod-us-central1.grafana.net"
  username "#{ENV['LOKI_USERNAME']}"
  password "#{ENV['LOKI_PASSWORD']}"
  extra_labels {"env":"dev"}
  flush_interval 10s
  flush_at_shutdown true
  buffer_chunk_limit 1m
</match>

Adding labels

Simple label from top level attribute

<match mytag>
  @type loki
  # ...
  <label>
    fluentd_worker
  </label>
  # ...
</match>

You can rewrite the label keys as well as the following

<match mytag>
  @type loki
  # ...
  <label>
    worker fluentd_worker
  </label>
  # ...
</match>

You can use record accessor syntax for nested field.

<match mytag>
  @type loki
  # ...
  <label>
    container $.kubernetes.container
  </label>
  # ...
</match>

Extracting Kubernetes labels

As Kubernetes labels are a list of nested key-value pairs there is a separate option to extract them. Note that special characters like “. - /” will be overwritten with _. Use with the remove_keys kubernetes option to eliminate metadata from the log.

<match mytag>
  @type loki
  # ...
  extract_kubernetes_labels true
  remove_keys kubernetes
  <label>
    container $.kubernetes.container
  </label>
  # ...
</match>

You can also include automatically all kubernetes labels by using extract_kubernetes_labels true in your configuration.

Multi-worker usage

Out-of-order inserts are enabled by default in Loki; refer to accept out-of-order writes. If out-of-order inserts are disabled, attempting to insert a log entry with an earlier timestamp after a log entry with identical labels but a later timestamp, the insert will fail with HTTP status code: 500, message: rpc error: code = Unknown desc = Entry out of order. Therefore, in order to use this plugin in a multi worker Fluentd setup, you’ll need to include the worker ID in the labels or otherwise ensure log streams are always sent to the same worker.

For example, using fluent-plugin-record-modifier:

<filter mytag>
    @type record_modifier
    <record>
        fluentd_worker "#{worker_id}"
    </record>
</filter>

<match mytag>
  @type loki
  # ...
  <label>
    fluentd_worker
  </label>
  # ...
</match>

Using multiple buffer flush threads

Similarly, when using flush_thread_count > 1 in the buffer section, a thread identifier must be added as a label to ensure that log chunks flushed in parallel to loki by fluentd always have increasing times for their unique label sets.

This plugin automatically adds a fluentd_thread label with the name of the buffer flush thread when flush_thread_count > 1.

Configuration

`url`

The URL of the Loki server to send logs to. When sending data, the publish path (../api/loki/v1/push) will automatically be appended. By default the url is set to https://logs-prod-us-central1.grafana.net, the url of the Grafana Labs hosted Loki service.

Proxy Support

Starting with version 0.8.0, this gem uses excon, which supports proxy with environment variables.

`username` / `password`

Specify a username and password if the Loki server requires authentication. If using the GrafanaLab’s hosted Loki, the username needs to be set to your instanceId and the password should be a Grafana.com api key.

tenant

Loki is a multi-tenant log storage platform and all requests sent must include a tenant. For some installations the tenant will be set automatically by an authenticating proxy. Otherwise you can define a tenant to be passed through. The tenant can be any string value.

The tenant field also supports placeholders, so it can dynamically change based on tag and record fields. Each placeholder must be added as a buffer chunk key. The following is an example of setting the tenant based on a k8s pod label:

<match **>
  @type loki
  url "https://logs-prod-us-central1.grafana.net"
  tenant ${$.kubernetes.labels.tenant}
  # ...
  <buffer $.kubernetes.labels.tenant>
    @type memory
    flush_interval 5s
  </buffer>
</match>

Client certificate verification

Specify a pair of client certificate and private key with cert and key if a reverse proxy with client certificate verification is configured in front of Loki. ca_cert can also be specified if the server uses custom certificate authority.

<match **>
  @type loki

  url "https://loki"

  cert /path/to/certificate.pem
  key /path/to/key.key
  ca_cert /path/to/ca.pem

  ...
</match>

Server certificate verification

A flag to disable a server certificate verification. By default the insecure_tls is set to false.

<match **>
  @type loki

  url "https://loki"

  insecure_tls true

  ...
</match>

Output format

Loki is intended to index and group log streams using only a small set of labels. It is not intended for full-text indexing. When sending logs to Loki the majority of log message will be sent as a single log “line”.

There are few configurations settings to control the output format.

extra_labels: (default: nil) set of labels to include with every Loki stream. eg {"env":"dev", "datacenter": "dc1"}
remove_keys: (default: nil) comma separated list of needless record keys to remove. All other keys will be placed into the log line. You can use record_accessor syntax.
line_format (default:key_value): format to use when flattening the record to a log line. Valid values are “json” or “key_value”. If set to “json” the log line sent to Loki will be the fluentd record (excluding any keys extracted out as labels) dumped as json. If set to “key_value”, the log line will be each item in the record concatenated together (separated by a single space) in the format <key>=<value>.
drop_single_key: if set to true and a record only has 1 key after extracting <label></label> blocks, set the log line to the value and discard the key.
include_thread_label (default: true): whether or not to include the fluentd_thread label when multiple threads are used for flushing.

Buffer options

fluentd-plugin-loki extends Fluentd’s builtin Output plugin and use compat_parameters plugin helper. It adds the following options:

buffer_type memory
flush_interval 10s
retry_limit 17
retry_wait 1.0
num_threads 1

Logstash

Tue, 16 Jul 2024 15:42:20 +0000

Logstash

Grafana Loki has a Logstash output plugin called logstash-output-loki that enables shipping logs to a Loki instance or Grafana Cloud.

Installation

Local

If you need to install the Loki output plugin manually you can do simply so by using the command below:

$ bin/logstash-plugin install logstash-output-loki

This will download the latest gem for the output plugin and install it in logstash.

Docker

We also provide a docker image on docker hub. The image contains logstash and the Loki output plugin already pre-installed.

For example if you want to run logstash in docker with the loki.conf as pipeline configuration you can use the command bellow :

docker run -v `pwd`/loki-test.conf:/home/logstash/ --rm grafana/logstash-output-loki:1.0.1 -f loki-test.conf

Kubernetes

We also provide default helm values for scraping logs with Filebeat and forward them to Loki with logstash in our loki-stack umbrella chart. You can switch from Promtail to logstash by using the following command:

helm upgrade --install loki loki/loki-stack \
    --set filebeat.enabled=true,logstash.enabled=true,promtail.enabled=false \
    --set loki.fullnameOverride=loki,logstash.fullnameOverride=logstash-loki

This will automatically scrape all pods logs in the cluster and send them to Loki with Kubernetes metadata attached as labels. You can use the values.yaml file as a starting point for your own configuration.

Usage and Configuration

To configure Logstash to forward logs to Loki, simply add the loki output to your Logstash configuration file as documented below :

output {
  loki {
    [url => "" | default = none | required=true]

    [tenant_id => string | default = nil | required=false]

    [message_field => string | default = "message" | required=false]
    
    [include_fields => array | default = [] | required=false]

    [batch_wait => number | default = 1(s) | required=false]

    [batch_size => number | default = 102400(bytes) | required=false]

    [min_delay => number | default = 1(s) | required=false]

    [max_delay => number | default = 300(s) | required=false]

    [retries => number | default = 10 | required=false]

    [username => string | default = nil | required=false]

    [password => secret | default = nil | required=false]

    [cert => path | default = nil | required=false]

    [key => path | default = nil| required=false]

    [ca_cert => path | default = nil | required=false]

    [insecure_skip_verify => boolean | default = false | required=false]
  }
}

By default Loki will create entry from event fields it receives. A logstash event as shown below.

{
  "@timestamp" => 2017-04-26T19:33:39.257Z,
  "src"        => "localhost",
  "@version"   => "1",
  "host"       => "localhost.localdomain",
  "pid"        => "1",
  "message"    => "Apr 26 12:20:02 localhost systemd[1]: Starting system activity accounting tool...",
  "type"       => "stdin",
  "prog"       => "systemd",
}

Contains a message and @timestamp fields, which are respectively used to form the Loki entry log line and timestamp.

You can use a different property for the log line by using the configuration property message_field. If you also need to change the timestamp value use the Logstash date filter to change the @timestamp field.

All other fields (except nested fields) will form the label set (key value pairs) attached to the log line. This means you’re responsible for mutating and dropping high cardinality labels such as client IPs. You can usually do so by using a mutate filter.

Note: In version 1.1.0 and greater of this plugin you can also specify a list of labels to allowlist via the include_fields configuration.

For example the configuration below :

input {
  ...
}

filter {
  mutate {
    add_field => {
      "cluster" => "us-central1"
      "job" => "logstash"
    }
    replace => { "type" => "stream"}
    remove_field => ["src"]
  }
}
output {
  loki {
    url => "http://myloki.domain:3100/loki/api/v1/push"
  }
}

Will add cluster and job static labels, remove src fields and replace type to be named stream.

If you want to include nested fields or metadata fields (starting with @) you need to rename them.

For example when using Filebeat with the add_kubernetes_metadata processor, it will attach Kubernetes metadata to your events like below:

{
  "kubernetes" : {
    "labels" : {
      "app" : "MY-APP",
      "pod-template-hash" : "959f54cd",
      "serving" : "true",
      "version" : "1.0",
      "visualize" : "true"
    },
    "pod" : {
      "uid" : "e20173cb-3c5f-11ea-836e-02c1ee65b375",
      "name" : "MY-APP-959f54cd-lhd5p"
    },
    "node" : {
      "name" : "ip-xxx-xx-xx-xxx.ec2.internal"
    },
    "container" : {
      "name" : "istio"
    },
    "namespace" : "production",
    "replicaset" : {
      "name" : "MY-APP-959f54cd"
    }
  },
  "message": "Failed to parse configuration",
  "@timestamp": "2017-04-26T19:33:39.257Z",
}

The filter below show you how to extract those Kubernetes fields into labels (container_name,namespace,pod and host):

filter {
  if [kubernetes] {
    mutate {
      add_field => {
        "container_name" => "%{[kubernetes][container][name]}"
        "namespace" => "%{[kubernetes][namespace]}"
        "pod" => "%{[kubernetes][pod][name]}"
      }
      replace => { "host" => "%{[kubernetes][node][name]}"}
    }
  }
  mutate {
    remove_field => ["tags"]
  }
}

Configuration Properties

url

The url of the Loki server to send logs to. When sending data the push path need to also be provided e.g. http://localhost:3100/loki/api/v1/push.

If you want to send to GrafanaCloud you would use https://logs-prod-us-central1.grafana.net/loki/api/v1/push.

username / password

Specify a username and password if the Loki server requires basic authentication. If using the GrafanaLab’s hosted Loki, the username needs to be set to your instance/user id and the password should be a Grafana.com api key.

message_field

Message field to use for log lines. You can use logstash key accessor language to grab nested property, for example : [log][message].

include_fields

An array of fields which will be mapped to labels and sent to Loki, when this list is configured only these fields will be sent, all other fields will be ignored.

batch_wait

Interval in seconds to wait before pushing a batch of records to Loki. This means even if the batch size is not reached after batch_wait a partial batch will be sent, this is to ensure freshness of the data.

batch_size

Maximum batch size to accrue before pushing to loki. Defaults to 102400 bytes

Backoff config

min_delay => 1(1s)

Initial backoff time between retries

max_delay => 300(5m)

Maximum backoff time between retries

retries => 10

Maximum number of retries to do. Setting it to 0 will retry indefinitely.

tenant_id

client certificate verification

insecure_skip_verify

A flag to disable server certificate verification. By default it is set to false.

Full configuration example

input {
  beats {
    port => 5044
  }
}

filter {
  if [kubernetes] {
    mutate {
      add_field => {
        "container_name" => "%{[kubernetes][container][name]}"
        "namespace" => "%{[kubernetes][namespace]}"
        "pod" => "%{[kubernetes][pod][name]}"
      }
      replace => { "host" => "%{[kubernetes][node][name]}"}
    }
  }
  mutate {
    remove_field => ["tags"]  # Note: with include_fields defined below this wouldn't be necessary
  }
}

output {
  loki {
    url => "https://logs-prod-us-central1.grafana.net/loki/api/v1/push"
    username => "3241"
    password => "REDACTED"
    batch_size => 112640 #112.64 kilobytes
    retries => 5
    min_delay => 3
    max_delay => 500
    message_field => "message"
    include_fields => ["container_name","namespace","pod","host"]
  }
  # stdout { codec => rubydebug }
}

k6 load testing

Mon, 14 Apr 2025 21:05:47 +0000

k6 Loki extension load testing

Grafana k6 is a modern load-testing tool. Its clean and approachable scripting API works locally or in the cloud. Its configuration makes it flexible.

The xk6-loki extension permits pushing logs to and querying logs from a Loki instance. It acts as a Loki client, simulating real-world load to test the scalability, reliability, and performance of your Loki installation.

Before you begin

k6 is written in Golang. Download and install a Go environment.

Installation

xk6-loki is an extension to the k6 binary. Build a custom k6 binary that includes the xk6-loki extension.

Install the xk6 extension bundler:
Bash
```
go install go.k6.io/xk6/cmd/xk6@latest
```

Check out the grafana/xk6-loki repository:

git clone https://github.com/grafana/xk6-loki
cd xk6-loki

Build k6 with the extension:
Bash
```
make k6
```

Usage

Use the custom-built k6 binary in the same way as a non-custom k6 binary:

./k6 run test.js

test.js is a Javascript load test. Refer to the k6 documentation to get started.

Scripting API

The custom-built k6 binary provides a Javascript loki module.

Your Javascript load test imports the module:

import loki from 'k6/x/loki';

Classes of this module are:

class	description
`Config`	configuration for the `Client` class
`Client`	client for writing and reading logs from Loki

Config and Client must be called on the k6 init context (see Test life cycle) outside of the default function so the client is only configured once and shared between all VU iterations.

The Client class exposes the following instance methods:

method	description
`push()`	shortcut for `pushParameterized(5, 8001024, 10241024)`
`pushParameterized(streams, minSize, maxSize)`	execute push request (POST /loki/api/v1/push)
`instantQuery(query, limit)`	execute instant query (GET /loki/api/v1/query)
`client.rangeQuery(query, duration, limit)`	execute range query (GET /loki/api/v1/query_range)
`client.labelsQuery(duration)`	execute labels query (GET /loki/api/v1/labels)
`client.labelValuesQuery(label, duration)`	execute label values query (GET /loki/api/v1/label/<name>/values)
`client.seriesQuery(matchers, duration)`	execute series query (GET /loki/api/v1/series)

Javascript load test example:

import loki from 'k6/x/loki';

const timeout = 5000; // ms
const conf = loki.Config("http://localhost:3100", timeout);
const client = loki.Client(conf);

export default () => {
   client.pushParameterized(2, 512*1024, 1024*1024);
};

Refer to grafana/xk6-loki for the complete k6/x/loki module API reference.