Documentation Grafana Alloy Reference Components loki loki.source.kafka

loki.source.kafka

loki.source.kafka reads messages from Kafka using a consumer group and forwards them to other loki.* components.

The component starts a new Kafka consumer group for the given arguments and fans out incoming entries to the list of receivers in forward_to.

Before using loki.source.kafka, Kafka should have at least one producer writing events to at least one topic. Follow the steps in the Kafka Quick Start to get started with Kafka.

You can specify multiple loki.source.kafka components by giving them different labels.

Usage

Alloy Copy

loki.source.kafka "<LABEL>" {
  brokers    = "<BROKER_LIST>"
  topics     = "<TOPIC_LIST>"
  forward_to = <RECEIVER_LIST>
}

Arguments

You can use the following arguments with loki.source.kafka:

assignor values can be either "range", "roundrobin", or "sticky".

If a topic starts with a ‘^’, it’s treated as a regular expression and may match multiple topics.

Labels from the labels argument are applied to every message that the component reads.

The relabel_rules field can make use of the rules export value from a loki.relabel component to apply one or more relabeling rules to log entries before they’re forwarded to the list of receivers in forward_to.

In addition to custom labels, the following internal labels prefixed with __ are available:

• __meta_kafka_group_id
• __meta_kafka_member_id
• __meta_kafka_message_key
• __meta_kafka_message_offset
• __meta_kafka_partition
• __meta_kafka_topic

All labels starting with __ are removed prior to forwarding log entries. To keep these labels, relabel them using a loki.relabel component and pass its rules export to the relabel_rules argument.

Blocks

You can use the following blocks with loki.source.kafka:

The > symbol indicates deeper levels of nesting. For example, authentication > sasl_config refers to a sasl_config block defined inside a authentication block.

authentication

The authentication block defines the authentication method when communicating with the Kafka event brokers.

type supports the values "none", "ssl", and "sasl". If "ssl" is used, you must set the tls_config block. If "sasl" is used, you must set the sasl_config block.

sasl_config

The sasl_config block defines the listen address and port where the listener expects Kafka messages to be sent to.

oauth_config

The oauth_config is required when the SASL mechanism is set to OAUTHBEARER.

tls_config

Exported fields

loki.source.kafka doesn’t export any fields.

Component health

loki.source.kafka is only reported as unhealthy if given an invalid configuration.

Debug information

loki.source.kafka doesn’t expose additional debug info.

Example

This example consumes Kafka events from the specified brokers and topics then forwards them to a loki.write component using the Kafka timestamp.

Alloy Copy

loki.source.kafka "local" {
  brokers                = ["localhost:9092"]
  topics                 = ["quickstart-events"]
  labels                 = {component = "loki.source.kafka"}
  forward_to             = [loki.write.local.receiver]
  use_incoming_timestamp = true
  relabel_rules          = loki.relabel.kafka.rules
}

loki.relabel "kafka" {
  forward_to      = [loki.write.local.receiver]

  rule {
    source_labels = ["__meta_kafka_topic"]
    target_label  = "topic"
  }
}

loki.write "local" {
  endpoint {
    url = "loki:3100/api/v1/push"
  }
}

Compatible components

loki.source.kafka can accept arguments from the following components:

• Components that export Loki LogsReceiver