OpenTelemetry OpenTelemetry Collector Send logs to Loki Send logs to Loki with Loki receiver

Send logs to Loki with Loki receiver

Let’s imagine you have your application running, and you would like to collect metrics, traces, and logs from it.

Using the OpenTelemetry Collector allows you to collect all these signals in one place, and control various backends that the individual signals send to.

In this guide, we will focus on logs.

Let’s imagine that you collect metrics and traces with OpenTelemetry collector and your application logs with Promtail. And now you want to move the logs collection configuration into the OpenTelemetry Collector along with other signals.

The Loki receiver component will help you with that. All you need to do is add loki receiver to your OpenTelemetry Collector pipeline and change the current Promtail configuration, so it sends collected logs to the collector instead of Loki.

Loki receiver will get this data in the Loki format and transform it to OTLP format and send it further through the processing pipeline.

Here is an example of an application that uses Promtail to send logs to Loki.

Prerequisites

Before we start, we need to have Loki, Grafana, Promtail and the application are running. In this guide we will run them using docker-compose. Let’s create a new directory example:

$ mkdir example
$ cd example

Then create file promtail-local-config.yaml Inside the example directory:

---
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: flog_scrape 
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
    relabel_configs:
      - source_labels: ['__meta_docker_container_name']
        regex: '/(.*)'
        target_label: 'container'

Create the docker-compose.yaml file:

version: "3"

networks:
  loki:

services:
  loki:
    image: grafana/loki:latest
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml
    networks:
      - loki

  grafana:
    image: grafana/grafana:latest
    environment:
      - "GF_AUTH_DISABLE_LOGIN_FORM=true"
      - "GF_AUTH_ANONYMOUS_ENABLED=true"
      - "GF_AUTH_ANONYMOUS_ORG_ROLE=Admin"
    entrypoint:
      - sh
      - -euc
      - |
        mkdir -p /etc/grafana/provisioning/datasources
        cat <<EOF > /etc/grafana/provisioning/datasources/ds.yaml
        apiVersion: 1
        datasources:
          - name: Loki
            type: loki
            access: proxy
            url: http://loki:3100
        EOF
        /run.sh
    ports:
      - "3000:3000"
    depends_on:
      - loki
    networks:
      - loki

  promtail:
    image: grafana/promtail:2.8.0
    volumes:
      - ./promtail-local-config.yaml:/etc/promtail/config.yaml:ro
      - /var/run/docker.sock:/var/run/docker.sock
    command: -config.file=/etc/promtail/config.yaml
    networks:
      - loki

  flog:
    image: mingrammer/flog
    command: -f json -d 1s -l
    networks:
      - loki

Run:

docker-compose up -d

Now Grafana is running on http://localhost:3000 and Loki is running on http://localhost:3100. Flog app generates logs, and Promtail collects logs and sends them to Loki.

Open http://localhost:3000 and go to Explore, choose Loki datasource. Select container from the label filter dropdown and promtail-lokireceiver-flog-1 from the value dropdown. You will see flog application logs.

Use OpenTelemetry Collector to send logs to Loki

Now let’s change the configuration so Promtail sends logs to the OpenTelemetry collector instead of Loki. Create otel-config.yaml file in the example directory:

receivers:
  loki:
    protocols:
      http:
    use_incoming_timestamp: true

processors:
  attributes:
    actions:
      - action: insert
        key: loki.attribute.labels
        value: container
      - action: insert
        key: loki.format
        value: raw
exporters:
  loki:
    endpoint: http://loki:3100/loki/api/v1/push

service:
  pipelines:
    logs:
      receivers: [loki]
      processors: [attributes]
      exporters: [loki]

Here we use attributes processor to promote container attribute to Loki label and to specify the format used for writing log lines to Loki. More details on labels and format could be found in Loki exporter README

Add service to the docker-compose.yaml to run the OpenTelemetry collector in the container:

otelcol:
  image: otel/opentelemetry-collector-contrib:0.76.1
  deploy:
    resources:
      limits:
        memory: 125M
  restart: unless-stopped
  command: [ "--config=/etc/otelcol-config.yaml" ]
  volumes:
    - ./otelcol-config.yaml:/etc/otelcol-config.yaml
  ports:
    - "3500:3500" # loki receiver HTTP
  networks:
    - loki

And change the promtail-local-config.yaml to send logs to the OpenTelemetry collector instead of Loki:

clients:
  - url: http://otelcol:3500/loki/api/v1/push

Restart docker-compose:

docker-compose restart

Open http://localhost:3000 and go to Explore, choose Loki datasource. Select container from the label filter dropdown and promtail-lokireceiver-flog-1 from the value dropdown.

You will see the same logs as you saw before, except the new label exporter=OTLP wad added. But now the backend where these logs are sent is configured in the same place as other signals: in the OpenTelemetry collector config otel-config.yaml.

Using OpenTelemetry collector as a client of the Promtail allows to ingest loki formatted logs into the collector where they can be transformed to the OTLP format and processed by the collector pipeline. If you like you can change the backend and send your logs to another storage.

Summary

In this guide, we provided all the steps you need to configure Promtail and OpenTelemetry Collector. Promtail scrapes application logs and sends them to the OpenTelemetry collector for further processing. This setup allows us to keep the configuration for all telemetry signals in one place - OpenTelemetry collector config.