Menu

Send logs to Grafana Loki using the OpenTelemetry Collector

In this guide, we’ll set up an OpenTelemetry Collector that collects logs and sends them to Loki Push API endpoint.

Collector has multiple receivers to get the logs in: journald, filelog, syslog, promtail and others. You can choose the one or multiple of them that suits your needs.

In this guide we will show you how to collect system logs with a filelog receiver and send them to Loki via OpenTelemetry Collector using Loki exporter.

Prerequisites

Before we start, we need to have Loki and Grafana running. In this guide we will run them using docker-compose. Here is a docker-compose.yaml file:

version: "3"

networks:
  loki:

services:
  loki:
    image: grafana/loki:latest
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml
    networks:
      - loki

  grafana:
    image: grafana/grafana:latest
    environment:
      - "GF_AUTH_DISABLE_LOGIN_FORM=true"
      - "GF_AUTH_ANONYMOUS_ENABLED=true"
      - "GF_AUTH_ANONYMOUS_ORG_ROLE=Admin"
    ports:
      - "3000:3000"
    networks:
      - loki

Run:

docker-compose up -d

Now Grafana is running on http://localhost:3000 and Loki is running on http://localhost:3100. To visualize Loki logs in Grafana you will need to add a Loki data source. To do so open http://localhost:3000. Go to Configuration > Data sources, and click Add datasource. Choose Loki from the list of data sources. Input http://loki:3100 as a HTTP URL, and click Save & Test. The data source should be successfully added.

Download the last version of OpenTelemetry Collector

You can download the binary for most architectures/OS on GitHub for v0.63.1 or later versions, if available. The examples in this guide are based on Mac OS:

$ curl -O -L "https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.63.1/otelcol-contrib_0.63.1_darwin_amd64.tar.gz"
$ gunzip -c otelcol-contrib_0.63.1_darwin_amd64.tar.gz | tar xopf -

The Filelog receiver tails and parses logs from files. In this example, we will tail system logs and send them to Grafana Loki.

Prepare OpenTelemetry Collector config file

Create a config.yaml file:

receivers:
  filelog:
    include: [/var/log/*.log]

processors:
 attributes:
   actions:
     - action: insert
       key: log_file_name
       from_attribute: log.file.name
     - action: insert
       key: loki.attribute.labels
       value: log_file_name

exporters:
  logging:
    logLevel: debug

  loki:
    endpoint: "http://localhost:3100/loki/api/v1/push"
service:
  pipelines:
    logs:
      receivers: [filelog]
      processors: [attributes]
      exporters: [logging, loki]

To convert OTLP attribute log.file.name into a Loki label, we need to use attributes procesor. Currently, Loki does not support labels with dots. That’s why to add Loki label based on log.file.name OTLP attribute we need to specify two actions. The first one inserts a new attribute log_file_name from the OTLP attribute log.file.name. The second one is a hint for Loki, specifying that the log_file_name attribute should be placed as a Loki label. Then run:

$ ./otelcol-contrib --config ./config.yaml

Now you can open Grafana, go to Explore, choose Loki, and see logs.

Summary

In this guide, we provided all the steps you need to configure the OpenTelemetry Collector to collect system logs with filelog receiver and send them to Loki.