Send logs to Loki with Loki receiver
Let’s imagine you have your application running, and you would like to collect metrics, traces, and logs from it.
Using the OpenTelemetry Collector allows you to collect all these signals in one place, and control various backends that the individual signals send to.
In this guide, we will focus on logs.
Let’s imagine that you collect metrics and traces with OpenTelemetry collector and your application logs with Promtail. And now you want to move the logs collection configuration into the OpenTelemetry Collector along with other signals.
The Loki receiver component will help you with that. All you need to do is add loki receiver to your OpenTelemetry Collector pipeline and change the current Promtail configuration, so it sends collected logs to the collector instead of Loki.
Loki receiver will get this data in the Loki format and transform it to OTLP format and send it further through the processing pipeline.
Here is an example of an application that uses Promtail to send logs to Loki.
Prerequisites
Before we start, we need to have Loki, Grafana, Promtail and the application are running. In this guide we will run them using docker-compose.
Let’s create a new directory example
:
$ mkdir example
$ cd example
Then create file promtail-local-config.yaml
Inside the example
directory:
---
server:
http_listen_port: 9080
grpc_listen_port: 0
positions:
filename: /tmp/positions.yaml
clients:
- url: http://loki:3100/loki/api/v1/push
scrape_configs:
- job_name: flog_scrape
docker_sd_configs:
- host: unix:///var/run/docker.sock
refresh_interval: 5s
relabel_configs:
- source_labels: ['__meta_docker_container_name']
regex: '/(.*)'
target_label: 'container'
Create the docker-compose.yaml
file:
version: "3"
networks:
loki:
services:
loki:
image: grafana/loki:latest
ports:
- "3100:3100"
command: -config.file=/etc/loki/local-config.yaml
networks:
- loki
grafana:
image: grafana/grafana:latest
environment:
- "GF_AUTH_DISABLE_LOGIN_FORM=true"
- "GF_AUTH_ANONYMOUS_ENABLED=true"
- "GF_AUTH_ANONYMOUS_ORG_ROLE=Admin"
entrypoint:
- sh
- -euc
- |
mkdir -p /etc/grafana/provisioning/datasources
cat <<EOF > /etc/grafana/provisioning/datasources/ds.yaml
apiVersion: 1
datasources:
- name: Loki
type: loki
access: proxy
url: http://loki:3100
EOF
/run.sh
ports:
- "3000:3000"
depends_on:
- loki
networks:
- loki
promtail:
image: grafana/promtail:2.8.0
volumes:
- ./promtail-local-config.yaml:/etc/promtail/config.yaml:ro
- /var/run/docker.sock:/var/run/docker.sock
command: -config.file=/etc/promtail/config.yaml
networks:
- loki
flog:
image: mingrammer/flog
command: -f json -d 1s -l
networks:
- loki
Run:
docker-compose up -d
Now Grafana is running on http://localhost:3000 and Loki is running on http://localhost:3100. Flog app generates logs, and Promtail collects logs and sends them to Loki.
Open http://localhost:3000 and go to Explore, choose Loki datasource. Select container
from the label filter dropdown and promtail-lokireceiver-flog-1
from the value dropdown.
You will see flog application logs.
Use OpenTelemetry Collector to send logs to Loki
Now let’s change the configuration so Promtail sends logs to the OpenTelemetry collector instead of Loki.
Create otel-config.yaml
file in the example
directory:
receivers:
loki:
protocols:
http:
use_incoming_timestamp: true
processors:
attributes:
actions:
- action: insert
key: loki.attribute.labels
value: container
- action: insert
key: loki.format
value: raw
exporters:
loki:
endpoint: http://loki:3100/loki/api/v1/push
service:
pipelines:
logs:
receivers: [loki]
processors: [attributes]
exporters: [loki]
Here we use attributes processor to promote container
attribute to Loki label and to specify the format used for writing log lines to Loki.
More details on labels and format could be found in Loki exporter README
Add service to the docker-compose.yaml
to run the OpenTelemetry collector in the container:
otelcol:
image: otel/opentelemetry-collector-contrib:0.76.1
deploy:
resources:
limits:
memory: 125M
restart: unless-stopped
command: [ "--config=/etc/otelcol-config.yaml" ]
volumes:
- ./otelcol-config.yaml:/etc/otelcol-config.yaml
ports:
- "3500:3500" # loki receiver HTTP
networks:
- loki
And change the promtail-local-config.yaml
to send logs to the OpenTelemetry collector instead of Loki:
clients:
- url: http://otelcol:3500/loki/api/v1/push
Restart docker-compose:
docker-compose restart
Open http://localhost:3000 and go to Explore, choose Loki datasource. Select container
from the label filter dropdown
and promtail-lokireceiver-flog-1
from the value dropdown. Review this
guidance doc for assistance with using
Explore to query and view Loki data.
You will see the same logs as you saw before, except the new label exporter=OTLP
wad added. But now the backend where these logs are sent is configured in the same place as other signals: in the OpenTelemetry collector config otel-config.yaml
.
Using OpenTelemetry collector as a client of the Promtail allows to ingest loki formatted logs into the collector where they can be transformed to the OTLP format and processed by the collector pipeline. If you like you can change the backend and send your logs to another storage.
Summary
In this guide, we provided all the steps you need to configure Promtail and OpenTelemetry Collector. Promtail scrapes application logs and sends them to the OpenTelemetry collector for further processing. This setup allows us to keep the configuration for all telemetry signals in one place - OpenTelemetry collector config.