This is documentation for the next version of Loki. For the latest stable release, go to the latest version.
Docker driver client configuration
The Docker daemon on each machine has a default logging driver and each container will use the default driver unless configured otherwise.
Installation
Before configuring the plugin, install or upgrade the Grafana Loki Docker Driver Client
Change the logging driver for a container
The docker run
command can be configured to use a different logging driver
than the Docker daemon’s default with the --log-driver
flag. Any options that
the logging driver supports can be set using the --log-opt <NAME>=<VALUE>
flag.
--log-opt
can be passed multiple times for each option to be set.
The following command will start Grafana in a container and send logs to Grafana Cloud, using a batch size of 400 entries and no more than 5 retries if a send fails.
docker run --log-driver=loki \
--log-opt loki-url="https://<user_id>:<password>@logs-us-west1.grafana.net/loki/api/v1/push" \
--log-opt loki-retries=5 \
--log-opt loki-batch-size=400 \
grafana/grafana
Note
The Loki logging driver still uses the json-log driver in combination with sending logs to Loki, this is mainly useful to keep thedocker logs
command working. You can adjust file size and rotation using the respective log optionmax-size
andmax-file
. Keep in mind that default values for these options are not taken from json-log configuration. You can deactivate this behavior by setting the log optionno-file
to true.
Change the default logging driver
If you want the Loki logging driver to be the default for all containers,
change Docker’s daemon.json
file (located in /etc/docker
on Linux) and set
the value of log-driver
to loki
:
{
"debug": true,
"log-driver": "loki"
}
Options for the logging driver can also be configured with log-opts
in the
daemon.json
:
{
"debug" : true,
"log-driver": "loki",
"log-opts": {
"loki-url": "https://<user_id>:<password>@logs-us-west1.grafana.net/loki/api/v1/push",
"loki-batch-size": "400"
}
}
Note
log-opt configuration options in daemon.json must be provided as
strings. Boolean and numeric values (such as the value for loki-batch-size in the example above) must therefore be enclosed in quotes (
"
).
After changing daemon.json
, restart the Docker daemon for the changes to take
effect. All newly created containers from that host will then send logs to Loki via the driver.
Configure the logging driver for a Swarm service or Compose
You can also configure the logging driver for a swarm service
directly in your compose file. This also applies for docker-compose
:
version: "3.7"
services:
logger:
image: grafana/grafana
logging:
driver: loki
options:
loki-url: "https://<user_id>:<password>@logs-prod-us-central1.grafana.net/loki/api/v1/push"
You can then deploy your stack using:
docker stack deploy my_stack_name --compose-file docker-compose.yaml
Or with docker-compose
:
docker-compose -f docker-compose.yaml up
Once deployed, the Grafana service will send its logs to Loki.
Note
Stack name and service name for each swarm service and project name and service name for each compose service are automatically discovered and sent as Loki labels, this way you can filter by them in Grafana.
Labels
Loki can receive a set of labels along with log line. These labels are used to index log entries and query back logs using LogQL stream selector.
By default, the Docker driver will add the following labels to each log line:
filename
: where the log is written to on diskhost
: the hostname where the log has been generatedswarm_stack
,swarm_service
: added when deploying from Docker Swarm.compose_project
,compose_service
: added when deploying with Docker Compose.
Custom labels can be added using the loki-external-labels
, loki-pipeline-stages
,
loki-pipeline-stage-file
, labels
, env
, and env-regex
options. See the
next section for all supported options.
loki-external-labels
have the default value of container_name={{.Name}}
. If you have custom value for loki-external-labels
then that will replace the default value, meaning you won’t have container_name
label unless you explcity add it (e.g: loki-external-labels: "job=docker,container_name={{.Name}}"
.
Pipeline stages
While you can provide loki-pipeline-stage-file
it can be hard to mount the configuration file to the driver root filesystem.
This is why another option loki-pipeline-stages
is available allowing you to pass a list of stages inlined. Pipeline stages are run at last on every lines.
The example docker-compose below configures 2 stages, one to extract level values and one to set it as a label:
version: "3"
services:
grafana:
image: grafana/grafana
logging:
driver: loki
options:
loki-url: http://host.docker.internal:3100/loki/api/v1/push
loki-pipeline-stages: |
- regex:
expression: '(level|lvl|severity)=(?P<level>\w+)'
- labels:
level:
ports:
- "3000:3000"
Note
Note theloki-pipeline-stages: |
letting you keep the indentation correct.
When using docker run you can also pass the value via a string parameter like such:
read -d '' stages << EOF
- regex:
expression: '(level|lvl|severity)=(?P<level>\\\w+)'
- labels:
level:
EOF
docker run --log-driver=loki \
--log-opt loki-url="http://host.docker.internal:3100/loki/api/v1/push" \
--log-opt loki-pipeline-stages="$stages" \
-p 3000:3000 grafana/grafana
This is a bit more difficult as you need to properly escape bash special characters. (note \\\w+
for \w+
)
Providing both loki-pipeline-stage-file
and loki-pipeline-stages
will cause an error.
Relabeling
You can use Prometheus relabeling configuration to modify labels discovered by the driver. The configuration must be passed as a YAML string like the pipeline stages.
Relabeling phase will happen only once per container and it is applied on the container metadata when it starts. So you can for example rename the labels that are only available during the starting of the container, not the labels available on log lines. Use pipeline stages instead.
For example the configuration below will rename the label swarm_stack
and swarm_service
to respectively namespace
and service
.
version: "3"
services:
grafana:
image: grafana/grafana
logging:
driver: loki
options:
loki-url: http://host.docker.internal:3100/loki/api/v1/push
loki-relabel-config: |
- action: labelmap
regex: swarm_stack
replacement: namespace
- action: labelmap
regex: swarm_(service)
ports:
- "3000:3000"
Supported log-opt options
To specify additional logging driver options, you can use the –log-opt NAME=VALUE flag.
Troubleshooting
Plugin logs can be found as docker daemon log. To enable debug mode refer to the Docker daemon documentation.
The standard output (stdout
) of a plugin is redirected to Docker logs. Such
entries are prefixed with plugin=
.
To find out the plugin ID of the Loki logging driver, use docker plugin ls
and
look for the loki
entry.
Depending on your system, location of Docker daemon logging may vary. Refer to Docker documentation for Docker daemon log location for your specific platform.