Loki Canary is a standalone app that audits the log-capturing performance of a Grafana Loki cluster.
Loki Canary generates artificial log lines. These log lines are sent to the Loki cluster. Loki Canary communicates with the Loki cluster to capture metrics about the artificial log lines, such that Loki Canary forms information about the performance of the Loki cluster. The information is available as Prometheus time series metrics.
Loki Canary writes a log to a file and stores the timestamp in an internal array. The contents look something like this:
The relevant part of the log entry is the timestamp; the
ps are just filler
bytes to make the size of the log configurable.
An agent (like Promtail) should be configured to read the log file and ship it to Loki.
Meanwhile, Loki Canary will open a WebSocket connection to Loki and will tail the logs it creates. When a log is received on the WebSocket, the timestamp in the log message is compared to the internal array.
If the received log is:
- The next in the array to be received, it is removed from the array and the
(current time - log timestamp) is recorded in the
response_latencyhistogram. This is the expected behavior for well behaving logs.
- Not the next in the array to be received, it is removed from the array, the
response time is recorded in the
response_latencyhistogram, and the
out_of_order_entriescounter is incremented.
- Not in the array at all, it is checked against a separate list of received
logs to either increment the
duplicate_entriescounter or the
In the background, Loki Canary also runs a timer which iterates through all of
the entries in the internal array. If any of the entries are older than the
duration specified by the
-wait flag (defaulting to 60s), they are removed
from the array and the
websocket_missing_entries counter is incremented. An
additional query is then made directly to Loki for any missing entries to
determine if they are truly missing or only missing from the WebSocket. If
missing entries are not found in the direct query, the
Starting with version 1.6.0, the canary will spot check certain results over time to make sure they are present in Loki, this is helpful for testing the transition of inmemory logs in the ingester to the store to make sure nothing is lost.
-spot-check-max are used to tune this feature,
-spot-check-interval will pull a log entry from the stream at this interval
and save it in a separate list up to
-spot-check-query-rate, Loki will be queried for each entry in this list and
loki_canary_spot_check_entries_total will be incremented, if a result
loki_canary_spot_check_missing_entries_total will be incremented.
The defaults of
means that after 4 hours of running the canary will have a list of 16 entries
it will query every minute (default
spot-check-query-rate interval is 1m),
so be aware of the query load this can put on Loki if you have a lot of canaries.
NOTE: if you are using
out-of-order-percentage to test ingestion of out-of-order
log lines be sure not to set the two out of order time range flags too far in the past.
The defaults are already enough to test this functionality properly, and setting them
too far in the past can cause issues with the spot check test.
out-of-order-percentage you also need to make use of pipeline stages
in your Promtail configuration in order to set the timestamps correctly as the logs are pushed
to Loki. The
client/promtail/pipelines docs have examples of how to do this.
Loki Canary will run a metric query
verify that the rate of logs being stored in Loki corresponds to the rate they are being
created by Loki Canary.
-metric-test-range are used to tune this feature, but
by default every
15m the canary will run a
count_over_time instant-query to Loki
for a range of
If the canary has not run for
24h) the query range is adjusted
to the amount of time the canary has been running such that the rate can be calculated
since the canary was started.
The canary calculates what the expected count of logs would be for the range
(also adjusting this based on canary runtime) and compares the expected result with
the actual result returned from Loki. The difference is stored as the value in
It’s expected that there will be some deviation, the method of creating an expected calculation based on the query rate compared to actual query data is imperfect and will lead to a deviation of a few log entries.
It’s not expected for there to be a deviation of more than 3-4 log entries.
Loki Canary responds to two endpoints to allow dynamic suspending/resuming of the
canary process. This can be useful if you’d like to quickly disable or reenable the
canary. To stop or start the canary issue an HTTP GET request against the
Loki Canary is provided as a pre-compiled binary as part of the Loki Releases on GitHub.
Loki Canary is also provided as a Docker container image:
# change tag to the most recent release
$ docker pull grafana/loki-canary:2.9.4
To run on Kubernetes, you can do something simple like:
kubectl run loki-canary --generator=run-pod/v1 --image=grafana/loki-canary:latest --restart=Never --image-pull-policy=IfNotPresent --labels=name=loki-canary -- -addr=loki:3100
Or you can do something more complex like deploy it as a DaemonSet, there is a
Tanka setup for this in the
production folder, you can import it using
jb install github.com/grafana/loki-canary/production/ksonnet/loki-canary
Then in your Tanka environment’s
main.jsonnet you’ll want something like
local loki_canary = import 'loki-canary/loki-canary.libsonnet';
Standalone Pod Implementation of loki-canary
- name: metrics
DaemonSet Implementation of loki-canary
- name: metrics
If the other options are not sufficient for your use case, you can compile
# clone the source tree
$ git clone https://github.com/grafana/loki
# build the binary
$ make loki-canary
# (optionally build the container image)
$ make loki-canary-image
The address of Loki must be passed in with the
-addr flag or by setting the
LOKI_ADDRESS, and if your Loki server uses TLS,
must also be provided. Note that using TLS will cause the WebSocket connection
wss:// instead of
-labelvalue flags should also be provided, as these are
used by Loki Canary to filter the log stream to only process logs for the
current instance of the canary. Ensure that the values provided to the flags are
unique to each instance of Loki Canary. Grafana Labs’ Tanka config
accomplishes this by passing in the pod name as the label value.
If Loki Canary reports a high number of
unexpected_entries, Loki Canary may
not be waiting long enough and the value for the
-wait flag should be
increased to a larger value than 60s.
Be aware of the relationship between
pruneinterval and the
For example, with an interval of 10ms (100 logs per second) and a prune interval
of 60s, you will write 6000 logs per minute. If those logs were not received
over the WebSocket, the canary will attempt to query Loki directly to see if
they are completely lost. However the query return is limited to 1000
results so you will not be able to return all the logs even if they did make it
Likewise, if you lower the
pruneinterval you risk causing a denial of
service attack as all your canaries attempt to query for missing logs at
pruneinterval is defined at.
The Loki server URL:Port, e.g. loki:3100. Loki address can also be set using the environment variable LOKI_ADDRESS.
Number of buckets in the response_latency histogram (default 10)
Client certificate authority for optional use with TLS connection to Loki
Client PEM encoded X.509 certificate for optional use with TLS connection to Loki
Allow insecure TLS connections
Duration between log entries (default 1s)
Client PEM encoded X.509 key for optional use with TLS connection to Loki
The label name for this instance of loki-canary to use in the log selector (default "name")
The unique label value for this instance of loki-canary to use in the log selector (default "loki-canary")
Duration to keep querying Loki for missing websocket entries before reporting them missing (default 5m0s)
The interval the metric test query should be run (default 1h0m0s)
The range value [24h] used in the metric test instant-query. Note: this value is truncated to the running time of the canary until this value is reached (default 24h0m0s)
Maximum amount of time to go back for out of order entries (in seconds). (default 1m0s)
Minimum amount of time to go back for out of order entries (in seconds). (default 30s)
Percentage (0-100) of log entries that should be sent out of order.
Loki password. This credential should have both read and write permissions to Loki endpoints
Port which loki-canary should expose metrics (default 3500)
Frequency to check sent vs received logs, also the frequency which queries for missing logs will be dispatched to loki (default 1m0s)
Push the logs directly to given Loki address
How long to wait for a query response from Loki (default 10s)
Size in bytes of each log line (default 100)
How long should the spot check query wait before starting to check for entries (default 10s)
Interval that a single result will be kept from sent entries and spot-checked against Loki, e.g. 15min default one entry every 15 min will be saved and then queried again every 15min until spot-check-max is reached (default 15m0s)
How far back to check a spot check entry before dropping it (default 4h0m0s)
Interval that the canary will query Loki for the current list of all spot check entries (default 1m0s)
The stream name for this instance of loki-canary to use in the log selector (default "stream")
The unique stream value for this instance of loki-canary to use in the log selector (default "stdout")
Tenant ID to be set in X-Scope-OrgID header.
Does the loki connection use TLS?
Print this builds version information
Duration to wait for log entries on websocket before querying loki for them (default 1m0s)
Maximum backoff time between retries (default 5m0s)
Maximum number of retries when push a log entry (default 10)
Initial backoff time before first retry (default 500ms)
How long to wait write response from Loki (default 10s)