Grafana Loki on Grafana Labs

LogCLI

Tue, 16 Jul 2024 15:42:20 +0000

LogCLI, Grafana Loki’s command-line interface

LogCLI is the command-line interface to Grafana Loki. It facilitates running LogQL queries against a Loki instance.

Installation

Binary (Recommended)

Download the logcli binary from the Loki releases page.

Build LogCLI from source

Clone the Loki repository and build logcli from source:

git clone https://github.com/grafana/loki.git
cd loki
make logcli

Optionally, move the binary into a directory that is part of your $PATH.

cp cmd/logcli/logcli /usr/local/bin/logcli

LogCLI usage

Grafana Cloud example

If you are running on Grafana Cloud, use:

export LOKI_ADDR=https://logs-us-west1.grafana.net
export LOKI_USERNAME=<username>
export LOKI_PASSWORD=<password>

Otherwise you can point LogCLI to a local instance directly without needing a username and password:

export LOKI_ADDR=http://localhost:3100

Note: If you are running Loki behind a proxy server and you have authentication configured, you will also have to pass in LOKI_USERNAME and LOKI_PASSWORD, LOKI_BEARER_TOKEN or LOKI_BEARER_TOKEN_FILE accordingly.

$ logcli labels job
https://logs-dev-ops-tools1.grafana.net/api/prom/label/job/values
loki-ops/consul
loki-ops/loki-gw
...

$ logcli query '{job="loki-ops/consul"}'
https://logs-dev-ops-tools1.grafana.net/api/prom/query?query=%7Bjob%3D%22loki-ops%2Fconsul%22%7D&limit=30&start=1529928228&end=1529931828&direction=backward&regexp=
Common labels: {job="loki-ops/consul", namespace="loki-ops"}
2018-06-25T12:52:09Z {instance="consul-8576459955-pl75w"} 2018/06/25 12:52:09 [INFO] raft: Snapshot to 475409 complete
2018-06-25T12:52:09Z {instance="consul-8576459955-pl75w"} 2018/06/25 12:52:09 [INFO] raft: Compacting logs from 456973 to 465169
...

$ logcli series -q --match='{namespace="loki",container_name="loki"}'
{app="loki", container_name="loki", controller_revision_hash="loki-57c9df47f4", filename="/var/log/pods/loki_loki-0_8ed03ded-bacb-4b13-a6fe-53a445a15887/loki/0.log", instance="loki-0", job="loki/loki", name="loki", namespace="loki", release="loki", statefulset_kubernetes_io_pod_name="loki-0", stream="stderr"}

Batched queries

LogCLI sends queries to Loki such that query results arrive in batches.

The --limit option for a logcli query command caps the quantity of log lines for a single query. When not set, --limit defaults to 30. The limit protects the user from overwhelming the system for cases in which the specified query would have returned a large quantity of log lines. The limit also protects the user from unexpectedly large responses.

The quantity of log line results that arrive in each batch is set by the --batch option in a logcli query command. When not set, --batch defaults to 1000.

Setting a --limit value larger than the --batch value causes the requests from LogCLI to Loki to be batched. Loki has a server-side limit that defaults to 5000 for the maximum quantity of lines returned for a single query. The batching of requests allows you to query for a results set that is larger than the server-side limit, as long as the --batch value is less than the server limit.

Query metadata is output to stderr for each batch. Set the --quiet option on the logcli query command line to suppress the output of the query metadata.

Configuration

Configuration values are considered in the following order (lowest to highest):

Environment variables
Command-line options

LogCLI command reference

The output of logcli help:

usage: logcli [<flags>] <command> [<args> ...]

A command-line for loki.

Flags:
      --help             Show context-sensitive help (also try --help-long and
                         --help-man).
      --version          Show application version.
  -q, --quiet            Suppress query metadata
      --stats            Show query statistics
  -o, --output=default   Specify output mode [default, raw, jsonl]. raw
                         suppresses log labels and timestamp.
  -z, --timezone=Local   Specify the timezone to use when formatting output
                         timestamps [Local, UTC]
      --cpuprofile=""    Specify the location for writing a CPU profile.
      --memprofile=""    Specify the location for writing a memory profile.
      --stdin            Take input logs from stdin
      --addr="http://localhost:3100"
                         Server address. Can also be set using LOKI_ADDR env
                         var.
      --username=""      Username for HTTP basic auth. Can also be set using
                         LOKI_USERNAME env var.
      --password=""      Password for HTTP basic auth. Can also be set using
                         LOKI_PASSWORD env var.
      --ca-cert=""       Path to the server Certificate Authority. Can also be
                         set using LOKI_CA_CERT_PATH env var.
      --tls-skip-verify  Server certificate TLS skip verify.
      --cert=""          Path to the client certificate. Can also be set using
                         LOKI_CLIENT_CERT_PATH env var.
      --key=""           Path to the client certificate key. Can also be set
                         using LOKI_CLIENT_KEY_PATH env var.
      --org-id=""        adds X-Scope-OrgID to API requests for representing
                         tenant ID. Useful for requesting tenant data when
                         bypassing an auth gateway.

Commands:
  help [<command>...]
    Show help.

  query [<flags>] <query>
    Run a LogQL query.

    The "query" command is useful for querying for logs. Logs can be returned in
    a few output modes:

      raw: log line
      default: log timestamp + log labels + log line
      jsonl: JSON response from Loki API of log line

    The output of the log can be specified with the "-o" flag, for example, "-o
    raw" for the raw output format.

    The "query" command will output extra information about the query and its
    results, such as the API URL, set of common labels, and set of excluded
    labels. This extra information can be suppressed with the --quiet flag.

    By default we look over the last hour of data; use --since to modify or
    provide specific start and end times with --from and --to respectively.

    Notice that when using --from and --to then ensure to use RFC3339Nano time
    format, but without timezone at the end. The local timezone will be added
    automatically or if using --timezone flag.

    Example:

      logcli query
         --timezone=UTC
         --from="2021-01-19T10:00:00Z"
         --to="2021-01-19T20:00:00Z"
         --output=jsonl
         'my-query'

    The output is limited to 30 entries by default; use --limit to increase.

    While "query" does support metrics queries, its output contains multiple
    data points between the start and end query time. This output is used to
    build graphs, similar to what is seen in the Grafana Explore graph view. If
    you are querying metrics and just want the most recent data point (like what
    is seen in the Grafana Explore table view), then you should use the
    "instant-query" command instead.

  instant-query [<flags>] <query>
    Run an instant LogQL query.

    The "instant-query" command is useful for evaluating a metric query for a
    single point in time. This is equivalent to the Grafana Explore table view;
    if you want a metrics query that is used to build a Grafana graph, you
    should use the "query" command instead.

    This command does not produce useful output when querying for log lines; you
    should always use the "query" command when you are running log queries.

    For more information about log queries and metric queries, refer to the
    LogQL documentation:

    https://grafana.com/docs/loki/latest/logql/

  labels [<flags>] [<label>]
    Find values for a given label.

  series [<flags>] <matcher>
    Run series query.

    The "series" command will take the provided label matcher and return all the
    log streams found in the time window.

    It is possible to send an empty label matcher '{}' to return all streams.

    Use the --analyze-labels flag to get a summary of the labels found in all
    streams. This is helpful to find high cardinality labels.

LogCLI query command reference

The output of logcli help query:

usage: logcli query [<flags>] <query>

Run a LogQL query.

The "query" command is useful for querying for logs. Logs can be returned in a
few output modes:

  raw: log line
  default: log timestamp + log labels + log line
  jsonl: JSON response from Loki API of log line

The output of the log can be specified with the "-o" flag, for example, "-o raw"
for the raw output format.

The "query" command will output extra information about the query and its
results, such as the API URL, set of common labels, and set of excluded labels.
This extra information can be suppressed with the --quiet flag.

By default we look over the last hour of data; use --since to modify or provide
specific start and end times with --from and --to respectively.

Notice that when using --from and --to then ensure to use RFC3339Nano time
format, but without timezone at the end. The local timezone will be added
automatically or if using --timezone flag.

Example:

  logcli query
     --timezone=UTC
     --from="2021-01-19T10:00:00Z"
     --to="2021-01-19T20:00:00Z"
     --output=jsonl
     'my-query'

The output is limited to 30 entries by default; use --limit to increase.

While "query" does support metrics queries, its output contains multiple data
points between the start and end query time. This output is used to build
graphs, similar to what is seen in the Grafana Explore graph view. If you are
querying metrics and just want the most recent data point (like what is seen in
the Grafana Explore table view), then you should use the "instant-query" command
instead.

Flags:
      --help               Show context-sensitive help (also try --help-long and
                           --help-man).
      --version            Show application version.
  -q, --quiet              Suppress query metadata
      --stats              Show query statistics
  -o, --output=default     Specify output mode [default, raw, jsonl]. raw
                           suppresses log labels and timestamp.
  -z, --timezone=Local     Specify the timezone to use when formatting output
                           timestamps [Local, UTC]
      --cpuprofile=""      Specify the location for writing a CPU profile.
      --memprofile=""      Specify the location for writing a memory profile.
      --stdin              Take input logs from stdin
      --addr="http://localhost:3100"
                           Server address. Can also be set using LOKI_ADDR env
                           var.
      --username=""        Username for HTTP basic auth. Can also be set using
                           LOKI_USERNAME env var.
      --password=""        Password for HTTP basic auth. Can also be set using
                           LOKI_PASSWORD env var.
      --ca-cert=""         Path to the server Certificate Authority. Can also be
                           set using LOKI_CA_CERT_PATH env var.
      --tls-skip-verify    Server certificate TLS skip verify.
      --cert=""            Path to the client certificate. Can also be set using
                           LOKI_CLIENT_CERT_PATH env var.
      --key=""             Path to the client certificate key. Can also be set
                           using LOKI_CLIENT_KEY_PATH env var.
      --org-id=""          adds X-Scope-OrgID to API requests for representing
                           tenant ID. Useful for requesting tenant data when
                           bypassing an auth gateway.
      --limit=30           Limit on number of entries to print.
      --since=1h           Lookback window.
      --from=FROM          Start looking for logs at this absolute time
                           (inclusive)
      --to=TO              Stop looking for logs at this absolute time
                           (exclusive)
      --step=STEP          Query resolution step width, for metric queries.
                           Evaluate the query at the specified step over the
                           time range.
      --interval=INTERVAL  Query interval, for log queries. Return entries at
                           the specified interval, ignoring those between.
                           **This parameter is experimental, please see Issue
                           1779**
      --batch=1000         Query batch size to use until 'limit' is reached
      --forward            Scan forwards through logs.
      --no-labels          Do not print any labels
      --exclude-label=EXCLUDE-LABEL ...
                           Exclude labels given the provided key during output.
      --include-label=INCLUDE-LABEL ...
                           Include labels given the provided key during output.
      --labels-length=0    Set a fixed padding to labels
      --store-config=""    Execute the current query using a configured storage
                           from a given Loki configuration file.
      --colored-output     Show output with colored labels
  -t, --tail               Tail the logs
  -f, --follow             Alias for --tail
      --delay-for=0        Delay in tailing by number of seconds to accumulate
                           logs for re-ordering

Args:
  <query>  eg '{foo="bar",baz=~".*blip"} |~ ".*error.*"'

LogCLI labels command reference

The output of logcli help labels:

usage: logcli labels [<flags>] [<label>]

Find values for a given label.

Flags:
      --help             Show context-sensitive help (also try --help-long and
                         --help-man).
      --version          Show application version.
  -q, --quiet            Suppress query metadata
      --stats            Show query statistics
  -o, --output=default   Specify output mode [default, raw, jsonl]. raw
                         suppresses log labels and timestamp.
  -z, --timezone=Local   Specify the timezone to use when formatting output
                         timestamps [Local, UTC]
      --cpuprofile=""    Specify the location for writing a CPU profile.
      --memprofile=""    Specify the location for writing a memory profile.
      --stdin            Take input logs from stdin
      --addr="http://localhost:3100"
                         Server address. Can also be set using LOKI_ADDR env
                         var.
      --username=""      Username for HTTP basic auth. Can also be set using
                         LOKI_USERNAME env var.
      --password=""      Password for HTTP basic auth. Can also be set using
                         LOKI_PASSWORD env var.
      --ca-cert=""       Path to the server Certificate Authority. Can also be
                         set using LOKI_CA_CERT_PATH env var.
      --tls-skip-verify  Server certificate TLS skip verify.
      --cert=""          Path to the client certificate. Can also be set using
                         LOKI_CLIENT_CERT_PATH env var.
      --key=""           Path to the client certificate key. Can also be set
                         using LOKI_CLIENT_KEY_PATH env var.
      --org-id=""        adds X-Scope-OrgID to API requests for representing
                         tenant ID. Useful for requesting tenant data when
                         bypassing an auth gateway.
      --since=1h         Lookback window.
      --from=FROM        Start looking for labels at this absolute time
                         (inclusive)
      --to=TO            Stop looking for labels at this absolute time
                         (exclusive)

Args:
  [<label>]  The name of the label.

LogCLI series command reference

The output of logcli help series:

usage: logcli series [<flags>] <matcher>

Run series query.

The "series" command will take the provided label matcher and return all the log
streams found in the time window.

It is possible to send an empty label matcher '{}' to return all streams.

Use the --analyze-labels flag to get a summary of the labels found in all
streams. This is helpful to find high cardinality labels.

Flags:
      --help             Show context-sensitive help (also try --help-long and
                         --help-man).
      --version          Show application version.
  -q, --quiet            Suppress query metadata
      --stats            Show query statistics
  -o, --output=default   Specify output mode [default, raw, jsonl]. raw
                         suppresses log labels and timestamp.
  -z, --timezone=Local   Specify the timezone to use when formatting output
                         timestamps [Local, UTC]
      --cpuprofile=""    Specify the location for writing a CPU profile.
      --memprofile=""    Specify the location for writing a memory profile.
      --stdin            Take input logs from stdin
      --addr="http://localhost:3100"
                         Server address. Can also be set using LOKI_ADDR env
                         var.
      --username=""      Username for HTTP basic auth. Can also be set using
                         LOKI_USERNAME env var.
      --password=""      Password for HTTP basic auth. Can also be set using
                         LOKI_PASSWORD env var.
      --ca-cert=""       Path to the server Certificate Authority. Can also be
                         set using LOKI_CA_CERT_PATH env var.
      --tls-skip-verify  Server certificate TLS skip verify.
      --cert=""          Path to the client certificate. Can also be set using
                         LOKI_CLIENT_CERT_PATH env var.
      --key=""           Path to the client certificate key. Can also be set
                         using LOKI_CLIENT_KEY_PATH env var.
      --org-id=""        adds X-Scope-OrgID to API requests for representing
                         tenant ID. Useful for requesting tenant data when
                         bypassing an auth gateway.
      --since=1h         Lookback window.
      --from=FROM        Start looking for logs at this absolute time
                         (inclusive)
      --to=TO            Stop looking for logs at this absolute time (exclusive)
      --analyze-labels   Printout a summary of labels including count of label
                         value combinations, useful for debugging high
                         cardinality series

Args:
  <matcher>  eg '{foo="bar",baz=~".*blip"}'

LogCLI `--stdin` usage

You can consume log lines from your stdin instead of Loki servers.

Say you have log files in your local, and just want to do run some LogQL queries for that, --stdin flag can help.

NOTE: Currently it doesn’t support any type of metric queries

You may have to use stdin flag for several reasons

Quick way to check and validate a LogQL expressions.
Learn basics of LogQL with just Log files and LogCLItool ( without needing set up Loki servers, Grafana etc.)
Easy discussion on public forums. Like Q&A, Share the LogQL expressions.

NOTES on Usage

--limits flag doesn’t have any meaning when using --stdin (use pager like less for that)
Be aware there are no labels when using --stdin
- So stream selector in the query is optional e.g just |="timeout"|logfmt|level="error" is same as {foo="bar"}|="timeout|logfmt|level="error"

Examples

Line filter - cat mylog.log | logcli --stdin query '|="too many open connections"'
Label matcher - echo 'msg="timeout happened" level="warning"' | logcli --stdin query '|logfmt|level="warning"'
Different parsers (logfmt, json, pattern, regexp) - cat mylog.log | logcli --stdin query '|pattern <ip> - - <_> "<method> <uri> <_>" <status> <size> <_> "<agent>" <_>'
Line formatters - cat mylog.log | logcli --stdin query '|logfmt|line_format "{{.query}} {{.duration}}"'

Fundamentals

Tue, 16 Jul 2024 15:42:20 +0000

Grafana Loki Fundamentals

Configuration

Tue, 16 Jul 2024 15:42:20 +0000

Configuring Grafana Loki

Grafana Loki is configured in a YAML file (usually referred to as loki.yaml ) which contains information on the Loki server and its individual components, depending on which mode Loki is launched in.

Configuration examples can be found in the Configuration Examples document.

Printing Loki Config At Runtime

If you pass Loki the flag -print-config-stderr or -log-config-reverse-order, (or -print-config-stderr=true) Loki will dump the entire config object it has created from the built in defaults combined first with overrides from config file, and second by overrides from flags.

The result is the value for every config object in the Loki config struct, which is very large…

Many values will not be relevant to your install such as storage configs which you are not using and which you did not define, this is expected as every option has a default value if it is being used or not.

This config is what Loki will use to run, it can be invaluable for debugging issues related to configuration and is especially useful in making sure your config files and flags are being read and loaded properly.

-print-config-stderr is nice when running Loki directly e.g. ./loki as you can get a quick output of the entire Loki config.

-log-config-reverse-order is the flag we run Loki with in all our environments, the config entries are reversed so that the order of configs reads correctly top to bottom when viewed in Grafana’s Explore.

Configuration File Reference

To specify which configuration file to load, pass the -config.file flag at the command line. The value can be a list of comma separated paths, then the first file that exists will be used. If no -config.file argument is specified, Loki will look up the config.yaml in the current working directory and the config/ sub-directory and try to use that.

The file is written in YAML format, defined by the scheme below. Brackets indicate that a parameter is optional. For non-list parameters the value is set to the specified default.

Use environment variables in the configuration

Note: This feature is only available in Loki 2.1+.

You can use environment variable references in the configuration file to set values that need to be configurable during deployment. To do this, pass -config.expand-env=true and use:

${VAR}

Where VAR is the name of the environment variable.

Each variable reference is replaced at startup by the value of the environment variable. The replacement is case-sensitive and occurs before the YAML file is parsed. References to undefined variables are replaced by empty strings unless you specify a default value or custom error text.

To specify a default value, use:

${VAR:-default_value}

Where default_value is the value to use if the environment variable is undefined.

Pass the -config.expand-env flag at the command line to enable this way of setting configs.

Generic placeholders

<boolean> : a boolean that can take the values true or false
<int> : any integer matching the regular expression [1-9]+[0-9]*
<duration> : a duration matching the regular expression [0-9]+(ns|us|µs|ms|[smh])
<labelname> : a string matching the regular expression [a-zA-Z_][a-zA-Z0-9_]*
<labelvalue> : a string of unicode characters
<filename> : a valid path relative to current working directory or an absolute path.
<host> : a valid string consisting of a hostname or IP followed by an optional port number
<string> : a string
<secret> : a string that represents a secret, such as a password

Supported contents and default values of `loki.yaml`

# A comma-separated list of components to run.
# The default value "all" runs Loki in single binary mode.
# The value "read" is an alias to run only read-path related components such as
# the querier and query-frontend, but all in the same process.
# The value "write" is an alias to run only write-path related components such as
# the distributor and compactor, but all in the same process.
# Supported values: all, compactor, distributor, ingester, querier, query-scheduler,
#  ingester-querier, query-frontend, index-gateway, ruler, table-manager, read, write.
# A full list of available targets can be printed when running Loki with the
# `-list-targets` command line flag.
[target: <string> | default = "all"]

# Enables authentication through the X-Scope-OrgID header, which must be present
# if true. If false, the OrgID will always be set to "fake".
[auth_enabled: <boolean> | default = true]

# The amount of virtual memory in bytes to reserve as ballast in order to optimize
# garbage collection. Larger ballasts result in fewer garbage collection passes, reducing
# CPU overhead at the cost of heap size. The ballast will not consume physical memory,
# because it is never read from. It will, however, distort metrics, because it is
# counted as live memory.
[ballast_bytes: <int> | default = 0]

# Configures the server of the launched module(s).
[server: <server>]

# Configures the distributor.
[distributor: <distributor>]

# Configures the querier. Only appropriate when running all modules or
# just the querier.
[querier: <querier>]

# The query_scheduler block configures the Loki query scheduler.
# When configured it separates the tenant query queues from the query-frontend
[query_scheduler: <query_scheduler>]

# The frontend block configures the Loki query-frontend.
[frontend: <frontend>]

# The query_range block configures the query splitting and caching in the Loki
# query-frontend.
[query_range: <query_range>]

# The ruler block configures the Loki ruler.
[ruler: <ruler>]

# The ingester_client block configures how the distributor will connect
# to ingesters. Only appropriate when running all components, the distributor,
# or the querier.
[ingester_client: <ingester_client>]

# The ingester block configures the ingester and how the ingester will register itself
# to a key value store.
[ingester: <ingester>]

# Configures the index gateway server.
[index_gateway: <index_gateway>]

# Configures where Loki will store data.
[storage_config: <storage_config>]

# Configures how Loki will store data in the specific store.
[chunk_store_config: <chunk_store_config>]

# Configures the chunk index schema and where it is stored.
[schema_config: <schema_config>]

# The compactor block configures the compactor component, which compacts index shards
# for performance.
[compactor: <compactor>]

# Configures limits per-tenant or globally.
[limits_config: <limits_config>]

# The frontend_worker configures the worker - running within the Loki
# querier - picking up and executing queries enqueued by the query-frontend.
[frontend_worker: <frontend_worker>]

# The table_manager block configures the table manager for retention.
[table_manager: <table_manager>]

# Configuration for "runtime config" module, responsible for reloading runtime
# configuration file.
[runtime_config: <runtime_config>]

# Configuration for tracing.
[tracing: <tracing>]

# Common configuration to be shared between multiple modules.
# If a more specific configuration is given in other sections,
# the related configuration within this section will be ignored.
[common: <common>]

# Configuration for usage report
[analytics: <analytics>]

server

The server block configures the HTTP and gRPC server communication of the launched service(s).

# HTTP server listen host
# CLI flag: -server.http-listen-address
[http_listen_address: <string>]

# HTTP server listen port
# CLI flag: -server.http-listen-port
[http_listen_port: <int> | default = 80]

# gRPC server listen host
# CLI flag: -server.grpc-listen-address
[grpc_listen_address: <string>]

# gRPC server listen port
# CLI flag: -server.grpc-listen-port
[grpc_listen_port: <int> | default = 9095]

# Register instrumentation handlers (/metrics, etc.)
# CLI flag: -server.register-instrumentation
[register_instrumentation: <boolean> | default = true]

# Timeout for graceful shutdowns
# CLI flag: -server.graceful-shutdown-timeout
[graceful_shutdown_timeout: <duration> | default = 30s]

# Read timeout for HTTP server
# CLI flag: -server.http-read-timeout
[http_server_read_timeout: <duration> | default = 30s]

# Write timeout for HTTP server
# CLI flag: -server.http-write-timeout
[http_server_write_timeout: <duration> | default = 30s]

# Idle timeout for HTTP server
# CLI flag: -server.http-idle-timeout
[http_server_idle_timeout: <duration> | default = 120s]

# Max gRPC message size that can be received
# CLI flag: -server.grpc-max-recv-msg-size-bytes
[grpc_server_max_recv_msg_size: <int> | default = 4194304]

# Max gRPC message size that can be sent
# CLI flag: -server.grpc-max-send-msg-size-bytes
[grpc_server_max_send_msg_size: <int> | default = 4194304]

# Limit on the number of concurrent streams for gRPC calls (0 = unlimited)
# CLI flag: -server.grpc-max-concurrent-streams
[grpc_server_max_concurrent_streams: <int> | default = 100]

# Log only messages with the given severity or above. Supported values [debug,
# info, warn, error]
# CLI flag: -log.level
[log_level: <string> | default = "info"]

# Base path to serve all API routes from (e.g., /v1/).
# CLI flag: -server.path-prefix
[http_path_prefix: <string> | default = ""]

distributor

The distributor block configures the distributor component.

# Configures the distributors ring, used when the "global" ingestion rate
# strategy is enabled.
ring:
  kvstore:
    # The backend storage to use for the ring. Supported values are
    # consul, etcd, inmemory, memberlist
    # CLI flag: -distributor.ring.store
    store: <string>

    # The prefix for the keys in the store. Should end with a /.
    # CLI flag: -distributor.ring.prefix
    [prefix: <string> | default = "collectors/"]

    # Configuration for a Consul client. Only applies if store is "consul"
    # The CLI flags prefix for this block config is: distributor.ring
    [consul: <consul_config>]

    # Configuration for an ETCD v3 client. Only applies if store is "etcd"
    # The CLI flags prefix for this block config is: distributor.ring
    [etcd: <etcd_config>]

  # The heartbeat timeout after which ingesters are skipped for
  # reading and writing.
  # CLI flag: -distributor.ring.heartbeat-timeout
  [heartbeat_timeout: <duration> | default = 1m]

querier

The querier block configures the Loki Querier.

# Timeout when querying ingesters or storage during the execution of a query request.
# CLI flag: -querier.query-timeout
[query_timeout: <duration> | default = 1m]

# Maximum duration for which the live tailing requests should be served.
# CLI flag: -querier.tail-max-duration
[tail_max_duration: <duration> | default = 1h]

# Time to wait before sending more than the minimum successful query requests.
# CLI flag: -querier.extra-query-delay
[extra_query_delay: <duration> | default = 0s]

# Maximum lookback beyond which queries are not sent to ingester.
# 0 means all queries are sent to ingester.
# CLI flag: -querier.query-ingesters-within
[query_ingesters_within: <duration> | default = 3h]

# The maximum number of concurrent queries allowed.
# CLI flag: -querier.max-concurrent
[max_concurrent: <int> | default = 10]

# Only query the store, and not attempt any ingesters.
# This is useful for running a standalone querier pool operating only against
# stored data.
# CLI flag: -querier.query-store-only
[query_store_only: <boolean> | default = false]

# When true, queriers only query the ingesters, and not stored data.
# This is useful when the object store is unavailable.
# CLI flag: -querier.query-ingester-only
[query_ingester_only: <boolean> | default = false]

# When true, allow queries to span multiple tenants.
# CLI flag: -querier.multi-tenant-queries-enabled
[multi_tenant_queries_enabled: <boolean> | default = false]

# Configuration options for the LogQL engine.
engine:
  # Timeout for query execution
  # CLI flag: -querier.engine.timeout
  [timeout: <duration> | default = 3m]

  # The maximum amount of time to look back for log lines. Only
  # applicable for instant log queries.
  # CLI flag: -querier.engine.max-lookback-period
  [max_look_back_period: <duration> | default = 30s]

query_scheduler

The query_scheduler block configures the Loki query scheduler.

# Maximum number of outstanding requests per tenant per query-scheduler.
# In-flight requests above this limit will fail with HTTP response status code
# 429.
# CLI flag: -query-scheduler.max-outstanding-requests-per-tenant
[max_outstanding_requests_per_tenant: <int> | default = 100]

# If a querier disconnects without sending notification about graceful shutdown,
# the query-scheduler will keep the querier in the tenant's shard until the forget delay has passed.
# This feature is useful to reduce the blast radius when shuffle-sharding is enabled.
# CLI flag: -query-scheduler.querier-forget-delay
[querier_forget_delay: <duration> | default = 0]

# This configures the gRPC client used to report errors back to the
# query-frontend.
[grpc_client_config: <grpc_client_config>]

# Set to true to have the query schedulers create and place themselves in a ring.
# If no frontend_address or scheduler_address are present
# anywhere else in the configuration, Loki will toggle this value to true.
[use_scheduler_ring: <boolean> | default = false]

# The hash ring configuration. This option is required only if use_scheduler_ring is true
# The CLI flags prefix for this block config is scheduler.ring
[scheduler_ring: <ring>]

frontend

The frontend block configures the Loki query-frontend.

# Maximum number of outstanding requests per tenant per frontend; requests
# beyond this error with HTTP 429.
# CLI flag: -querier.max-outstanding-requests-per-tenant
[max_outstanding_per_tenant: <int> | default = 2048]

# In the event a tenant is repeatedly sending queries that lead the querier to crash
# or be killed due to an out-of-memory error, the crashed querier will be disconnected
# from the query frontend and a new querier will be immediately assigned to the tenant’s shard.
# This invalidates the assumption that shuffle sharding can be used to reduce the
# impact on tenants. This option mitigates the impact by configuring a delay between when
# a querier disconnects because of a crash and when the crashed querier is actually removed
# from the tenant's shard.
# CLI flag: -query-frontend.querier-forget-delay
[querier_forget_delay: <duration> | default = 0s]

# Compress HTTP responses.
# CLI flag: -querier.compress-http-responses
[compress_responses: <boolean> | default = false]

# URL of downstream Loki.
# CLI flag: -frontend.downstream-url
[downstream_url: <string> | default = ""]

# Log queries that are slower than the specified duration. Set to 0 to disable.
# Set to < 0 to enable on all queries.
# CLI flag: -frontend.log-queries-longer-than
[log_queries_longer_than: <duration> | default = 0s]

# URL of querier for tail proxy.
# CLI flag: -frontend.tail-proxy-url
[tail_proxy_url: <string> | default = ""]

tail_tls_config:
  # Path to the client certificate file, which will be used for authenticating
  # with the server. Also requires the key path to be configured.
  # CLI flag: -frontend.tail-tls-config.tls-cert-path
  [tls_cert_path: <string> | default = ""]
  
  # Path to the key file for the client certificate. Also requires the client
  # certificate to be configured.
  # CLI flag: -frontend.tail-tls-config.tls-key-path
  [tls_key_path: <string> | default = ""]
  # Path to the CA certificates file to validate server certificate against. If
  # not set, the host's root CA certificates are used.
  # CLI flag: -frontend.tail-tls-config.tls-ca-path
  [tls_ca_path: <string> | default = ""]
  # Skip validating server certificate.
  # CLI flag: -frontend.tail-tls-config.tls-insecure-skip-verify
  [tls_insecure_skip_verify: <boolean> | default = false]

# DNS hostname used for finding query-schedulers.
# CLI flag: -frontend.scheduler-address
[scheduler_address: <string> | default = ""]

# How often to resolve the scheduler-address, in order to look for new
# query-scheduler instances.
# Also used to determine how often to poll the scheduler-ring for addresses if configured.
# CLI flag: -frontend.scheduler-dns-lookup-period
[scheduler_dns_lookup_period: <duration> | default = 10s]

# Number of concurrent workers forwarding queries to single query-scheduler.
# CLI flag: -frontend.scheduler-worker-concurrency
[scheduler_worker_concurrency: <int> | default = 5]

query_range

The query_range block configures query splitting and caching in the Loki query-frontend.

# Deprecated: Split queries by day and execute in parallel.
# Use -querier.split-queries-by-interval instead.
# CLI flag: -querier.split-queries-by-day
[split_queries_by_day: <boolean> | default = false]

# Mutate incoming queries to align their start and end with their step.
# CLI flag: -querier.align-querier-with-step
[align_queries_with_step: <boolean> | default = false]

results_cache:
  # The CLI flags prefix for this block config is: frontend
  cache: <cache_config>

# Cache query results.
# CLI flag: -querier.cache-results
[cache_results: <boolean> | default = false]

# Maximum number of retries for a single request; beyond this, the downstream
# error is returned.
# CLI flag: -querier.max-retries-per-request
[max_retries: <int> | default = 5]

# Perform query parallelisations based on storage sharding configuration and
# query ASTs. This feature is supported only by the chunks storage engine.
# CLI flag: -querier.parallelise-shardable-queries
[parallelise_shardable_queries: <boolean> | default = true]

ruler

The ruler block configures the Loki ruler.

# URL of alerts return path.
# CLI flag: -ruler.external.url
[external_url: <url> | default = ]

# Labels to add to all alerts
external_labels:
  [<labelname>: <labelvalue> ...]

ruler_client:
  # Path to the client certificate file, which will be used for authenticating
  # with the server. Also requires the key path to be configured.
  # CLI flag: -ruler.client.tls-cert-path
  [tls_cert_path: <string> | default = ""]

  # Path to the key file for the client certificate. Also requires the client
  # certificate to be configured.
  # CLI flag: -ruler.client.tls-key-path
  [tls_key_path: <string> | default = ""]

  # Path to the CA certificates file to validate server certificate against. If
  # not set, the host's root CA certificates are used.
  # CLI flag: -ruler.client.tls-ca-path
  [tls_ca_path: <string> | default = ""]

  # Skip validating server certificate.
  # CLI flag: -ruler.client.tls-insecure-skip-verify
  [tls_insecure_skip_verify: <boolean> | default = false]

# How frequently to evaluate rules.
# CLI flag: -ruler.evaluation-interval
[evaluation_interval: <duration> | default = 1m]

# How frequently to poll for rule changes.
# CLI flag: -ruler.poll-interval
[poll_interval: <duration> | default = 1m]

storage:
  # Method to use for backend rule storage (azure, gcs, s3, swift, local, bos).
  # CLI flag: -ruler.storage.type
  [type: <string> ]

  # Configures backend rule storage for Azure.
  [azure: <azure_storage_config>]

  # Configures backend rule storage for GCS.
  [gcs: <gcs_storage_config>]

  # Configures backend rule storage for S3.
  [s3: <s3_storage_config>]

  # Configures backend rule storage for Swift.
  [swift: <swift_storage_config>]

  # Configures backend rule storage for a local file system directory.
  [local: <local_storage_config>]

  # Configures backend rule storage for Baidu Object Storage (BOS).
  [bos: <bos_storage_config>]

  # The `hedging` block configures how to hedge storage requests.
  [hedging: <hedging>]

# Remote-write configuration to send rule samples to a Prometheus remote-write endpoint.
remote_write:
  # Enable remote-write functionality.
  # CLI flag: -ruler.remote-write.enabled
  [enabled: <boolean> | default = false]
  # Minimum period to wait between refreshing remote-write reconfigurations.
  # This should be greater than or equivalent to -limits.per-user-override-period.
  [config_refresh_period: <duration> | default = 10s]

  client:
    # The URL of the endpoint to send samples to.
    url: <string>

    # Timeout for requests to the remote write endpoint.
    [remote_timeout: <duration> | default = 30s]

    # Custom HTTP headers to be sent along with each remote write request.
    # Be aware that headers that are set by Loki itself can't be overwritten.
    headers:
      [<string>: <string> ...]

    # List of remote write relabel configurations.
    write_relabel_configs:
      [- <relabel_config> ...]

    # Name of the remote write config, which if specified must be unique among remote
    # write configs.
    # The name will be used in metrics and logging in place of a generated value
    # to help users distinguish between remote write configs.
    [name: <string>]

    # Sets the `Authorization` header on every remote write request with the
    # configured username and password.
    # password and password_file are mutually exclusive.
    basic_auth:
      [username: <string>]
      [password: <secret>]
      [password_file: <string>]

    # Optional `Authorization` header configuration.
    authorization:
      # Sets the authentication type.
      [type: <string> | default: Bearer]
      # Sets the credentials. It is mutually exclusive with
      # `credentials_file`.
      [credentials: <secret>]
      # Sets the credentials to the credentials read from the configured file.
      # It is mutually exclusive with `credentials`.
      [credentials_file: <filename>]

    # Optionally configures AWS's Signature Verification 4 signing process to
    # sign requests. Cannot be set at the same time as basic_auth, authorization, or oauth2.
    # To use the default credentials from the AWS SDK, use `sigv4: {}`.
    sigv4:
      # The AWS region. If blank, the region from the default credentials chain
      # is used.
      [region: <string>]

      # The AWS API keys. If blank, the environment variables `AWS_ACCESS_KEY_ID`
      # and `AWS_SECRET_ACCESS_KEY` are used.
      [access_key: <string>]
      [secret_key: <secret>]

      # Named AWS profile used to authenticate.
      [profile: <string>]

      # AWS Role ARN, an alternative to using AWS API keys.
      [role_arn: <string>]

    # Configures the remote write request's TLS settings.
    tls_config:
      # CA certificate to validate API server certificate with.
      [ca_file: <filename>]
      # Certificate and key files for client cert authentication to the server.
      [cert_file: <filename>]
      [key_file: <filename>]
      # ServerName extension to indicate the name of the server.
      # https://tools.ietf.org/html/rfc4366#section-3.1
      [server_name: <string>]
      # Disable validation of the server certificate.
      [insecure_skip_verify: <boolean>]

    # Optional proxy URL.
    [proxy_url: <string>]

    # Configure whether HTTP requests follow HTTP 3xx redirects.
    [follow_redirects: <boolean> | default = true]

    # Configures the queue used to write to remote storage.
    queue_config:
      # Number of samples to buffer per shard before we block reading of more
      # samples from the WAL. It is recommended to have enough capacity in each
      # shard to buffer several requests to keep throughput up while processing
      # occasional slow remote requests.
      [capacity: <int> | default = 2500]
      # Maximum number of shards, i.e. amount of concurrency.
      [max_shards: <int> | default = 200]
      # Minimum number of shards, i.e. amount of concurrency.
      [min_shards: <int> | default = 1]
      # Maximum number of samples per send.
      [max_samples_per_send: <int> | default = 500]
      # Maximum time a sample will wait in buffer.
      [batch_send_deadline: <duration> | default = 5s]
      # Initial retry delay. Gets doubled for every retry.
      [min_backoff: <duration> | default = 30ms]
      # Maximum retry delay.
      [max_backoff: <duration> | default = 100ms]
      # Retry upon receiving a 429 status code from the remote-write storage.
      # This is experimental and might change in the future.
      [retry_on_http_429: <boolean> | default = false]

wal:
  # The directory in which to write tenant WAL files. Each tenant will have its own
  # directory one level below this directory.
  [dir: <string> | default = "ruler-wal"]
  # Frequency with which to run the WAL truncation process.
  [truncate_frequency: <duration> | default = 60m]
  # Minimum and maximum time series should exist in the WAL for.
  [min_age: <duration> | default = 5m]
  [max_age: <duration> | default = 4h]

wal_cleaner:
  # The minimum age of a WAL to consider for cleaning.
  [min_age: <duration> | default = 12h]
  # How often to run the WAL cleaner.
  [period: <duration> | default = 0s (disabled)]

# File path to store temporary rule files.
# CLI flag: -ruler.rule-path
[rule_path: <filename> | default = "/rules"]

# Comma-separated list of Alertmanager URLs to send notifications to.
# Each Alertmanager URL is treated as a separate group in the configuration.
# Multiple Alertmanagers in HA per group can be supported by using DNS
# resolution via -ruler.alertmanager-discovery.
# CLI flag: -ruler.alertmanager-url
[alertmanager_url: <string> | default = ""]


alertmanager_client:
  # Sets the `Authorization` header on every remote write request with the
  # configured username and password.
  # password and password_file are mutually exclusive.
  basic_auth:
    [username: <string>]
    [password: <secret>]

  # Optional `Authorization` header configuration.
  authorization:
    # Sets the authentication type.
    [type: <string> | default: Bearer]
    # Sets the credentials. It is mutually exclusive with
    # `credentials_file`.
    [credentials: <secret>]
    # Sets the credentials to the credentials read from the configured file.
    # It is mutually exclusive with `credentials`.
    [credentials_file: <filename>]

# Use DNS SRV records to discover Alertmanager hosts.
# CLI flag: -ruler.alertmanager-discovery
[enable_alertmanager_discovery: <boolean> | default = false]

# How long to wait between refreshing DNS resolutions of Alertmanager hosts.
# CLI flag: -ruler.alertmanager-refresh-interval
[alertmanager_refresh_interval: <duration> | default = 1m]

# If enabled, then requests to Alertmanager use the v2 API.
# CLI flag: -ruler.alertmanager-use-v2
[enable_alertmanager_v2: <boolean> | default = false]

# List of alert relabel configs
alert_relabel_configs:
  [- <relabel_config> ...]

# Capacity of the queue for notifications to be sent to the Alertmanager.
# CLI flag: -ruler.notification-queue-capacity
[notification_queue_capacity: <int> | default = 10000]

# HTTP timeout duration when sending notifications to the Alertmanager.
# CLI flag: -ruler.notification-timeout
[notification_timeout: <duration> | default = 10s]

# Max time to tolerate outage for restoring "for" state of alert.
# CLI flag: -ruler.for-outage-tolerance
[for_outage_tolerance: <duration> | default = 1h]

# Minimum duration between alert and restored "for" state. This is maintained
# only for alerts with configured "for" time greater than the grace period.
# CLI flag: -ruler.for-grace-period
[for_grace_period: <duration> | default = 10m]

# Minimum amount of time to wait before resending an alert to Alertmanager.
# CLI flag: -ruler.resend-delay
[resend_delay: <duration> | default = 1m]

# Distribute rule evaluation using ring backend.
# CLI flag: -ruler.enable-sharding
[enable_sharding: <boolean> | default = false]

# Time to spend searching for a pending ruler when shutting down.
# CLI flag: -ruler.search-pending-for
[search_pending_for: <duration> | default = 5m]

# Ring used by Loki ruler.
# The CLI flags prefix for this block config is ruler.ring
ring:
  kvstore:
    # Backend storage to use for the ring. Supported values are: consul, etcd,
    # inmemory, memberlist, multi.
    # CLI flag: -<prefix>.store
    [store: <string> | default = "memberlist"]

    # The prefix for the keys in the store. Should end with a /.
    # CLI flag: -<prefix>.prefix
    [prefix: <string> | default = "collectors/"]

    # The consul_config configures the consul client.
    [consul: <consul_config>]

    # The etcd_config configures the etcd client.
    [etcd: <etcd_config>]

    multi:
      # Primary backend storage used by multi-client.
      # CLI flag: -<prefix>.multi.primary
      [primary: <string> | default = ""]

      # Secondary backend storage used by multi-client.
      # CLI flag: -<prefix>.multi.secondary
      [secondary: <string> | default = ""]

      # Mirror writes to secondary store.
      # CLI flag: -<prefix>.multi.mirror-enabled
      [mirror_enabled: <boolean> | default = false]

      # Timeout for storing value to secondary store.
      # CLI flag: -<prefix>.multi.mirror-timeout
      [mirror_timeout: <duration> | default = 2s]

  # Interval between heartbeats sent to the ring. 0 = disabled.
  # CLI flag: -<prefix>.heartbeat-period
  [heartbeat_period: <duration> | default = 15s]

  # The heartbeat timeout after which ruler ring members are considered unhealthy
  # within the ring. 0 = never (timeout disabled).
  # CLI flag: -<prefix>.heartbeat-timeout
  [heartbeat_timeout: <duration> | default = 1m]

  # Name of network interface to read addresses from.
  # CLI flag: -<prefix>.instance-interface-names
  [instance_interface_names: <list of string> | default = [<private network interfaces>]]

  # The number of tokens the lifecycler will generate and put into the ring if
  # it joined without transferring tokens from another lifecycler.
  # CLI flag: -<prefix>.num-tokens
  [num_tokens: <int> | default = 128]

bos_storage_config

The bos_storage_config block configures Baidu Object Storage (BOS) as general storage for data generated by Loki.

# Name of BOS bucket.
# CLI flag: <prefix>.baidubce.bucket-name
[ bucket_name: <string> | default = "" ]
# BOS endpoint to connect to.
# CLI flag: <prefix>.baidubce.endpoint
[ endpoint: <string> | default = "bj.bcebos.com" ]
# Baidu Cloud Engine (BCE) Access Key ID
# CLI flag: <prefix>.baidubce.access-key-id
[ access_key_id: <string> | default = "" ]
# BCE Secret Access Key
# CLI flag: <prefix>.baidubce.secret-access-key
[ secret_access_key: <string> | default = "" ]

azure_storage_config

The azure_storage_config configures Azure as a general storage for different data generated by Loki.

# Azure Cloud environment. Supported values are: AzureGlobal,
# AzureChinaCloud, AzureGermanCloud, AzureUSGovernment.
# CLI flag: -<prefix>.azure.environment
[environment: <string> | default = "AzureGlobal"]

# Azure storage account name.
# CLI flag: -<prefix>.azure.account-name
[account_name: <string> | default = ""]

# Azure storage account key.
# CLI flag: -<prefix>.azure.account-key
[account_key: <string> | default = ""]

# Name of the storage account blob container used to store chunks.
# This container must be created before running Loki.
# CLI flag: -<prefix>.azure.container-name
[container_name: <string> | default = "loki"]

# Azure storage endpoint suffix without schema. The storage account name will
# be prefixed to this value to create the FQDN.
# CLI flag: -<prefix>.azure.endpoint-suffix
[endpoint_suffix: <string> | default = ""]

# Use Managed Identity to authenticate to the Azure storage account.
# CLI flag: -<prefix>.azure.use-managed-identity
[use_managed_identity: <boolean> | default = false]

# User assigned identity ID to authenticate to the Azure storage account.
# CLI flag: -<prefix>.azure.user-assigned-id
[user_assigned_id: <string> | default = ""]

# Chunk delimiter to build the blobID
# CLI flag: -<prefix>.azure.chunk-delimiter
[chunk_delimiter: <string> | default = "-"]

# Preallocated buffer size for downloads.
# CLI flag: -<prefix>.azure.download-buffer-size
[download_buffer_size: <int> | default = 512000]

# Preallocated buffer size for uploads.
# CLI flag: -<prefix>.azure.upload-buffer-size
[upload_buffer_size: <int> | default = 256000]

# Number of buffers used to used to upload a chunk.
# CLI flag: -<prefix>.azure.download-buffer-count
[upload_buffer_count: <int> | default = 1]

# Timeout for requests made against azure blob storage.
# CLI flag: -<prefix>.azure.request-timeout
[request_timeout: <duration> | default = 30s]

# Number of retries for a request which times out.
# CLI flag: -<prefix>.azure.max-retries
[max_retries: <int> | default = 5]

# Minimum time to wait before retrying a request.
# CLI flag: -<prefix>.azure.min-retry-delay
[min_retry_delay: <duration> | default = 10ms]

# Maximum time to wait before retrying a request.
# CLI flag: -<prefix>.azure.max-retry-delay
[max_retry_delay: <duration> | default = 500ms]

gcs_storage_config

The gcs_storage_config configures GCS as a general storage for different data generated by Loki.

# Name of GCS bucket to put chunks in.
# CLI flag: -<prefix>.gcs.bucketname
[bucket_name: <string> | default = ""]

# The size of the buffer that GCS client for each PUT request. 0 to disable
# buffering.
# CLI flag: -<prefix>.gcs.chunk-buffer-size
[chunk_buffer_size: <int> | default = 0]

# The duration after which the requests to GCS should be timed out.
# CLI flag: -<prefix>.gcs.request-timeout
[request_timeout: <duration> | default = 0s]

# Enable HTTP/2 when connecting to GCS. This configuration only applies to GET operations.
# CLI flag: -<prefix>.gcs.enable-http2
[enable_http2: <boolean> | default = true]

s3_storage_config

The s3_storage_config configures S3 as a general storage for different data generated by Loki.

# S3 endpoint URL with escaped Key and Secret encoded. If only region is
# specified as a host, proper endpoint will be deduced. Use
# inmemory:///<bucket-name> to use a mock in-memory implementation.
# CLI flag: -<prefix>.s3.url
[s3: <url> | default = ]

# Set this to `true` to force the request to use path-style addressing.
# CLI flag: -<prefix>.s3.force-path-style
[s3forcepathstyle: <boolean> | default = false]

# Comma separated list of bucket names to evenly distribute chunks over.
# Overrides any buckets specified in s3.url flag
# CLI flag: -<prefix>.s3.buckets
[bucketnames: <string> | default = ""]

# S3 Endpoint to connect to.
# CLI flag: -<prefix>.s3.endpoint
[endpoint: <string> | default = ""]

# AWS region to use.
# CLI flag: -<prefix>.s3.region
[region: <string> | default = ""]

# AWS Access Key ID
# CLI flag: -<prefix>.s3.access-key-id
[access_key_id: <string> | default = ""]

# AWS Secret Access Key
# CLI flag: -<prefix>.s3.secret-access-key
[secret_access_key: <string> | default = ""]

# Disable https on S3 connection.
# CLI flag: -<prefix>.s3.insecure
[insecure: <boolean> | default = false]

# Enable AES256 AWS server-side encryption
# CLI flag: -<prefix>.s3.sse-encryption
[sse_encryption: <boolean> | default = false]

http_config:
  # The maximum amount of time an idle connection will be held open.
  # CLI flag: -<prefix>.s3.http.idle-conn-timeout
  [idle_conn_timeout: <duration> | default = 1m30s]

  # If non-zero, specifies the amount of time to wait for a server's
  # response headers after fully writing the request.
  # CLI flag: -<prefix>.s3.http.response-header-timeout
  [response_header_timeout: <duration> | default = 0s]

  # Set to true to skip verifying the certificate chain and hostname.
  # CLI flag: -<prefix>.s3.http.insecure-skip-verify
  [insecure_skip_verify: <boolean> | default = false]

  # Path to the trusted CA file that signed the SSL certificate of the S3
  # endpoint.
  # CLI flag: -<prefix>.s3.http.ca-file
  [ca_file: <string> | default = ""]

swift_storage_config

The swift_storage_config configures Swift as a general storage for different data generated by Loki.

# Openstack authentication URL.
# CLI flag: -<prefix>.swift.auth-url
[auth_url: <string> | default = ""]

# Openstack username for the api.
# CLI flag: -<prefix>.swift.username
[username: <string> | default = ""]

# Openstack user's domain name.
# CLI flag: -<prefix>.swift.user-domain-name
[user_domain_name: <string> | default = ""]

# Openstack user's domain ID.
# CLI flag: -<prefix>.swift.user-domain-id
[user_domain_id: <string> | default = ""]

# Openstack user ID for the API.
# CLI flag: -<prefix>.swift.user-id
[user_id: <string> | default = ""]

# Openstack API key.
# CLI flag: -<prefix>.swift.password
[password: <string> | default = ""]

# Openstack user's domain ID.
# CLI flag: -<prefix>.swift.domain-id
[domain_id: <string> | default = ""]

# Openstack user's domain name.
# CLI flag: -<prefix>.swift.domain-name
[domain_name: <string> | default = ""]

# Openstack project ID (v2,v3 auth only).
# CLI flag: -<prefix>.swift.project-id
[project_id: <string> | default = ""]

# Openstack project name (v2,v3 auth only).
# CLI flag: -<prefix>.swift.project-name
[project_name: <string> | default = ""]

# ID of the project's domain (v3 auth only), only needed if it differs the
# from user domain.
# CLI flag: -<prefix>.swift.project-domain-id
[project_domain_id: <string> | default = ""]

# Name of the project's domain (v3 auth only), only needed if it differs
# from the user domain.
# CLI flag: -<prefix>.swift.project-domain-name
[project_domain_name: <string> | default = ""]

# Openstack Region to use eg LON, ORD - default is use first region (v2,v3
# auth only)
# CLI flag: -<prefix>.swift.region-name
[region_name: <string> | default = ""]

# Name of the Swift container to put chunks in.
# CLI flag: -<prefix>.swift.container-name
[container_name: <string> | default = ""]

hedging

The hedging block configures how to hedge storage requests.

The hedging implementation sends a second storage request once a first request has been outstanding for more than a configured expected latency for this class of requests. Calculate your latency to be the 99th percentile of object storage response times.

# An optional duration that sets the quantity of time after a first storage request
# is sent and before a second request is sent, when no response is received for the first
# storage request. The recommended duration is the measured 99th percentile of object
# storage response times, to reduce long tail latency. This option is most impactful
# when used with queriers, and has minimal to no impact on other components.
# The default value of 0 disables the hedging of storage requests.
# Example: "at: 500ms"
[at: <duration> | default = 0]

# An optional maximum quantity of hedged requests to be issued for a given request.
[up_to: <int> | default = 2]

# Caps the rate of hedged requests by optionally defining the maximum quantity of
# hedged requests issued per second.
[max_per_second: <int> | default = 5]

local_storage_config

The local_storage_config configures a (local) file system as a general storage for different data generated by Loki.

# Filesystem directory to be used as storage.
# CLI flag: -<prefix>.local.directory
[directory: <filename> | default = ""]

frontend_worker

The frontend_worker configures the worker - running within the Loki querier - picking up and executing queries enqueued by the query-frontend.

# Address of query frontend service, in host:port format.
# CLI flag: -querier.frontend-address
[frontend_address: <string> | default = ""]

# Number of simultaneous queries to process.
# CLI flag: -querier.worker-parallelism
[parallelism: <int> | default = 10]

# Force worker concurrency to match the -querier.max-concurrent option. Overrides querier.worker-parallelism.
# CLI flag: -querier.worker-match-max-concurrent
[match_max_concurrent: <boolean> | default = true]

# How often to query the frontend_address DNS to resolve frontend addresses.
# Also used to determine how often to poll the scheduler-ring for addresses if configured.
# CLI flag: -querier.dns-lookup-period
[dns_lookup_duration: <duration> | default = 3s]

# The CLI flags prefix for this block config is: querier.frontend-client
[grpc_client_config: <grpc_client_config>]

# DNS hostname used for finding query-schedulers.
# CLI flag: -querier.scheduler-address
[scheduler_address: <string> | default = ""]

ingester_client

The ingester_client block configures how connections to ingesters operate.

# Configures how connections are pooled
pool_config:
  # Whether or not to do health checks.
  # CLI flag: -distributor.health-check-ingesters
  [health_check_ingesters: <boolean> | default = false]

  # How frequently to clean up clients for servers that have gone away after
  # a health check.
  # CLI flag: -distributor.client-cleanup-period
  [client_cleanup_period: <duration> | default = 15s]

  # How quickly a dead client will be removed after it has been detected
  # to disappear. Set this to a value to allow time for a secondary
  # health check to recover the missing client.
  # CLI flag: -ingester.client.healthcheck-timeout
  [remote_timeout: <duration> | default = 1s]

# The remote request timeout on the client side.
# CLI flag: -ingester.client.timeout
[remote_timeout: <duration> | default = 5s]

# Configures how the gRPC connection to ingesters work as a client
# The CLI flags prefix for this block config is: ingester.client
[grpc_client_config: <grpc_client_config>]

ingester

The ingester block configures the Loki Ingesters.

# Configures how the lifecycle of the ingester will operate
# and where it will register for discovery.
lifecycler:
  ring:
    kvstore:
      # Backend storage to use for the ring. Supported values are: consul, etcd,
      # inmemory, memberlist
      # CLI flag: -ring.store
      [store: <string> | default = "consul"]

      # The prefix for the keys in the store. Should end with a /.
      # CLI flag: -ring.prefix
      [prefix: <string> | default = "collectors/"]

      # The consul_config configures the consul client.
      # CLI flag: <no prefix>
      [consul: <consul_config>]

      # The etcd_config configures the etcd client.
      # CLI flag: <no prefix>
      [etcd: <etcd_config>]

    # The heartbeat timeout after which ingesters are skipped for reads/writes.
    # CLI flag: -ring.heartbeat-timeout
    [heartbeat_timeout: <duration> | default = 1m]

    # The number of ingesters to write to and read from.
    # CLI flag: -distributor.replication-factor
    [replication_factor: <int> | default = 3]

  # The number of tokens the lifecycler will generate and put into the ring if
  # it joined without transferring tokens from another lifecycler.
  # CLI flag: -ingester.num-tokens
  [num_tokens: <int> | default = 128]

  # Period at which to heartbeat to the underlying ring.
  # CLI flag: -ingester.heartbeat-period
  [heartbeat_period: <duration> | default = 5s]

  # How long to wait to claim tokens and chunks from another member when
  # that member is leaving. Will join automatically after the duration expires.
  # CLI flag: -ingester.join-after
  [join_after: <duration> | default = 0s]

  # Observe tokens after generating to resolve collisions. Useful when using a gossip ring.
  # CLI flag: -ingester.observe-period
  [observe_period: <duration> | default = 0s]

  # Minimum duration to wait before becoming ready. This is to work around race
  # conditions with ingesters exiting and updating the ring.
  # CLI flag: -ingester.min-ready-duration
  [min_ready_duration: <duration> | default = 15s]

  # Name of network interfaces to read addresses from.
  # CLI flag: -ingester.lifecycler.interface
  interface_names:

    - [<string> ... | default = [<private network interfaces>]]

  # Duration to sleep before exiting to ensure metrics are scraped.
  # CLI flag: -ingester.final-sleep
  [final_sleep: <duration> | default = 30s]

# Number of times to try and transfer chunks when leaving before
# falling back to flushing to the store. Zero = no transfers are done.
# CLI flag: -ingester.max-transfer-retries
[max_transfer_retries: <int> | default = 0]

# How many flushes can happen concurrently from each stream.
# CLI flag: -ingester.concurrent-flushes
[concurrent_flushes: <int> | default = 32]

# How often should the ingester see if there are any blocks to flush
# CLI flag: -ingester.flush-check-period
[flush_check_period: <duration> | default = 30s]

# The timeout before a flush is cancelled
# CLI flag: -ingester.flush-op-timeout
[flush_op_timeout: <duration> | default = 10m]

# How long chunks should be retained in-memory after they've been flushed.
# CLI flag: -ingester.chunks-retain-period
[chunk_retain_period: <duration> | default = 0s]

# How long chunks should sit in-memory with no updates before
# being flushed if they don't hit the max block size. This means
# that half-empty chunks will still be flushed after a certain
# period as long as they receive no further activity.
# CLI flag: -ingester.chunks-idle-period
[chunk_idle_period: <duration> | default = 30m]

# The targeted _uncompressed_ size in bytes of a chunk block
# When this threshold is exceeded the head block will be cut and compressed inside the chunk.
# CLI flag: -ingester.chunks-block-size
[chunk_block_size: <int> | default = 262144]

# A target _compressed_ size in bytes for chunks.
# This is a desired size not an exact size, chunks may be slightly bigger
# or significantly smaller if they get flushed for other reasons (e.g. chunk_idle_period)
# A value of 0 creates chunks with a fixed 10 blocks,
# A non zero value will create chunks with a variable number of blocks to meet the target size.
# CLI flag: -ingester.chunk-target-size
[chunk_target_size: <int> | default = 1572864]

# The compression algorithm to use for chunks. (supported: gzip, lz4, snappy)
# You should choose your algorithm depending on your need:
# - `gzip` highest compression ratio but also slowest decompression speed. (144 kB per chunk)
# - `lz4` fastest compression speed (188 kB per chunk)
# - `snappy` fast and popular compression algorithm (272 kB per chunk)
# CLI flag: -ingester.chunk-encoding
[chunk_encoding: <string> | default = gzip]

# Parameters used to synchronize ingesters to cut chunks at the same moment.
# Sync period is used to roll over incoming entry to a new chunk. If chunk's utilization
# isn't high enough (eg. less than 50% when sync_min_utilization is set to 0.5), then
# this chunk rollover doesn't happen.
# CLI flag: -ingester.sync-period
[sync_period: <duration> | default = 0]

# CLI flag: -ingester.sync-min-utilization
[sync_min_utilization: <float> | Default = 0]

# The maximum number of errors a stream will report to the user
# when a push fails. 0 to make unlimited.
# CLI flag: -ingester.max-ignored-stream-errors
[max_returned_stream_errors: <int> | default = 10]

# The maximum duration of a timeseries chunk in memory. If a timeseries runs for longer than this,
# the current chunk will be flushed to the store and a new chunk created.
# CLI flag: -ingester.max-chunk-age
[max_chunk_age: <duration> | default = 2h]

# How far in the past an ingester is allowed to query the store for data.
# This is only useful for running multiple Loki binaries with a shared ring
# with a `filesystem` store, which is NOT shared between the binaries.
# When using any "shared" object store like S3 or GCS, this value must always be left as 0.
# It is an error to configure this to a non-zero value when using any object store other
# than `filesystem`.
# Use a value of -1 to allow the ingester to query the store infinitely far back in time.
# CLI flag: -ingester.query-store-max-look-back-period
[query_store_max_look_back_period: <duration> | default = 0]

# Forget about ingesters having heartbeat timestamps older than `ring.kvstore.heartbeat_timeout`.
# This is equivalent to clicking on the `/ring` `forget` button in the UI:
# the ingester is removed from the ring.
# This is a useful setting when you are sure that an unhealthy node won't return.
# An example is when not using stateful sets or the equivalent.
# Use `memberlist.rejoin_interval` > 0 to handle network partition cases when using a memberlist.
# CLI flag: -ingester.autoforget-unhealthy
[autoforget_unhealthy: <boolean> | default = false]

# The ingester WAL (Write Ahead Log) records incoming logs and stores them on
# the local file systems in order to guarantee persistence of acknowledged data
# in the event of a process crash.
wal:
  # Enables writing to WAL.
  # CLI flag: -ingester.wal-enabled
  [enabled: <boolean> | default = true]

  # Directory where the WAL data should be stored and/or recovered from.
  # CLI flag: -ingester.wal-dir
  [dir: <filename> | default = "wal"]

  # When WAL is enabled, should chunks be flushed to long-term storage on shutdown.
  # CLI flag: -ingester.flush-on-shutdown
  [flush_on_shutdown: <boolean> | default = false]

  # Interval at which checkpoints should be created.
  # CLI flag: ingester.checkpoint-duration
  [checkpoint_duration: <duration> | default = 5m]

  # Maximum memory size the WAL may use during replay. After hitting this,
  # it will flush data to storage before continuing.
  # A unit suffix (KB, MB, GB) may be applied.
  [replay_memory_ceiling: <string> | default = 4GB]

# Shard factor used in the ingesters for the in process reverse index.
# This MUST be evenly divisible by ALL schema shard factors or Loki will not start.
[index_shards: <int> | default = 32]

consul_config

The consul_config configures the consul client. The supported CLI flags used to reference this config block are:

 # The hostname and port of Consul.
# CLI flag: -<prefix>.consul.hostname
[host: <string> | default = "localhost:8500"]

# The ACL Token used to interact with Consul.
# CLI flag: -<prefix>.consul.acl-token
[acl_token: <string>]

# The HTTP timeout when communicating with Consul
# CLI flag: -<prefix>.consul.client-timeout
[http_client_timeout: <duration> | default = 20s]

# Whether or not consistent reads to Consul are enabled.
# CLI flag: -<prefix>.consul.consistent-reads
[consistent_reads: <boolean> | default = true]

etcd_config

The etcd_config configures the etcd client. The supported CLI flags used to reference this config block are:

# The etcd endpoints to connect to.
# CLI flag: -<prefix>.etcd.endpoints
[endpoints: <list of string> | default = []]

# The dial timeout for the etcd connection.
# CLI flag: -<prefix>.etcd.dial-timeout
[dial_timeout: <duration> | default = 10s]

# The maximum number of retries to do for failed ops.
# CLI flag: -<prefix>.etcd.max-retries
[max_retries: <int> | default = 10]

memberlist_config

The memberlist_config block configures the gossip ring to discover and connect between distributors, ingesters and queriers. The configuration is unique for all three components to ensure a single shared ring.

When a memberlist_config with least 1 join_members is defined, a kvstore of type memberlist is automatically configured for the distributor, ingester, and ruler rings unless otherwise specified in those components specific configuration sections.

# Name of the node in memberlist cluster. Defaults to hostname.
# CLI flag: -memberlist.nodename
[node_name: <string> | default = ""]

# Add random suffix to the node name.
# CLI flag: -memberlist.randomize-node-name
[randomize_node_name: <boolean> | default = true]

# The timeout for establishing a connection with a remote node, and for
# read/write operations. Uses memberlist LAN defaults if 0.
# CLI flag: -memberlist.stream-timeout
[stream_timeout: <duration> | default = 0s]

# Multiplication factor used when sending out messages (factor * log(N+1)).
# CLI flag: -memberlist.retransmit-factor
[retransmit_factor: <int> | default = 0]

# How often to use pull/push sync. Uses memberlist LAN defaults if 0.
# CLI flag: -memberlist.pullpush-interval
[pull_push_interval: <duration> | default = 0s]

# How often to gossip. Uses memberlist LAN defaults if 0.
# CLI flag: -memberlist.gossip-interval
[gossip_interval: <duration> | default = 0s]

# How many nodes to gossip to. Uses memberlist LAN defaults if 0.
# CLI flag: -memberlist.gossip-nodes
[gossip_nodes: <int> | default = 0]

# How long to keep gossiping to dead nodes, to give them chance to refute their
# death. Uses memberlist LAN defaults if 0.
# CLI flag: -memberlist.gossip-to-dead-nodes-time
[gossip_to_dead_nodes_time: <duration> | default = 0s]

# How soon can dead node's name be reclaimed with new address. Defaults to 0,
# which is disabled.
# CLI flag: -memberlist.dead-node-reclaim-time
[dead_node_reclaim_time: <duration> | default = 0s]

# Other cluster members to join. Can be specified multiple times. It can be an
# IP, hostname or an entry specified in the DNS Service Discovery format (see
# https://grafana.com/docs/mimir/latest/operators-guide/configuring/about-dns-service-discovery/
# for more details).
# CLI flag: -memberlist.join
[join_members: <list of string> | default = ]

# Min backoff duration to join other cluster members.
# CLI flag: -memberlist.min-join-backoff
[min_join_backoff: <duration> | default = 1s]

# Max backoff duration to join other cluster members.
# CLI flag: -memberlist.max-join-backoff
[max_join_backoff: <duration> | default = 1m]

# Max number of retries to join other cluster members.
# CLI flag: -memberlist.max-join-retries
[max_join_retries: <int> | default = 10]

# If this node fails to join memberlist cluster, abort.
# CLI flag: -memberlist.abort-if-join-fails
[abort_if_cluster_join_fails: <boolean> | default = true]

# If not 0, how often to rejoin the cluster. Occasional rejoin can help to fix
# the cluster split issue, and is harmless otherwise. For example when using
# only few components as a seed nodes (via -memberlist.join), then it's
# recommended to use rejoin. If -memberlist.join points to dynamic service that
# resolves to all gossiping nodes (eg. Kubernetes headless service), then rejoin
# is not needed.
# CLI flag: -memberlist.rejoin-interval
[rejoin_interval: <duration> | default = 0s]

# How long to keep LEFT ingesters in the ring.
# CLI flag: -memberlist.left-ingesters-timeout
[left_ingesters_timeout: <duration> | default = 5m]

# Timeout for leaving memberlist cluster.
# CLI flag: -memberlist.leave-timeout
[leave_timeout: <duration> | default = 5s]

# IP address to listen on for gossip messages. Multiple addresses may be
# specified. Defaults to 0.0.0.0
# CLI flag: -memberlist.bind-addr
[bind_addr: <list of string> | default = ]

# Port to listen on for gossip messages.
# CLI flag: -memberlist.bind-port
[bind_port: <int> | default = 7946]

# Timeout used when connecting to other nodes to send packet.
# CLI flag: -memberlist.packet-dial-timeout
[packet_dial_timeout: <duration> | default = 5s]

# Timeout for writing 'packet' data.
# CLI flag: -memberlist.packet-write-timeout
[packet_write_timeout: <duration> | default = 5s]

storage_config

The storage_config block configures one of many possible stores for both the index and chunks. Which configuration to be picked should be defined in schema_config block.

# Configures storing chunks in AWS. Required options only required when aws is
# present.
aws:
  # S3 or S3-compatible endpoint URL with escaped Key and Secret encoded.
  # If only region is specified as a host, the proper endpoint will be deduced.
  # Use inmemory:///<bucket-name> to use a mock in-memory implementation.
  # CLI flag: -s3.url
  [s3: <string>]

  # Set to true to force the request to use path-style addressing
  # CLI flag: -s3.force-path-style
  [s3forcepathstyle: <boolean> | default = false]

  # Comma separated list of bucket names to evenly distribute chunks over.
  # Overrides any buckets specified in s3.url flag
  # CLI flag: -s3.buckets
  [bucketnames: <string> | default = ""]

  # S3 Endpoint to connect to.
  # CLI flag: -s3.endpoint
  [endpoint: <string> | default = ""]

  # AWS region to use.
  # CLI flag: -s3.region
  [region: <string> | default = ""]

  # AWS Access Key ID.
  # CLI flag: -s3.access-key-id
  [access_key_id: <string> | default = ""]

  # AWS Secret Access Key.
  # CLI flag: -s3.secret-access-key
  [secret_access_key: <string> | default = ""]

  # Disable https on S3 connection.
  # CLI flag: -s3.insecure
  [insecure: <boolean> | default = false]

  # Enable AES256 AWS Server Side Encryption.
  # CLI flag: -s3.sse-encryption
  [sse_encryption: <boolean> | default = false]

  http_config:
    # The maximum amount of time an idle connection will be held open.
    # CLI flag: -s3.http.idle-conn-timeout
    [idle_conn_timeout: <duration> | default = 1m30s]

    # If non-zero, specifies the amount of time to wait for a server's response
    # headers after fully writing the request.
    # CLI flag: -s3.http.response-header-timeout
    [response_header_timeout: <duration> | default = 0s]

    # Set to true to skip verifying the certificate chain and hostname.
    # CLI flag: -s3.http.insecure-skip-verify
    [insecure_skip_verify: <boolean> | default = false]

    # Path to the trusted CA file that signed the SSL certificate of the S3
    # endpoint.
    # CLI flag: -s3.http.ca-file
    [ca_file: <string> | default = ""]

  # Configures back off when s3 get Object.
  backoff_config:
    # Minimum duration to back off.
    # CLI flag: -s3.backoff-min-period
    [min_period: <duration> | default = 100ms]

    # The duration to back off.
    # CLI flag: -s3.backoff-max-period
    [max_period: <duration> | default = 3s]

    # Number of times to back off and retry before failing.
    # CLI flag: -s3.backoff-retries
    [max_retries: <int> | default = 5]

  # Configure the DynamoDB connection
  dynamodb:
    # URL for DynamoDB with escaped Key and Secret encoded. If only region is specified as a
    # host, the proper endpoint will be deduced. Use inmemory:///<bucket-name> to
    # use a mock in-memory implementation.
    # CLI flag: -dynamodb.url
    dynamodb_url: <string>

    # DynamoDB table management requests per-second limit.
    # CLI flag: -dynamodb.api-limit
    [api_limit: <float> | default = 2.0]

    # DynamoDB rate cap to back off when throttled.
    # CLI flag: -dynamodb.throttle-limit
    [throttle_limit: <float> | default = 10.0]

    # Metrics-based autoscaling configuration.
    metrics:
      # Use metrics-based autoscaling via this Prometheus query URL.
      # CLI flag: -metrics.url
      [url: <string>]

      # Queue length above which we will scale up capacity.
      # CLI flag: -metrics.target-queue-length
      [target_queue_length: <int> | default = 100000]

      # Scale up capacity by this multiple
      # CLI flag: -metrics.scale-up-factor
      [scale_up_factor: <float64> | default = 1.3]

      # Ignore throttling below this level (rate per second)
      # CLI flag: -metrics.ignore-throttle-below
      [ignore_throttle_below: <float64> | default = 1]

    # Number of chunks to group together to parallelise fetches (0 to disable)
    # CLI flag: -dynamodb.chunk-gang-size
    [chunk_gang_size: <int> | default = 10]

    # Max number of chunk get operations to start in parallel.
    # CLI flag: -dynamodb.chunk.get-max-parallelism
    [chunk_get_max_parallelism: <int> | default = 32]

# Configures storing indexes in Bigtable. Required fields only required
# when bigtable is defined in config.
bigtable:
  # BigTable project ID
   # CLI flag: -bigtable.project
  project: <string>

  # BigTable instance ID
  # CLI flag: -bigtable.instance
  instance: <string>

  # Configures the gRPC client used to connect to Bigtable.
  # The CLI flags prefix for this block config is: bigtable
  [grpc_client_config: <grpc_client_config>]

# Configures storing chunks in GCS. Required fields only required
# when gcs is defined in config.
gcs:
  # Name of GCS bucket to put chunks in.
  # CLI flag: -gcs.bucketname
  bucket_name: <string>

  # The size of the buffer that the GCS client uses for each PUT request. 0
  # to disable buffering.
  # CLI flag: -gcs.chunk-buffer-size
  [chunk_buffer_size: <int> | default = 0]

  # The duration after which the requests to GCS should be timed out.
  # CLI flag: -gcs.request-timeout
  [request_timeout: <duration> | default = 0s]

# Configures storing chunks and/or the index in Cassandra
cassandra:
  # Comma-separated hostnames or IPs of Cassandra instances
  # CLI flag: -cassandra.addresses
  addresses: <string>

  # Port that cassandra is running on
  # CLI flag: -cassandra.port
  [port: <int> | default = 9042]

  # Keyspace to use in Cassandra
  # CLI flag: -cassandra.keyspace
  keyspace: <string>

  # Consistency level for Cassandra
  # CLI flag: -cassandra.consistency
  [consistency: <string> | default = "QUORUM"]

  # Replication factor to use in Cassandra.
  # CLI flag: -cassandra.replication-factor
  [replication_factor: <int> | default = 1]

  # Instruct the Cassandra driver to not attempt to get host
  # info from the system.peers table.
  # CLI flag: -cassandra.disable-initial-host-lookup
  [disable_initial_host_lookup: <boolean> | default = false]

  # Use SSL when connecting to Cassandra instances.
  # CLI flag: -cassandra.ssl
  [SSL: <boolean> | default = false]

  # Require SSL certificate validation when SSL is enabled.
  # CLI flag: -cassandra.host-verification
  [host_verification: <boolean> | default = true]

  # Path to CA certificate file to verify the peer when SSL is enabled.
  # CLI flag: -cassandra.ca-path
  [CA_path: <string>]

  # Path to client certificate file when SSL is enabled.
  # CLI flag: -cassandra.tls-cert-path
  [tls_cert_path: <string>]

  # Path to key certificate file when SSL is enabled.
  # CLI flag: -cassandra.tls-key-path
  [tls_key_path: <string>]

  # Enable password authentication when connecting to Cassandra.
  # CLI flag: -cassandra.auth
  [auth: <boolean> | default = false]

  # Username for password authentication when auth is true.
  # CLI flag: -cassandra.username
  [username: <string>]

  # Password for password authentication when auth is true.
  # CLI flag: -cassandra.password
  [password: <string>]

  # Timeout when connecting to Cassandra.
  # CLI flag: -cassandra.timeout
  [timeout: <duration> | default = 600ms]

  # Initial connection timeout during initial dial to server.
  # CLI flag: -cassandra.connect-timeout
  [connect_timeout: <duration> | default = 600ms]

swift:
  # Openstack authentication URL.
  # CLI flag: -ruler.storage.swift.auth-url
  [auth_url: <string> | default = ""]

  # Openstack username for the api.
  # CLI flag: -ruler.storage.swift.username
  [username: <string> | default = ""]

  # Openstack user's domain name.
  # CLI flag: -ruler.storage.swift.user-domain-name
  [user_domain_name: <string> | default = ""]

  # Openstack user's domain id.
  # CLI flag: -ruler.storage.swift.user-domain-id
  [user_domain_id: <string> | default = ""]

  # Openstack userid for the api.
  # CLI flag: -ruler.storage.swift.user-id
  [user_id: <string> | default = ""]

  # Openstack api key.
  # CLI flag: -ruler.storage.swift.password
  [password: <string> | default = ""]

  # Openstack user's domain id.
  # CLI flag: -ruler.storage.swift.domain-id
  [domain_id: <string> | default = ""]

  # Openstack user's domain name.
  # CLI flag: -ruler.storage.swift.domain-name
  [domain_name: <string> | default = ""]

  # Openstack project id (v2,v3 auth only).
  # CLI flag: -ruler.storage.swift.project-id
  [project_id: <string> | default = ""]

  # Openstack project name (v2,v3 auth only).
  # CLI flag: -ruler.storage.swift.project-name
  [project_name: <string> | default = ""]

  # Id of the project's domain (v3 auth only), only needed if it differs the
  # from user domain.
  # CLI flag: -ruler.storage.swift.project-domain-id
  [project_domain_id: <string> | default = ""]

  # Name of the project's domain (v3 auth only), only needed if it differs
  # from the user domain.
  # CLI flag: -ruler.storage.swift.project-domain-name
  [project_domain_name: <string> | default = ""]

  # Openstack Region to use eg LON, ORD - default is use first region (v2,v3
  # auth only)
  # CLI flag: -ruler.storage.swift.region-name
  [region_name: <string> | default = ""]

  # Name of the Swift container to put chunks in.
  # CLI flag: -ruler.storage.swift.container-name
  [container_name: <string> | default = "cortex"]

# Configures storing index in BoltDB. Required fields only
# required when boltdb is present in the configuration.
boltdb:
  # Location of BoltDB index files.
  # CLI flag: -boltdb.dir
  directory: <string>

# Configures storing the chunks on the local file system. Required
# fields only required when filesystem is present in the configuration.
filesystem:
  # Directory to store chunks in.
  # CLI flag: -local.chunk-directory
  directory: <string>

# Configures storing index in an Object Store(GCS/S3/Azure/Swift/Filesystem) in the form of
# boltdb files.
# Required fields only required when boltdb-shipper is defined in config.
boltdb_shipper:
  # Directory where ingesters would write boltdb files which would then be
  # uploaded by shipper to configured storage
  # CLI flag: -boltdb.shipper.active-index-directory
  [active_index_directory: <string> | default = ""]

  # Shared store for keeping boltdb files. Supported types: gcs, s3, azure,
  # filesystem
  # CLI flag: -boltdb.shipper.shared-store
  [shared_store: <string> | default = ""]

  # Cache location for restoring boltDB files for queries
  # CLI flag: -boltdb.shipper.cache-location
  [cache_location: <string> | default = ""]

  # TTL for boltDB files restored in cache for queries
  # CLI flag: -boltdb.shipper.cache-ttl
  [cache_ttl: <duration> | default = 24h]

  # Resync downloaded files with the storage
  # CLI flag: -boltdb.shipper.resync-interval
  [resync_interval: <duration> | default = 5m]

  # Number of days of index to be kept downloaded for queries. Works only with
  # tables created with 24h period.
  # CLI flag: -boltdb.shipper.query-ready-num-days
  [query_ready_num_days: <int> | default = 0]

  index_gateway_client:
    # "Hostname or IP of the Index Gateway gRPC server.
    # CLI flag: -boltdb.shipper.index-gateway-client.server-address
    [server_address: <string> | default = ""]

    # Configures the gRPC client used to connect to the Index Gateway gRPC server.
    # The CLI flags prefix for this block config is: boltdb.shipper.index-gateway-client
    [grpc_client_config: <grpc_client_config>]

    # Configures if gateway requests should be logged or not.
    # CLI flag: -boltdb.shipper.index-gateway-client.log-gateway-requests
    [log_gateway_requests: <bool> | default = false]

# Cache validity for active index entries. Should be no higher than
# the chunk_idle_period in the ingester settings.
# CLI flag: -store.index-cache-validity
[index_cache_validity: <duration> | default = 5m]

# Disable broad index queries, which results in reduced cache usage and faster query
# performance at the expense of a somewhat higher QPS on the index store.
# CLI flag: -store.disable-broad-index-queries
[disable_broad_index_queries: <bool> | default = false]

# The maximum number of chunks to fetch per batch.
# CLI flag: -store.max-chunk-batch-size
[max_chunk_batch_size: <int> | default = 50]

# Config for how the cache for index queries should be built.
# The CLI flags prefix for this block config is: store.index-cache-read
index_queries_cache_config: <cache_config>

chunk_store_config

The chunk_store_config block configures how chunks will be cached and how long to wait before saving them to the backing store.

# The cache configuration for storing chunks
# The CLI flags prefix for this block config is: store.chunks-cache
[chunk_cache_config: <cache_config>]

# The cache configuration for deduplicating writes
# The CLI flags prefix for this block config is: store.index-cache-write
[write_dedupe_cache_config: <cache_config>]

# The minimum time between a chunk update and being saved
# to the store.
[min_chunk_age: <duration>]

# Cache index entries older than this period. Default is disabled.
# CLI flag: -store.cache-lookups-older-than
[cache_lookups_older_than: <duration>]

# Limit how long back data can be queried. Default is disabled.
# This should always be set to a value less than or equal to
# what is set in `table_manager.retention_period` .
# CLI flag: -store.max-look-back-period
[max_look_back_period: <duration>]

cache_config

The cache_config block configures how Loki will cache requests, chunks, and the index to a backing cache store.

The memcached configuration variable addresses is experimental.

# Enable in-memory cache.
# CLI flag: -<prefix>.cache.enable-fifocache
[enable_fifocache: <boolean>]

# The default validity of entries for caches unless overridden.
# NOTE In Loki versions older than 1.4.0 this was "defaul_validity".
# CLI flag: -<prefix>.default-validity
[default_validity: <duration>]

# Configures the background cache when memcached is used.
background:
  # How many goroutines to use to write back to memcached.
  # CLI flag: -<prefix>.background.write-back-concurrency
  [writeback_goroutines: <int> | default = 10]

  # How many chunks to buffer for background write back to memcached.
  # CLI flagL -<prefix>.background.write-back-buffer
  [writeback_buffer: <int> = 10000]

# Configures memcached settings.
memcached:
  # Configures how long keys stay in memcached.
  # CLI flag: -<prefix>.memcached.expiration
  expiration: <duration>

  # Configures how many keys to fetch in each batch request.
  # CLI flag: -<prefix>.memcached.batchsize
  batch_size: <int> | default = 1024

  # Maximum active requests to memcached.
  # CLI flag: -<prefix>.memcached.parallelism
  [parallelism: <int> | default = 100]

# Configures how to connect to one or more memcached servers.
memcached_client:
  # The hostname to use for memcached services when caching chunks. If
  # empty, no memcached will be used. A SRV lookup will be used.
  # CLI flag: -<prefix>.memcached.hostname
  [host: <string>]

  # SRV service used to discover memcached servers.
  # CLI flag: -<prefix>.memcached.service
  [service: <string> | default = "memcached"]

  # (Experimental) Comma-separated addresses list in DNS Service Discovery format:
  # https://grafana.com/docs/mimir/latest/operators-guide/configuring/about-dns-service-discovery/
  # CLI flag: -<prefix>.memcached.addresses
  [addresses: <string> | default = ""]

  # Maximum time to wait before giving up on memcached requests.
  # CLI flag: -<prefix>.memcached.timeout
  [timeout: <duration> | default = 100ms]

  # The maximum number of idle connections in the memcached client pool.
  # CLI flag: -<prefix>.memcached.max-idle-conns
  [max_idle_conns: <int> | default = 16]

  # The period with which to poll the DNS for memcached servers.
  # CLI flag: -<prefix>.memcached.update-interval
  [update_interval: <duration> | default = 1m]

  # Whether or not to use a consistent hash to discover multiple memcached servers.
  # CLI flag: -<prefix>.memcached.consistent-hash
  [consistent_hash: <boolean> | default = true]

  # Trip the circuit breaker after this number of consecutive dial failures.
  # A value of 0 disables the circuit breaker.
  # CLI flag: -<prefix>.memcached.circuit-breaker-consecutive-failures
  [circuit_breaker_consecutive_failures: <int> | default = 10]

  # Duration the circuit breaker remains open after tripping.
  # If set to 0, the duration is 60 seconds.
  # CLI flag: -<prefix>.memcached.circuit-breaker-timeout
  [circuit_breaker_timeout: <duration> | default = 10s]

  # Reset the circuit breaker counts after this duration.
  # A value of 0 never resets the circuit breaker.
  # CLI flag: -<prefix>.memcached.circuit-breaker-interval
  [circuit_breaker_interval: <duration> | default = 10s]

  # The maximum size of an item stored in memcached.
  # Bigger items are not stored. If set to 0, no maximum size is enforced.
  # CLI flag: -<prefix>.memcached.max-item-size
  [max_item_size: <int> | default = 0]

redis:
  # Redis Server or Cluster configuration endpoint to use for caching. A comma-separated list of endpoints
  # for Redis Cluster or Redis Sentinel. If empty, no redis will be used.
  # CLI flag: -<prefix>.redis.endpoint
  [endpoint: <string>]

  # Redis Sentinel master name. An empty string for Redis Server or Redis Cluster.
  # CLI flag: -<prefix>.redis.master-name
  [master_name: <string>]

  # Maximum time to wait before giving up on redis requests.
  # CLI flag: -<prefix>.redis.timeout
  [timeout: <duration> | default = 500ms]

  # How long keys stay in the redis.
  # CLI flag: -<prefix>.redis.expiration
  [expiration: <duration> | default = 0s]

  # Database index.
  # CLI flag: -<prefix>.redis.db
  [db: <int> | default = 0]

  # Maximum number of connections in the pool.
  # CLI flag: -<prefix>.redis.pool-size
  [pool_size: <int> | default = 0]

  # Password to use when connecting to redis.
  # CLI flag: -<prefix>.redis.password
  [password: <string>]

  # Enables connecting to redis with TLS.
  # CLI flag: -<prefix>.redis.tls-enabled
  [tls_enabled: <boolean> | default = false]

  # Skip validating server certificate.
  # CLI flag: -<prefix>.redis.tls-insecure-skip-verify
  [tls_insecure_skip_verify: <boolean> | default = false]

  # Close connections after remaining idle for this duration.
  # If the value is zero, then idle connections are not closed.
  # CLI flag: -<prefix>.redis.idle-timeout
  [idle_timeout: <duration> | default = 0s]

  # Close connections older than this duration. If the value is zero, then
  # the pool does not close connections based on age.
  # CLI flag: -<prefix>.redis.max-connection-age
  [max_connection_age: <duration> | default = 0s]

fifocache:
  # Maximum memory size of the cache in bytes. A unit suffix (KB, MB, GB) may be
  # applied.
  # CLI flag: -<prefix>.fifocache.max-size-bytes
  [max_size_bytes: <string> | default = "1GB"]

  # Maximum number of entries in the cache.
  # CLI flag: -<prefix>.fifocache.max-size-items
  [max_size_items: <int> | default = 0]

  # Deprecated: The expiry duration for the cache. Use `-<prefix>.fifocache.ttl`.
  # The default value of 0 disables expiration.
  # CLI flag: -<prefix>.fifocache.duration
  [validity: <duration>]

  # The time for items to live in the cache before those items are purged.
  # The value of 0 disables auto-expiration.
  # CLI flag: -<prefix>.fifocache.ttl
  [ttl: <duration> | default = 1h]

schema_config

The schema_config block configures schemas from given dates.

# The configuration for chunk index schemas.
configs:
- [<period_config>]

period_config

The period_config block configures what index schemas should be used for from specific time periods.

# The date of the first day that index buckets should be created. Use
# a date in the past if this is your only period_config, otherwise
# use a date when you want the schema to switch over.
# In YYYY-MM-DD format, for example: 2018-04-15.
[from: <daytime>]

# store and object_store below affect which <storage_config> key is
# used.

# Which store to use for the index. Either aws, aws-dynamo, gcp, bigtable, bigtable-hashed,
# cassandra, boltdb or boltdb-shipper.
store: <string>

# Which store to use for the chunks. Either aws, azure, gcp,
# bigtable, gcs, cassandra, swift or filesystem. If omitted, defaults to the same
# value as store.
[object_store: <string>]

# The schema version to use, current recommended schema is v11.
schema: <string>

# Configures how the index is updated and stored.
index:
  # Table prefix for all period tables.
  prefix: <string>
  # Table period.
  [period: <duration> | default = 168h]
  # A map to be added to all managed tables.
  tags:
    [<string>: <string> ...]

# Configured how the chunks are updated and stored.
chunks:
  # Table prefix for all period tables.
  prefix: <string>
  # Table period.
  [period: <duration> | default = 168h]
  # A map to be added to all managed tables.
  tags:
    [<string>: <string> ...]

# How many shards will be created. Only used if schema is v10 or greater.
[row_shards: <int> | default = 16]

compactor

The compactor block configures the compactor component. This component periodically compacts index shards to more performant forms.

# Directory where files can be downloaded for compaction.
# CLI flag: -boltdb.shipper.compactor.working-directory
[working_directory: <string>]

# The shared store used for storing boltdb files.
# Supported types: gcs, s3, azure, swift, filesystem, bos.
# CLI flag: -boltdb.shipper.compactor.shared-store
[shared_store: <string>]

# Prefix to add to object keys in shared store.
# Path separator(if any) should always be a '/'.
# Prefix should never start with a separator but should always end with it.
# CLI flag: -boltdb.shipper.compactor.shared-store.key-prefix
[shared_store_key_prefix: <string> | default = "index/"]

# Interval at which to re-run the compaction operation (or retention if enabled).
# CLI flag: -boltdb.shipper.compactor.compaction-interval
[compaction_interval: <duration> | default = 10m]

# (Experimental) Activate custom (per-stream,per-tenant) retention.
# CLI flag: -boltdb.shipper.compactor.retention-enabled
[retention_enabled: <boolean> | default = false]

# Delay after which chunks will be fully deleted during retention.
# CLI flag: -boltdb.shipper.compactor.retention-delete-delay
[retention_delete_delay: <duration> | default = 2h]

# The total amount of worker to use to delete chunks.
# CLI flag: -boltdb.shipper.compactor.retention-delete-worker-count
[retention_delete_worker_count: <int> | default = 150]

# Allow cancellation of delete request until duration after they are created.
# Data would be deleted only after delete requests have been older than this duration.
# Ideally this should be set to at least 24h.
# CLI flag: -boltdb.shipper.compactor.delete-request-cancel-period
[delete_request_cancel_period: <duration> | default = 24h]

# Which deletion mode to use. Supported values are: disabled,
# whole-stream-deletion, filter-only, and filter-and-delete.
# CLI flag: -boltdb.shipper.compactor.deletion-mode
[deletion_mode: <string> | default = "whole-stream-deletion"]

# Maximum number of tables to compact in parallel.
# While increasing this value, please make sure compactor has enough disk space
# allocated to be able to store and compact as many tables.
# CLI flag: -boltdb.shipper.compactor.max-compaction-parallelism
[max_compaction_parallelism: <int> | default = 1]

# Deletion mode.
# Can be one of "disabled", "filter-only", or "filter-and-delete".
# When set to "filter-only" or "filter-and-delete", and if
# retention_enabled is true, then the log entry deletion API endpoints are available.
# CLI flag: -boltdb.shipper.compactor.deletion-mode
[deletion_mode: <string> | default = "disabled"]

# The hash ring configuration used by compactors to elect a single instance for running compactions
# The CLI flags prefix for this block config is: boltdb.shipper.compactor.ring
[compactor_ring: <ring>]

limits_config

The limits_config block configures global and per-tenant limits in Loki.

# Whether the ingestion rate limit should be applied individually to each
# distributor instance (local), or evenly shared across the cluster (global).
# The ingestion rate strategy cannot be overridden on a per-tenant basis.
#
# - local: enforces the limit on a per distributor basis. The actual effective
#   rate limit will be N times higher, where N is the number of distributor
#   replicas.
# - global: enforces the limit globally, configuring a per-distributor local
#   rate limiter as "ingestion_rate / N", where N is the number of distributor
#   replicas (it's automatically adjusted if the number of replicas change).
#   The global strategy requires the distributors to form their own ring, which
#   is used to keep track of the current number of healthy distributor replicas.
# CLI flag: -distributor.ingestion-rate-limit-strategy
[ingestion_rate_strategy: <string> | default = "global"]

# Per-user ingestion rate limit in sample size per second. Units in MB.
# CLI flag: -distributor.ingestion-rate-limit-mb
[ingestion_rate_mb: <float> | default = 4]

# Per-user allowed ingestion burst size (in sample size). Units in MB.
# The burst size refers to the per-distributor local rate limiter even in the
# case of the "global" strategy, and should be set at least to the maximum logs
# size expected in a single push request.
# CLI flag: -distributor.ingestion-burst-size-mb
[ingestion_burst_size_mb: <int> | default = 6]

# Maximum length of a label name.
# CLI flag: -validation.max-length-label-name
[max_label_name_length: <int> | default = 1024]

# Maximum length of a label value.
# CLI flag: -validation.max-length-label-value
[max_label_value_length: <int> | default = 2048]

# Maximum number of label names per series.
# CLI flag: -validation.max-label-names-per-series
[max_label_names_per_series: <int> | default = 30]

# Whether or not old samples will be rejected.
# CLI flag: -validation.reject-old-samples
[reject_old_samples: <boolean> | default = true]

# Maximum accepted sample age before rejecting.
# CLI flag: -validation.reject-old-samples.max-age
[reject_old_samples_max_age: <duration> | default = 168h]

# Duration for a table to be created/deleted before/after it's
# needed. Samples won't be accepted before this time.
# CLI flag: -validation.create-grace-period
[creation_grace_period: <duration> | default = 10m]

# Enforce every sample has a metric name.
# CLI flag: -validation.enforce-metric-name
[enforce_metric_name: <boolean> | default = true]

# Maximum line size on ingestion path. Example: 256kb.
# There is no limit when unset or set to 0.
# CLI flag: -distributor.max-line-size
[max_line_size: <string> | default = 0 ]

# Truncate log lines when they exceed max_line_size.
# CLI flag: -distributor.max-line-size-truncate
[max_line_size_truncate: <boolean> | default = false ]

# Alter the log line timestamp during ingestion when the timestamp is the same as the
# previous entry for the same stream. When enabled, if a log line in a push request has
# the same timestamp as the previous line for the same stream, one nanosecond is added
# to the log line. This will preserve the received order of log lines with the exact
# same timestamp when they are queried, by slightly altering their stored timestamp.
# NOTE: This is imperfect, because Loki accepts out of order writes, and another push
# request for the same stream could contain duplicate timestamps to existing
# entries and they will not be incremented.
# CLI flag: -validation.increment-duplicate-timestamps
[increment_duplicate_timestamp: <boolean> | default = false ]

# Maximum number of log entries that will be returned for a query.
# CLI flag: -validation.max-entries-limit
[max_entries_limit_per_query: <int> | default = 5000 ]

# Maximum number of active streams per user, per ingester. 0 to make it unlimited.
# CLI flag: -ingester.max-streams-per-user
[max_streams_per_user: <int> | default 0]

# Maximum number of active streams per user, across the cluster. 0 to disable.
# When the global limit is enabled, each ingester is configured with a dynamic
# local limit based on the replication factor and the current number of healthy
# ingesters, and is kept updated whenever the number of ingesters change.
# CLI flag: -ingester.max-global-streams-per-user
[max_global_streams_per_user: <int> | default = 5000]

# When true, out-of-order writes are accepted.
# CLI flag: -ingester.unordered-writes
[unordered_writes: <boolean> | default = true]

# Maximum number of chunks that can be fetched by a single query.
# CLI flag: -store.query-chunk-limit
[max_chunks_per_query: <int> | default = 2000000]

# The limit to length of chunk store queries. 0 to disable.
# CLI flag: -store.max-query-length
[max_query_length: <duration> | default = 721h]

# Maximum number of queries that will be scheduled in parallel by the frontend.
# CLI flag: -querier.max-query-parallelism
[max_query_parallelism: <int> | default = 32]

# Limit the maximum of unique series that is returned by a metric query.
# When the limit is reached an error is returned.
# CLI flag: -querier.max-query-series
[max_query_series: <int> | default = 500]

# Cardinality limit for index queries.
# CLI flag: -store.cardinality-limit
[cardinality_limit: <int> | default = 100000]

# Maximum number of stream matchers per query.
# CLI flag: -querier.max-streams-matcher-per-query
[max_streams_matchers_per_query: <int> | default = 1000]

# Maximum number of concurrent tail requests.
# CLI flag: -querier.max-concurrent-tail-requests
[max_concurrent_tail_requests: <int> | default = 10]

# Duration to delay the evaluation of rules to ensure.
# CLI flag: -ruler.evaluation-delay-duration
[ruler_evaluation_delay_duration: <duration> | default = 0s]

# Maximum number of rules per rule group per-tenant. 0 to disable.
# CLI flag: -ruler.max-rules-per-rule-group
[ruler_max_rules_per_rule_group: <int> | default = 0]

# Maximum number of rule groups per-tenant. 0 to disable.
# CLI flag: -ruler.max-rule-groups-per-tenant
[ruler_max_rule_groups_per_tenant: <int> | default = 0]

# Retention to apply for the store, if the retention is enable on the compactor side.
# CLI flag: -store.retention
[retention_period: <duration> | default = 744h]

# Per-stream retention to apply, if the retention is enable on the compactor side.
# Example:
# retention_stream:
# - selector: '{namespace="dev"}'
#   priority: 1
#   period: 24h
# - selector: '{container="nginx"}'
#   priority: 1
#   period: 744h
# Selector is a Prometheus labels matchers that will apply the `period` retention only if
# the stream is matching. In case multiple stream are matching, the highest
# priority will be picked. If no rule is matched the `retention_period` is used.
[retention_stream: <array> | default = none]

# Feature renamed to 'runtime configuration', flag deprecated in favor of -runtime-config.file
# (runtime_config.file in YAML).
# CLI flag: -limits.per-user-override-config
[per_tenant_override_config: <string>]

# Feature renamed to 'runtime configuration'; flag deprecated in favor of
# -runtime-config.reload-period (runtime_config.period in YAML).
# CLI flag: -limits.per-user-override-period
[per_tenant_override_period: <duration> | default = 10s]

# Most recent allowed cacheable result per-tenant, to prevent caching very recent results that
# might still be in flux.
# CLI flag: -frontend.max-cache-freshness
[max_cache_freshness_per_query: <duration> | default = 1m]

# Maximum number of queriers that can handle requests for a single tenant. If
# set to 0 or value higher than number of available queriers, *all* queriers
# will handle requests for the tenant. Each frontend (or query-scheduler, if
# used) will select the same set of queriers for the same tenant (given that all
# queriers are connected to all frontends / query-schedulers). This option only
# works with queriers connecting to the query-frontend / query-scheduler, not
# when using downstream URL.
# CLI flag: -frontend.max-queriers-per-tenant
[max_queriers_per_tenant: <int> | default = 0]

# Maximum byte rate per second per stream,
# also expressible in human readable forms (1MB, 256KB, etc).
# CLI flag: -ingester.per-stream-rate-limit
[per_stream_rate_limit: <string|int> | default = "3MB"]

# Maximum burst bytes per stream,
# also expressible in human readable forms (1MB, 256KB, etc).
# This is how far above the rate limit a stream can "burst" before the stream is limited.
# CLI flag: -ingester.per-stream-rate-limit-burst
[per_stream_rate_limit_burst: <string|int> | default = "15MB"]

# Limit how far back in time series data and metadata can be queried,
# up until lookback duration ago.
# This limit is enforced in the query frontend, the querier and the ruler.
# If the requested time range is outside the allowed range, the request will not fail,
# but will be modified to only query data within the allowed time range.
# The default value of 0 does not set a limit.
# CLI flag: -querier.max-query-lookback
[max_query_lookback: <duration> | default = 0]

# Disable recording rules remote-write.
[ruler_remote_write_disabled: <boolean> | default = false]

# The URL of the endpoint to send samples to.
[ruler_remote_write_url: <string>]

# Timeout for requests to the remote write endpoint.
[ruler_remote_write_timeout: <duration>]

# Custom HTTP headers to be sent along with each remote write request.
# Be aware that headers that are set by Loki itself can't be overwritten.
[ruler_remote_write_headers: <headers>]

# List of remote write relabel configurations.
[ruler_remote_write_relabel_configs: <relabel_config>]

# Number of samples to buffer per shard before we block reading of more
# samples from the WAL. It is recommended to have enough capacity in each
# shard to buffer several requests to keep throughput up while processing
# occasional slow remote requests.
[ruler_remote_write_queue_capacity: <int>]

# Minimum number of shards, i.e. amount of concurrency.
[ruler_remote_write_queue_min_shards: <int>]

# Maximum number of shards, i.e. amount of concurrency.
[ruler_remote_write_queue_max_shards: <int>]

# Maximum number of samples per send.
[ruler_remote_write_queue_max_samples_per_send: <int>]

# Maximum time a sample will wait in buffer.
[ruler_remote_write_queue_batch_send_deadline: <duration>]

# Initial retry delay. Gets doubled for every retry.
[ruler_remote_write_queue_min_backoff: <duration>]

# Maximum retry delay.
[ruler_remote_write_queue_max_backoff: <duration>]
# Retry upon receiving a 429 status code from the remote-write storage.
# This is experimental and might change in the future.
[ruler_remote_write_queue_retry_on_ratelimit: <boolean>]

# Limit queries that can be sharded.
# Queries within the time range of now and now minus this sharding lookback
# are not sharded. The default value of 0s disables the lookback, causing
# sharding of all queries at all times.
# CLI flag: -frontend.min-sharding-lookback
[min_sharding_lookback: <duration> | default = 0s]

# Split queries by an interval and execute in parallel, any value less than zero disables it.
# This also determines how cache keys are chosen when result caching is enabled
# CLI flag: -querier.split-queries-by-interval
[split_queries_by_interval: <duration> | default = 30m]

# When true, access to the deletion API is enabled.
# CLI flag: -compactor.allow_deletes
[allow_deletes: <boolean> | default = false]

grpc_client_config

The grpc_client_config block configures a client connection to a gRPC service.

# The maximum size in bytes the client can receive.
# CLI flag: -<prefix>.grpc-max-recv-msg-size
[max_recv_msg_size: <int> | default = 104857600]

# The maximum size in bytes the client can send.
# CLI flag: -<prefix>.grpc-max-send-msg-size
[max_send_msg_size: <int> | default = 16777216]

# Use compression when sending messages. Supported values are: 'gzip', 'snappy',
# and '' (disable compression).
# CLI flag: -<prefix>.grpc-compression
[grpc_compression: <string> | default = '']

# Rate limit for gRPC client. 0 is disabled.
# CLI flag: -<prefix>.grpc-client-rate-limit
[rate_limit: <float> | default = 0]

# Rate limit burst for gRPC client.
# CLI flag: -<prefix>.grpc-client-rate-limit-burst
[rate_limit_burst: <int> | default = 0]

# Enable backoff and retry when a rate limit is hit.
# CLI flag: -<prefix>.backoff-on-ratelimits
[backoff_on_ratelimits: <boolean> | default = false]

# Configures backoff when enabled.
backoff_config:
  # Minimum delay when backing off.
  # CLI flag: -<prefix>.backoff-min-period
  [min_period: <duration> | default = 100ms]

  # The maximum delay when backing off.
  # CLI flag: -<prefix>.backoff-max-period
  [max_period: <duration> | default = 10s]

  # Number of times to backoff and retry before failing.
  # CLI flag: -<prefix>.backoff-retries
  [max_retries: <int> | default = 10]

index_gateway

The index_gateway block configures the Loki index gateway server, responsible for serving index queries without the need to constantly interact with the object store.

# Defines in which mode the index gateway server will operate (default to 'simple').
# It supports two modes:
# 'simple': an index gateway server instance is responsible for handling,
#     storing and returning requests for all indices for all tenants.
# 'ring': an index gateway server instance is responsible for a subset of tenants instead
#     of all tenants.
[mode: <string> | default = simple]

# Defines the ring to be used by the index gateway servers and clients in case the servers
# are configured to run in 'ring' mode. In case this isn't configured, this block supports
# inheriting configuration from the common ring section.
[ring: <ring>]

table_manager

The table_manager block configures the Loki table-manager.

# Master 'off-switch' for table capacity updates, e.g. when troubleshooting.
# CLI flag: -table-manager.throughput-updates-disabled
[throughput_updates_disabled: <boolean> | default = false]

# Master 'on-switch' for table retention deletions.
# CLI flag: -table-manager.retention-deletes-enabled
[retention_deletes_enabled: <boolean> | default = false]

# How far back tables will be kept before they are deleted. 0s disables
# deletion. The retention period must be a multiple of the index / chunks
# table "period" (see period_config).
# CLI flag: -table-manager.retention-period
[retention_period: <duration> | default = 0s]

# Period with which the table manager will poll for tables.
# CLI flag: -table-manager.poll-interval
[poll_interval: <duration> | default = 2m]

# Duration a table will be created before it is needed.
# CLI flag: -table-manager.periodic-table.grace-period
[creation_grace_period: <duration> | default = 10m]

# Configures management of the index tables for DynamoDB.
# The CLI flags prefix for this block config is: table-manager.index-table
index_tables_provisioning: <provision_config>

# Configures management of the chunk tables for DynamoDB.
# The CLI flags prefix for this block config is: table-manager.chunk-table
chunk_tables_provisioning: <provision_config>

provision_config

The provision_config block configures provisioning capacity for DynamoDB.

# Enables on-demand throughput provisioning for the storage
# provider, if supported. Applies only to tables which are not autoscaled.
# CLI flag: -<prefix>.enable-ondemand-throughput-mode
[enable_ondemand_throughput_mode: <boolean> | default = false]

# DynamoDB table default write throughput.
# CLI flag: -<prefix>.write-throughput
[provisioned_write_throughput: <int> | default = 3000]

# DynamoDB table default read throughput.
# CLI flag: -<prefix>.read-throughput
[provisioned_read_throughput: <int> | default = 300]

# Enables on-demand throughput provisioning for the storage provide,
# if supported. Applies only to tables which are not autoscaled.
# CLI flag: -<prefix>.inactive-enable-ondemand-throughput-mode
[enable_inactive_throughput_on_demand_mode: <boolean> | default = false]

# DynamoDB table write throughput for inactive tables.
# CLI flag: -<prefix>.inactive-write-throughput
[inactive_write_throughput: <int> | default = 1]

# DynamoDB table read throughput for inactive tables.
# CLI flag: -<prefix>.inactive-read-throughput
[inactive_read_throughput: <int> | default = 300]

# Active table write autoscale config.
# The CLI flags prefix for this block config is: -<prefix>.write-throughput
[write_scale: <auto_scaling_config>]

# Inactive table write autoscale config.
# The CLI flags prefix for this block config is: -<prefix>.inactive-write-throughput
[inactive_write_scale: <auto_scaling_config>]

# Number of last inactive tables to enable write autoscale.
# CLI flag: -<prefix>.enable-ondemand-throughput-mode
[inactive_write_scale_lastn: <int>]

# Active table read autoscale config.
# The CLI flags prefix for this block config is: -<prefix>.read-throughput
[read_scale: <auto_scaling_config>]

# Inactive table read autoscale config.
# The CLI flags prefix for this block config is: -<prefix>.inactive-read-throughput
[inactive_read_scale: <auto_scaling_config>]

# Number of last inactive tables to enable read autoscale.
# CLI flag: -<prefix>.enable-ondemand-throughput-mode
[inactive_read_scale_lastn: <int>]

auto_scaling_config

The auto_scaling_config block configures autoscaling for DynamoDB.

# Whether or not autoscaling should be enabled.
# CLI flag: -<prefix>.scale.enabled
[enabled: <boolean>: default = false]

# AWS AutoScaling role ARN.
# CLI flag: -<prefix>.scale.role-arn
[role_arn: <string>]

# DynamoDB minimum provision capacity.
# CLI flag: -<prefix>.scale.min-capacity
[min_capacity: <int> | default = 3000]

# DynamoDB maximum provision capacity.
# CLI flag: -<prefix>.scale.max-capacity
[max_capacity: <int> | default = 6000]

# DynamoDB minimum seconds between each autoscale up.
# CLI flag: -<prefix>.scale.out-cooldown
[out_cooldown: <int> | default = 1800]

# DynamoDB minimum seconds between each autoscale down.
# CLI flag: -<prefix>.scale.in-cooldown
[in_cooldown: <int> | default = 1800]

# DynamoDB target ratio of consumed capacity to provisioned capacity.
# CLI flag: -<prefix>.scale.target-value
[target: <float> | default = 80]

tracing

The tracing block configures tracing for Jaeger. Currently limited to disable auto-configuration per environment variables only.

# Whether or not tracing should be enabled.
# CLI flag: -tracing.enabled
[enabled: <boolean>: default = true]

common

The common block sets common definitions to be shared by different components. This way, one doesn’t have to replicate configuration in multiple places.

# A common storage configuration to be used by the different Loki components.
[storage: <storage>]

# When defined, the given prefix will be present in front of the endpoint paths.
[path_prefix: <string>]

# How many times incoming data should be replicated to the ingester component.
[replication_factor: <int> | default = 3]

# When true, the ingester, compactor, and query_scheduler ring tokens will be saved
# to files in the path_prefix directory. Loki will error if you set this to true
# and path_prefix is empty.
[persist_tokens: <boolean>: default = false]

# A common list of net interfaces used internally to look for addresses.
# If a more specific "instance_interface_names" is set, this is ignored.
# If "instance_interface_names" under the common ring section is configured,
# this common "instance_interface_names" is only applied to the frontend, but not for
# ring related components (ex: distributor, ruler, etc).
[instance_interface_names: <list of string> | default = [<private network interfaces>]]

# A common address used by Loki components to advertise their address.
# If a more specific "instance_addr" is set, this is ignored.
# If "instance_addr" under the common ring section is configured, this common "instance_addr"
# is only applied to the frontend, but not for ring related components (ex: distributor, ruler, etc).
[instance_addr: <string>]

# A common ring configuration to be used by all Loki rings.
# If a common ring is given, its values are used to define any undefined ring values.
# For instance, you can expect the `heartbeat_period` defined in the common section
# to be used by the distributor's ring, but only if the distributor's ring itself
# doesn't have a `heartbeat_period` set.
[ring: <ring>]

# Address and port number where the compactor API is served.
# CLI flag: -common.compactor-address
[compactor_address: <string> | default = ""]

analytics

The analytics block configures the reporting of Loki analytics to grafana.com.

# By default, Loki will send anonymous, but uniquely-identifiable usage and configuration
# analytics to Grafana Labs. These statistics are sent to https://stats.grafana.org/
#
# Statistics help us better understand how Loki is used, and they show us performance
# levels for most users. This helps us prioritize features and documentation.
# For more information on what's sent, look at
# https://github.com/grafana/loki/blob/main/pkg/usagestats/stats.go
# Refer to the buildReport method to see what goes into a report.
#
# When true, enables usage reporting.
# CLI flag: -reporting.enabled
[reporting_enabled: <boolean>: default = true]

storage

The common storage block defines a common storage to be reused by different components as a way to facilitate storage configuration. If any specific configuration for an object storage client have been provided elsewhere in the configuration file, the specific configuration will supersede the common storage configuration.

# Configures Azure as the common storage.
[azure: <azure_storage_config>]

# Configures GCS as the common storage.
[gcs: <gcs_storage_config>]

# Configures S3 as the common storage.
[s3: <s3_storage_config>]

# Configures Swift as the common storage.
[swift: <swift_storage_config>]

# Configures a (local) file system as the common storage.
[filesystem: <filesystem>]

# Configures Baidu Object Storage (BOS) as the common storage.
[bos: <bos_storage_config>]

# The `hedging_config` configures how to hedge requests for the storage.
[hedging: <hedging_config>]

filesystem

The common filesystem block configures a local file system as a general storage for various types of data generated by Loki.

# File system directory to be used for chunks storage.
[chunks_directory: <filename> | default = ""]

# File system directory to be used for rules storage.
[rules_directory: <filename> | default = ""]

ring

The common ring block defines a ring configuration used by a Loki component.

# The key-value store used to share the hash ring across multiple instances.
kvstore:
  # Backend storage to use for the ring. Supported values are: consul, etcd,
  # inmemory, memberlist, multi.
  # CLI flag: -<prefix>.store
  [store: <string> | default = "memberlist"]

  # The prefix for the keys in the store. Should end with a /.
  # CLI flag: -<prefix>.prefix
  [prefix: <string> | default = "collectors/"]

  # The consul_config configures the consul client.
  [consul: <consul_config>]

  # The etcd_config configures the etcd client.
  [etcd: <etcd_config>]

  multi:
    # Primary backend storage used by multi-client.
    # CLI flag: -<prefix>.multi.primary
    [primary: <string> | default = ""]

    # Secondary backend storage used by multi-client.
    # CLI flag: -<prefix>.multi.secondary
    [secondary: <string> | default = ""]

    # Mirror writes to secondary store.
    # CLI flag: -<prefix>.multi.mirror-enabled
    [mirror_enabled: <boolean> | default = false]

    # Timeout for storing value to secondary store.
    # CLI flag: -<prefix>.multi.mirror-timeout
    [mirror_timeout: <duration> | default = 2s]

# Interval between heartbeats sent to the ring. 0 = disabled.
# CLI flag: -<prefix>.heartbeat-period
[heartbeat_period: <duration> | default = 15s]

# The heartbeat timeout after which store gateways are considered unhealthy
# within the ring. 0 = never (timeout disabled). This option needs be set both
# on the store-gateway and querier when running in microservices mode.
# CLI flag: -<prefix>.heartbeat-timeout
[heartbeat_timeout: <duration> | default = 1m]

# File path where tokens are stored. If empty, tokens are neither stored at
# shutdown nor restored at startup.
# CLI flag: -<prefix>.tokens-file-path
[tokens_file_path: <string> | default = ""]

# True to enable zone-awareness and replicate blocks across different
# availability zones.
# CLI flag: -<prefix>.zone-awareness-enabled
[zone_awareness_enabled: <boolean> | default = false]

# Name of network interface to read addresses from.
# CLI flag: -<prefix>.instance-interface-names
[instance_interface_names: <list of string> | default = [<private network interfaces>]]

# IP address to advertise in the ring.
# CLI flag: -<prefix>.instance-addr
[instance_addr: <list of string> | default = first from instance_interface_names]

# Port to advertise in the ring
# CLI flag: -<prefix>.instance-port
[instance_port: <list of string> | default = server.grpc-listen-port]

# Instance ID to register in the ring.
# CLI flag: -<prefix>.instance-id
[instance_id: <list of string> | default = os.Hostname()]

# The availability zone where this instance is running. Required if
# zone-awareness is enabled.
# CLI flag: -<prefix>.instance-availability-zone
[instance_availability_zone: <string> | default = ""]

Runtime Configuration file

Loki has a concept of “runtime config” file, which is simply a file that is reloaded while Loki is running. It is used by some Loki components to allow operator to change some aspects of Loki configuration without restarting it. File is specified by using -runtime-config.file=<filename> flag and reload period (which defaults to 10 seconds) can be changed by -runtime-config.reload-period=<duration> flag. Previously this mechanism was only used by limits overrides, and flags were called -limits.per-user-override-config=<filename> and -limits.per-user-override-period=10s respectively. These are still used, if -runtime-config.file=<filename> is not specified.

At the moment, two components use runtime configuration: limits and multi KV store.

Options for runtime configuration reload can also be configured via YAML:

# Configuration file to periodically check and reload.
[file: <string>: default = empty]

# How often to check the file.
[period: <duration>: default 10s]

Example runtime configuration file:

overrides:
  tenant1:
    ingestion_rate_mb: 10
    max_streams_per_user: 100000
    max_chunks_per_query: 100000
  tenant2:
    max_streams_per_user: 1000000
    max_chunks_per_query: 1000000

multi_kv_config:
    mirror-enabled: false
    primary: consul

Accept out-of-order writes

Since the beginning of Loki, log entries had to be written to Loki in order by time. This limitation has been lifted. Out-of-order writes are enabled globally by default, but can be disabled/enabled on a cluster or per-tenant basis.

To disable out-of-order writes for all tenants, place in the limits_config section:
```
limits_config:
    unordered_writes: false
```
To disable out-of-order writes for specific tenants, configure a runtime configuration file:
```
runtime_config: overrides.yaml
```
In the overrides.yaml file, add unordered_writes for each tenant permitted to have out-of-order writes:
```
overrides:
  "tenantA":
    unordered_writes: false
```

How far into the past accepted out-of-order log entries may be is configurable with max_chunk_age. max_chunk_age defaults to 1 hour. Loki calculates the earliest time that out-of-order entries may have and be accepted with

time_of_most_recent_line - (max_chunk_age/2)

Log entries with timestamps that are after this earliest time are accepted. Log entries further back in time return an out-of-order error.

For example, if max_chunk_age is 2 hours and the stream {foo="bar"} has one entry at 8:00, Loki will accept data for that stream as far back in time as 7:00. If another log line is written at 10:00, Loki will accept data for that stream as far back in time as 9:00.

Clients

Mon, 14 Apr 2025 21:05:47 +0000

Grafana Loki clients

The following clients are developed and supported (for those customers who have purchased a support contract) by Grafana Labs for sending logs to Loki:

Grafana Agent - The Grafana Agent is the recommended client for the Grafana stack. It can collect telemetry data for metrics, logs, traces, and continuous profiles and is fully compatible with the Prometheus, OpenTelemetry, and Grafana open source ecosystems.
Promtail - Promtail is a popular choice when you’re running Kubernetes, as you can configure it to automatically scrape logs from pods running on the same node that Promtail runs on. Promtail and Prometheus running together in Kubernetes enables powerful debugging: if Prometheus and Promtail use the same labels, users can use tools like Grafana to switch between metrics and logs based on the label set.
Promtail is also the client of choice on bare-metal since it can be configured to tail logs from all files given a host path. It is the easiest way to send logs to Loki from plain-text files (for example, things that log to /var/log/*.log). Lastly, Promtail works well if you want to extract metrics from logs such as counting the occurrences of a particular message.
xk6-loki - The xk6-loki extension lets you perform load testing on Loki.

Third-party clients

The following clients have been developed by the Loki community or other third-parties and can be used to send log data to Loki.

Note
Grafana Labs cannot provide support for third-party clients. Once an issue has been determined to be with the client and not Loki, it is the responsibility of the customer to work with the associated vendor or project for bug fixes to these clients.

The following are popular third-party Loki clients:

Docker Driver - When using Docker and not Kubernetes, the Docker logging driver for Loki should be used as it automatically adds labels appropriate to the running container.
Fluent Bit - The Fluent Bit plugin is ideal when you already have Fluentd deployed and you already have configured Parser and Filter plugins.
Fluentd - The Fluentd plugin is ideal when you already have Fluentd deployed and you already have configured Parser and Filter plugins. Fluentd also works well for extracting metrics from logs when using itsPrometheus plugin.
Lambda Promtail - This is a workflow combining the Promtail push-api scrape config and the lambda-promtail AWS Lambda function which pipes logs from Cloudwatch to Loki. This is a good choice if you’re looking to try out Loki in a low-footprint way or if you wish to monitor AWS lambda logs in Loki.
Logstash - If you are already using logstash and/or beats, this will be the easiest way to start. By adding our output plugin you can quickly try Loki without doing big configuration changes.

Alerting and Recording Rules

Tue, 16 Jul 2024 15:42:20 +0000

Rules and the Ruler

Grafana Loki includes a component called the ruler. The ruler is responsible for continually evaluating a set of configurable queries and performing an action based on the result.

This example configuration sources rules from a local disk.

Ruler storage provides further details.

ruler:
  storage:
    type: local
    local:
      directory: /tmp/rules
  rule_path: /tmp/scratch
  alertmanager_url: http://localhost
  ring:
    kvstore:
      store: inmemory
  enable_api: true

We support two kinds of rules: alerting rules and recording rules.

Alerting Rules

We support Prometheus-compatible alerting rules. From Prometheus’ documentation:

Alerting rules allow you to define alert conditions based on Prometheus expression language expressions and to send notifications about firing alerts to an external service.

Loki alerting rules are exactly the same, except they use LogQL for their expressions.

Example

A complete example of a rules file:

groups:
  - name: should_fire
    rules:
      - alert: HighPercentageError
        expr: |
          sum(rate({app="foo", env="production"} |= "error" [5m])) by (job)
            /
          sum(rate({app="foo", env="production"}[5m])) by (job)
            > 0.05
        for: 10m
        labels:
            severity: page
        annotations:
            summary: High request latency
  - name: credentials_leak
    rules: 
      - alert: http-credentials-leaked
        annotations: 
          message: "{{ $labels.job }} is leaking http basic auth credentials."
        expr: 'sum by (cluster, job, pod) (count_over_time({namespace="prod"} |~ "http(s?)://(\\w+):(\\w+)@" [5m]) > 0)'
        for: 10m
        labels: 
          severity: critical

Recording Rules

We support Prometheus-compatible recording rules. From Prometheus’ documentation:

Recording rules allow you to precompute frequently needed or computationally expensive expressions and save their result as a new set of time series.

Querying the precomputed result will then often be much faster than executing the original expression every time it is needed. This is especially useful for dashboards, which need to query the same expression repeatedly every time they refresh.

Loki allows you to run metric queries over your logs, which means that you can derive a numeric aggregation from your logs, like calculating the number of requests over time from your NGINX access log.

Example

name: NginxRules
interval: 1m
rules:
  - record: nginx:requests:rate1m
    expr: |
      sum(
        rate({container="nginx"}[1m])
      )
    labels:
      cluster: "us-central1"

This query (expr) will be executed every 1 minute (interval), the result of which will be stored in the metric name we have defined (record). This metric named nginx:requests:rate1m can now be sent to Prometheus, where it will be stored just like any other metric.

Remote-Write

With recording rules, you can run these metric queries continually on an interval, and have the resulting metrics written to a Prometheus-compatible remote-write endpoint. They produce Prometheus metrics from log entries.

At the time of writing, these are the compatible backends that support this:

Prometheus (>=v2.25.0): Prometheus is generally a pull-based system, but since v2.25.0 has allowed for metrics to be written directly to it as well.
Grafana Mimir
Thanos (Receiver)

Here is an example remote-write configuration for sending to a local Prometheus instance:

ruler:
  ... other settings ...
  
  remote_write:
    enabled: true
    client:
      url: http://localhost:9090/api/v1/write

Further configuration options can be found under ruler.

Operations

Refer to the Recording Rules page.

Use cases

The Ruler’s Prometheus compatibility further accentuates the marriage between metrics and logs. For those looking to get started with metrics and alerts based on logs, or wondering why this might be useful, here are a few use cases we think fit very well.

Black box monitoring

We don’t always control the source code of applications we run. Load balancers and a myriad of other components, both open source and closed third-party, support our applications while they don’t expose the metrics we want. Some don’t expose any metrics at all. Loki’s alerting and recording rules can produce metrics and alert on the state of the system, bringing the components into our observability stack by using the logs. This is an incredibly powerful way to introduce advanced observability into legacy architectures.

Event alerting

Sometimes you want to know whether any instance of something has occurred. Alerting based on logs can be a great way to handle this, such as finding examples of leaked authentication credentials:

- name: credentials_leak
  rules: 
    - alert: http-credentials-leaked
      annotations: 
        message: "{{ $labels.job }} is leaking http basic auth credentials."
      expr: 'sum by (cluster, job, pod) (count_over_time({namespace="prod"} |~ "http(s?)://(\\w+):(\\w+)@" [5m]) > 0)'
      for: 10m
      labels: 
        severity: critical

Alerting on high-cardinality sources

Another great use case is alerting on high cardinality sources. These are things which are difficult/expensive to record as metrics because the potential label set is huge. A great example of this is per-tenant alerting in multi-tenanted systems like Loki. It’s a common balancing act between the desire to have per-tenant metrics and the cardinality explosion that ensues (adding a single tenant label to an existing Prometheus metric would increase it’s cardinality by the number of tenants).

Creating these alerts in LogQL is attractive because these metrics can be extracted at query time, meaning we don’t suffer the cardinality explosion in our metrics store.

Note As an example, we can use LogQL v2 to help Loki to monitor itself, alerting us when specific tenants have queries that take longer than 10s to complete! To do so, we’d use the following query: sum by (org_id) (rate({job="loki-prod/query-frontend"} |= "metrics.go" | logfmt | duration > 10s [1m]))

Interacting with the Ruler

Because the rule files are identical to Prometheus rule files, we can interact with the Loki Ruler via cortextool. The CLI is in early development, but it works with both Loki and Cortex. Pass the --backend=loki option when using it with Loki.

Note: Not all commands in cortextool currently support Loki.

Note: cortextool was intended to run against multi-tenant Loki, commands need an --id= flag set to the Loki instance ID or set the environment variable CORTEX_TENANT_ID. If Loki is running in single tenant mode, the required ID is fake (yes we know this might seem alarming but it’s totally fine, no it can’t be changed)

An example workflow is included below:

# lint the rules.yaml file ensuring it's valid and reformatting it if necessary
cortextool rules lint --backend=loki ./output/rules.yaml

# diff rules against the currently managed ruleset in Loki
cortextool rules diff --rule-dirs=./output --backend=loki

# ensure the remote ruleset matches your local ruleset, creating/updating/deleting remote rules which differ from your local specification.
cortextool rules sync --rule-dirs=./output --backend=loki

# print the remote ruleset
cortextool rules print --backend=loki

There is also a github action available for cortex-tool, so you can add it into your CI/CD pipelines!

For instance, you can sync rules on master builds via

name: sync-cortex-rules-and-alerts
on:
  push:
    branches:
      - master
env:
  CORTEX_ADDRESS: '<fill me in>'
  CORTEX_TENANT_ID: '<fill me in>'
  CORTEX_API_KEY: ${{ secrets.API_KEY }}
  RULES_DIR: 'output/'
jobs:
  sync-loki-alerts:
    runs-on: ubuntu-18.04
    steps:
      - name: Lint Rules
        uses: grafana/cortex-rules-action@v0.4.0
        env:
          ACTION: 'lint'
        with:
          args: --backend=loki
      - name: Diff rules
        uses: grafana/cortex-rules-action@v0.4.0
        env:
          ACTION: 'diff'
        with:
          args: --backend=loki
      - name: Sync rules
        if: ${{ !contains(steps.diff-rules.outputs.detailed, 'no changes detected') }}
        uses: grafana/cortex-rules-action@v0.4.0
        env:
          ACTION: 'sync'
        with:
          args: --backend=loki
      - name: Print rules
        uses: grafana/cortex-rules-action@v0.4.0
        env:
          ACTION: 'print'

Scheduling and best practices

One option to scale the Ruler is by scaling it horizontally. However, with multiple Ruler instances running they will need to coordinate to determine which instance will evaluate which rule. Similar to the ingesters, the Rulers establish a hash ring to divide up the responsibilities of evaluating rules.

The possible configurations are listed fully in the configuration documentation, but in order to shard rules across multiple Rulers, the rules API must be enabled via flag (-ruler.enable-api) or config file parameter. Secondly, the Ruler requires it’s own ring be configured. From there the Rulers will shard and handle the division of rules automatically. Unlike ingesters, Rulers do not hand over responsibility: all rules are re-sharded randomly every time a Ruler is added to or removed from the ring.

A full sharding-enabled Ruler example is:

ruler:
    alertmanager_url: <alertmanager_endpoint>
    enable_alertmanager_v2: true
    enable_api: true
    enable_sharding: true
    ring:
        kvstore:
            consul:
                host: consul.loki-dev.svc.cluster.local:8500
            store: consul
    rule_path: /tmp/rules
    storage:
        gcs:
            bucket_name: <loki-rules-bucket>

Ruler storage

The Ruler supports five kinds of storage: azure, gcs, s3, swift, and local. Most kinds of storage work with the sharded Ruler configuration in an obvious way, i.e. configure all Rulers to use the same backend.

The local implementation reads the rule files off of the local filesystem. This is a read-only backend that does not support the creation and deletion of rules through the Ruler API. Despite the fact that it reads the local filesystem this method can still be used in a sharded Ruler configuration if the operator takes care to load the same rules to every Ruler. For instance, this could be accomplished by mounting a Kubernetes ConfigMap onto every Ruler pod.

A typical local configuration might look something like:

  -ruler.storage.type=local
  -ruler.storage.local.directory=/tmp/loki/rules

With the above configuration, the Ruler would expect the following layout:

/tmp/loki/rules/<tenant id>/rules1.yaml
                           /rules2.yaml

Yaml files are expected to be Prometheus compatible but include LogQL expressions as specified in the beginning of this doc.

Future improvements

There are a few things coming to increase the robustness of this service. In no particular order:

WAL for recording rule.
Backend metric stores adapters for generated alert rule data.

Misc Details: Metrics backends vs in-memory

Currently the Loki Ruler is decoupled from a backing Prometheus store. Generally, the result of evaluating rules as well as the history of the alert’s state are stored as a time series. Loki is unable to store/retrieve these in order to allow it to run independently of i.e. Prometheus. As a workaround, Loki keeps a small in memory store whose purpose is to lazy load past evaluations when rescheduling or resharding Rulers. In the future, Loki will support optional metrics backends, allowing storage of these metrics for auditing & performance benefits.

LogQL

Tue, 16 Jul 2024 15:42:20 +0000

LogQL: Log query language

LogQL is Grafana Loki’s PromQL-inspired query language. Queries act as if they are a distributed grep to aggregate log sources. LogQL uses labels and operators for filtering.

There are two types of LogQL queries:

Log queries return the contents of log lines.
Metric queries extend log queries to calculate values based on query results.

Binary operators

Arithmetic operators

The following binary arithmetic operators exist in Loki:

+ (addition)
- (subtraction)
* (multiplication)
/ (division)
% (modulo)
^ (power/exponentiation)

Binary arithmetic operators are defined between two literals (scalars), a literal and a vector, and two vectors.

Between two literals, the behavior is obvious: They evaluate to another literal that is the result of the operator applied to both scalar operands (1 + 1 = 2).

Between a vector and a literal, the operator is applied to the value of every data sample in the vector, e.g. if a time series vector is multiplied by 2, the result is another vector in which every sample value of the original vector is multiplied by 2.

Between two vectors, a binary arithmetic operator is applied to each entry in the left-hand side vector and its matching element in the right-hand vector. The result is propagated into the result vector with the grouping labels becoming the output label set. Entries for which no matching entry in the right-hand vector can be found are not part of the result.

Pay special attention to operator order when chaining arithmetic operators.

Arithmetic Examples

Implement a health check with a simple query:

1 + 1

Double the rate of a a log stream’s entries:

sum(rate({app="foo"}[1m])) * 2

Get proportion of warning logs to error logs for the foo app

sum(rate({app="foo", level="warn"}[1m])) / sum(rate({app="foo", level="error"}[1m]))

Logical and set operators

These logical/set binary operators are only defined between two vectors:

and (intersection)
or (union)
unless (complement)

vector1 and vector2 results in a vector consisting of the elements of vector1 for which there are elements in vector2 with exactly matching label sets. Other elements are dropped.

vector1 or vector2 results in a vector that contains all original elements (label sets + values) of vector1 and additionally all elements of vector2 which do not have matching label sets in vector1.

vector1 unless vector2 results in a vector consisting of the elements of vector1 for which there are no elements in vector2 with exactly matching label sets. All matching elements in both vectors are dropped.

Binary operators examples

This contrived query will return the intersection of these queries, effectively rate({app="bar"}):

rate({app=~"foo|bar"}[1m]) and rate({app="bar"}[1m])

Comparison operators

== (equality)
!= (inequality)
> (greater than)
>= (greater than or equal to)
< (less than)
<= (less than or equal to)

Comparison operators are defined between scalar/scalar, vector/scalar, and vector/vector value pairs. By default they filter. Their behavior can be modified by providing bool after the operator, which will return 0 or 1 for the value rather than filtering.

Between two scalars, these operators result in another scalar that is either 0 (false) or 1 (true), depending on the comparison result. The bool modifier must not be provided.

1 >= 1 is equivalent to 1

Between a vector and a scalar, these operators are applied to the value of every data sample in the vector, and vector elements between which the comparison result is false get dropped from the result vector. If the bool modifier is provided, vector elements that would be dropped instead have the value 0 and vector elements that would be kept have the value 1.

Filters the streams which logged at least 10 lines in the last minute:

count_over_time({foo="bar"}[1m]) > 10

Attach the value(s) 0/1 to streams that logged less/more than 10 lines:

count_over_time({foo="bar"}[1m]) > bool 10

Between two vectors, these operators behave as a filter by default, applied to matching entries. Vector elements for which the expression is not true or which do not find a match on the other side of the expression get dropped from the result, while the others are propagated into a result vector. If the bool modifier is provided, vector elements that would have been dropped instead have the value 0 and vector elements that would be kept have the value 1, with the grouping labels again becoming the output label set.

Return the streams matching app=foo without app labels that have higher counts within the last minute than their counterparts matching app=bar without app labels:

sum without(app) (count_over_time({app="foo"}[1m])) > sum without(app) (count_over_time({app="bar"}[1m]))

Same as above, but vectors have their values set to 1 if they pass the comparison or 0 if they fail/would otherwise have been filtered out:

sum without(app) (count_over_time({app="foo"}[1m])) > bool sum without(app) (count_over_time({app="bar"}[1m]))

Order of operations

When chaining or combining operators, you have to consider operator precedence: Generally, you can assume regular mathematical convention with operators on the same precedence level being left-associative.

More details can be found in the Golang language documentation.

1 + 2 / 3 is equal to 1 + ( 2 / 3 ).

2 * 3 % 2 is evaluated as (2 * 3) % 2.

Keywords on and ignoring

The ignoring keyword causes specified labels to be ignored during matching. The syntax:

<vector expr> <bin-op> ignoring(<labels>) <vector expr>

This example will return the machines which total count within the last minutes exceed average value for app foo.

max by(machine) (count_over_time({app="foo"}[1m])) > bool ignoring(machine) avg(count_over_time({app="foo"}[1m]))

The on keyword reduces the set of considered labels to a specified list. The syntax:

<vector expr> <bin-op> on(<labels>) <vector expr>

This example will return every machine total count within the last minutes ratio in app foo:

sum by(machine) (count_over_time({app="foo"}[1m])) / on() sum(count_over_time({app="foo"}[1m]))

Many-to-one and one-to-many vector matches

Many-to-one and one-to-many matchings occur when each vector element on the “one”-side can match with multiple elements on the “many”-side. You must explicitly request matching by using the group_left or group_right modifier, where left or right determines which vector has the higher cardinality. The syntax:

<vector expr> <bin-op> ignoring(<labels>) group_left(<labels>) <vector expr>
<vector expr> <bin-op> ignoring(<labels>) group_right(<labels>) <vector expr>
<vector expr> <bin-op> on(<labels>) group_left(<labels>) <vector expr>
<vector expr> <bin-op> on(<labels>) group_right(<labels>) <vector expr>

The label list provided with the group modifier contains additional labels from the “one”-side that are included in the result metrics. And a label should only appear in one of the lists specified by on and group_x. Every time series of the result vector must be uniquely identifiable. Grouping modifiers can only be used for comparison and arithmetic. By default, the system matches and, unless, and or operations with all entries in the right vector.

The following example returns the rates requests partitioned by app and status as a percentage of total requests.

sum by (app, status) (
  rate(
    {job="http-server"}
      | json
      [5m]
  )
)
/ on (app) group_left
sum by (app) (
  rate(
    {job="http-server"}
      | json
      [5m]
  )
)

=>
[
  {app="foo", status="200"} => 0.8
  {app="foo", status="400"} => 0.1
  {app="foo", status="500"} => 0.1
]

This version uses group_left(<labels>) to include <labels> from the right hand side in the result and returns the cost of discarded events per user, organization, and namespace:

sum by (user, namespace) (
  rate(
    {job="events"}
      | logfmt
      | discarded="true"
      [5m]
  )
)
* on (user) group_left(organization)
max_over_time(
  {job="cost-calculator"}
    | logfmt
    | unwrap cost
    [5m]
) by (user, organization)

=>
[
  {user="foo", namespace="dev", organization="little-org"} => 10
  {user="foo", namespace="prod", organization="little-org"} => 50
  {user="bar", namespace="dev", organization="big-org"} => 70
  {user="bar", namespace="prod", organization="big-org"} => 200
]

Comments

LogQL queries can be commented using the # character:

{app="foo"} # anything that comes after will not be interpreted in your query

With multi-line LogQL queries, the query parser can exclude whole or partial lines using #:

{app="foo"}
    | json
    # this line will be ignored
    | bar="baz" # this checks if bar = "baz"

Pipeline Errors

There are multiple reasons which cause pipeline processing errors, such as:

A numeric label filter may fail to turn a label value into a number
A metric conversion for a label may fail.
A log line is not a valid json document.
etc…

When those failures happen, Loki won’t filter out those log lines. Instead they are passed into the next stage of the pipeline with a new system label named __error__. The only way to filter out errors is by using a label filter expressions. The __error__ label can’t be renamed via the language.

For example to remove json errors:

  {cluster="ops-tools1",container="ingress-nginx"}
    | json
    | __error__ != "JSONParserErr"

Alternatively you can remove all error using a catch all matcher such as __error__ = "" or even show only errors using __error__ != "".

The filter should be placed after the stage that generated this error. This means if you need to remove errors from an unwrap expression it needs to be placed after the unwrap.

quantile_over_time(
	0.99,
	{container="ingress-nginx",service="hosted-grafana"}
	| json
	| unwrap response_latency_seconds
	| __error__=""[1m]
	) by (cluster)

Metric queries cannot contain errors, in case errors are found during execution, Loki will return an error and appropriate status code.

Functions

Loki supports functions to operate on data.

label_replace()

For each timeseries in v,

label_replace(v instant-vector,
    dst_label string,
    replacement string,
    src_label string,
    regex string)

matches the regular expression regex against the label src_label. If it matches, then the timeseries is returned with the label dst_label replaced by the expansion of replacement.

$1 is replaced with the first matching subgroup, $2 with the second etc. If the regular expression doesn’t match, then the timeseries is returned unchanged.

This example will return a vector with each time series having a foo label with the value a added to it:

label_replace(rate({job="api-server",service="a:c"} |= "err" [1m]), "foo", "$1",
  "service", "(.*):.*")

Operations

Tue, 16 Jul 2024 15:42:20 +0000

Operations

HTTP API

Tue, 16 Jul 2024 15:42:20 +0000

Grafana Loki HTTP API

Grafana Loki exposes an HTTP API for pushing, querying, and tailing log data. Note that authenticating against the API is out of scope for Loki.

Microservices mode

When deploying Loki in microservices mode, the set of endpoints exposed by each component is different.

These endpoints are exposed by all components:

These endpoints are exposed by the querier and the query frontend:

GET /loki/api/v1/query
GET /loki/api/v1/query_range
GET /loki/api/v1/labels
GET /loki/api/v1/label/<name>/values
GET /loki/api/v1/series
GET /loki/api/v1/index/stats
GET /loki/api/v1/tail
POST /loki/api/v1/push
GET /ready
GET /metrics
Deprecated GET /api/prom/tail
Deprecated GET /api/prom/query
Deprecated GET /api/prom/label
Deprecated GET /api/prom/label/<name>/values
Deprecated POST /api/prom/push

These endpoints are exposed by the distributor:

These endpoints are exposed by the ingester:

POST /flush
POST /ingester/shutdown
Deprecated POST /ingester/flush_shutdown

The API endpoints starting with /loki/ are Prometheus API-compatible and the result formats can be used interchangeably.

These endpoints are exposed by the ruler:

These endpoints are exposed by the compactor:

A list of clients can be found in the clients documentation.

Matrix, vector, and streams

Some Loki API endpoints return a result of a matrix, a vector, or a stream:

Matrix: a table of values where each row represents a different label set and the columns are each sample value for that row over the queried time. Matrix types are only returned when running a query that computes some value.
Instant Vector: denoted in the type as just vector, an Instant Vector represents the latest value of a calculation for a given labelset. Instant Vectors are only returned when doing a query against a single point in time.
Stream: a Stream is a set of all values (logs) for a given label set over the queried time range. Streams are the only type that will result in log lines being returned.

Timestamp formats

The API accepts several formats for timestamps. An integer with ten or fewer digits is interpreted as a Unix timestamp in seconds. More than ten digits are interpreted as a Unix timestamp in nanoseconds. A floating point number is a Unix timestamp with fractions of a second.

The timestamps can also be written in RFC3339 and RFC3339Nano format, as supported by Go’s time package.

Query Loki

GET /loki/api/v1/query

/loki/api/v1/query allows for doing queries against a single point in time. The URL query parameters support the following values:

query: The LogQL query to perform
limit: The max number of entries to return. It defaults to 100. Only applies to query types which produce a stream(log lines) response.
time: The evaluation time for the query as a nanosecond Unix epoch or another supported format. Defaults to now.
direction: Determines the sort order of logs. Supported values are forward or backward. Defaults to backward.

In microservices mode, /loki/api/v1/query is exposed by the querier and the frontend.

Response format:

{
  "status": "success",
  "data": {
    "resultType": "vector" | "streams",
    "result": [<vector value>] | [<stream value>],
    "stats" : [<statistics>]
  }
}

where <vector value> is:

{
  "metric": {
    <label key-value pairs>
  },
  "value": [
    <number: second unix epoch>,
    <string: value>
  ]
}

and <stream value> is:

{
  "stream": {
    <label key-value pairs>
  },
  "values": [
    [
      <string: nanosecond unix epoch>,
      <string: log line>
    ],
    ...
  ]
}

The items in the values array are sorted by timestamp. The most recent item is first when using direction=backward. The oldest item is first when using direction=forward.

See statistics for information about the statistics returned by Loki.

Examples

This example query

curl -G -s  "http://localhost:3100/loki/api/v1/query" \
  --data-urlencode \
  'query=sum(rate({job="varlogs"}[10m])) by (level)' | jq

gave this response:

{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [
      {
        "metric": {},
        "value": [
          1588889221,
          "1267.1266666666666"
        ]
      },
      {
        "metric": {
          "level": "warn"
        },
        "value": [
          1588889221,
          "37.77166666666667"
        ]
      },
      {
        "metric": {
          "level": "info"
        },
        "value": [
          1588889221,
          "37.69"
        ]
      }
    ],
    "stats": {
      ...
    }
  }
}

If your cluster has Multi-Tenancy enabled, set the X-Scope-OrgID header to identify the tenant you want to query. Here is the same example query for the single tenant called Tenant1:

curl -H 'X-Scope-OrgID:Tenant1' \
  -G -s "http://localhost:3100/loki/api/v1/query" \
  --data-urlencode \
  'query=sum(rate({job="varlogs"}[10m])) by (level)' | jq

To query against the three tenants Tenant1, Tenant2, and Tenant3, specify the tenant names separated by the pipe (|) character:

curl -H 'X-Scope-OrgID:Tenant1|Tenant2|Tenant3' \
  -G -s "http://localhost:3100/loki/api/v1/query" \
  --data-urlencode \
  'query=sum(rate({job="varlogs"}[10m])) by (level)' | jq

The same example query for Grafana Enterprise Logs uses Basic Authentication and specifies the tenant names as a user. The tenant names are separated by the pipe (|) character. The password in this example is an access policy token that has been defined in the API_TOKEN environment variable:

curl -u "Tenant1|Tenant2|Tenant3:$API_TOKEN" \
  -G -s "http://localhost:3100/loki/api/v1/query" \
  --data-urlencode \
  'query=sum(rate({job="varlogs"}[10m])) by (level)' | jq

Query Loki over a range of time

GET /loki/api/v1/query_range

/loki/api/v1/query_range is used to do a query over a range of time and accepts the following query parameters in the URL:

query: The LogQL query to perform
limit: The max number of entries to return. It defaults to 100. Only applies to query types which produce a stream(log lines) response.
start: The start time for the query as a nanosecond Unix epoch or another supported format. Defaults to one hour ago.
end: The end time for the query as a nanosecond Unix epoch or another supported format. Defaults to now.
step: Query resolution step width in duration format or float number of seconds. duration refers to Prometheus duration strings of the form [0-9]+[smhdwy]. For example, 5m refers to a duration of 5 minutes. Defaults to a dynamic value based on start and end. Only applies to query types which produce a matrix response.
interval: This parameter is experimental; see the explanation under Step versus interval. Only return entries at (or greater than) the specified interval, can be a duration format or float number of seconds. Only applies to queries which produce a stream response.
direction: Determines the sort order of logs. Supported values are forward or backward. Defaults to backward.

In microservices mode, /loki/api/v1/query_range is exposed by the querier and the frontend.

Step versus interval

Use the step parameter when making metric queries to Loki, or queries which return a matrix response. It is evaluated in exactly the same way Prometheus evaluates step. First the query will be evaluated at start and then evaluated again at start + step and again at start + step + step until end is reached. The result will be a matrix of the query result evaluated at each step.

Use the interval parameter when making log queries to Loki, or queries which return a stream response. It is evaluated by returning a log entry at start, then the next entry will be returned an entry with timestampe >= start + interval, and again at start + interval + interval and so on until end is reached. It does not fill missing entries.

Note about the experimental nature of the interval parameter: This flag may be removed in the future, if so it will likely be in favor of a LogQL expression to perform similar behavior, however that is uncertain at this time. Issue 1779 was created to track the discussion, if you are using interval please go add your use case and thoughts to that issue.

Response:

{
  "status": "success",
  "data": {
    "resultType": "matrix" | "streams",
    "result": [<matrix value>] | [<stream value>]
    "stats" : [<statistics>]
  }
}

Where <matrix value> is:

{
  "metric": {
    <label key-value pairs>
  },
  "values": [
    [
      <number: second unix epoch>,
      <string: value>
    ],
    ...
  ]
}

The items in the values array are sorted by timestamp, and the oldest item is first.

And <stream value> is:

{
  "stream": {
    <label key-value pairs>
  },
  "values": [
    [
      <string: nanosecond unix epoch>,
      <string: log line>
    ],
    ...
  ]
}

The items in the values array are sorted by timestamp. The most recent item is first when using direction=backward. The oldest item is first when using direction=forward.

See statistics for information about the statistics returned by Loki.

Examples

$ curl -G -s  "http://localhost:3100/loki/api/v1/query_range" --data-urlencode 'query=sum(rate({job="varlogs"}[10m])) by (level)' --data-urlencode 'step=300' | jq
{
  "status": "success",
  "data": {
    "resultType": "matrix",
    "result": [
      {
       "metric": {
          "level": "info"
        },
        "values": [
          [
            1588889221,
            "137.95"
          ],
          [
            1588889221,
            "467.115"
          ],
          [
            1588889221,
            "658.8516666666667"
          ]
        ]
      },
      {
        "metric": {
          "level": "warn"
        },
        "values": [
          [
            1588889221,
            "137.27833333333334"
          ],
          [
            1588889221,
            "467.69"
          ],
          [
            1588889221,
            "660.6933333333334"
          ]
        ]
      }
    ],
    "stats": {
      ...
    }
  }
}

$ curl -G -s  "http://localhost:3100/loki/api/v1/query_range" --data-urlencode 'query={job="varlogs"}' | jq
{
  "status": "success",
  "data": {
    "resultType": "streams",
    "result": [
      {
        "stream": {
          "filename": "/var/log/myproject.log",
          "job": "varlogs",
          "level": "info"
        },
        "values": [
          [
            "1569266497240578000",
            "foo"
          ],
          [
            "1569266492548155000",
            "bar"
          ]
        ]
      }
    ],
    "stats": {
      ...
    }
  }
}

List labels within a range of time

GET /loki/api/v1/labels

/loki/api/v1/labels retrieves the list of known labels within a given time span. Loki may use a larger time span than the one specified. It accepts the following query parameters in the URL:

start: The start time for the query as a nanosecond Unix epoch. Defaults to 6 hours ago.
end: The end time for the query as a nanosecond Unix epoch. Defaults to now.

In microservices mode, /loki/api/v1/labels is exposed by the querier.

Response:

{
  "status": "success",
  "data": [
    <label string>,
    ...
  ]
}

Examples

$ curl -G -s  "http://localhost:3100/loki/api/v1/labels" | jq
{
  "status": "success",
  "data": [
    "foo",
    "bar",
    "baz"
  ]
}

List label values within a range of time

GET /loki/api/v1/label/<name>/values

/loki/api/v1/label/<name>/values retrieves the list of known values for a given label within a given time span. Loki may use a larger time span than the one specified. It accepts the following query parameters in the URL:

start: The start time for the query as a nanosecond Unix epoch. Defaults to 6 hours ago.
end: The end time for the query as a nanosecond Unix epoch. Defaults to now.

In microservices mode, /loki/api/v1/label/<name>/values is exposed by the querier.

Response:

{
  "status": "success",
  "data": [
    <label value>,
    ...
  ]
}

Examples

$ curl -G -s  "http://localhost:3100/loki/api/v1/label/foo/values" | jq
{
  "status": "success",
  "data": [
    "cat",
    "dog",
    "axolotl"
  ]
}

Stream log messages

GET /loki/api/v1/tail

/loki/api/v1/tail is a WebSocket endpoint that will stream log messages based on a query. It accepts the following query parameters in the URL:

query: The LogQL query to perform
delay_for: The number of seconds to delay retrieving logs to let slow loggers catch up. Defaults to 0 and cannot be larger than 5.
limit: The max number of entries to return. It defaults to 100.
start: The start time for the query as a nanosecond Unix epoch. Defaults to one hour ago.

In microservices mode, /loki/api/v1/tail is exposed by the querier.

Response (streamed):

{
  "streams": [
    {
      "stream": {
        <label key-value pairs>
      },
      "values": [
        [
          <string: nanosecond unix epoch>,
          <string: log line>
        ]
      ]
    }
  ],
  "dropped_entries": [
    {
      "labels": {
        <label key-value pairs>
      },
      "timestamp": "<nanosecond unix epoch>"
    }
  ]
}

Push log entries to Loki

POST /loki/api/v1/push

/loki/api/v1/push is the endpoint used to send log entries to Loki. The default behavior is for the POST body to be a snappy-compressed protobuf message:

Alternatively, if the Content-Type header is set to application/json, a JSON post body can be sent in the following format:

{
  "streams": [
    {
      "stream": {
        "label": "value"
      },
      "values": [
          [ "<unix epoch in nanoseconds>", "<log line>" ],
          [ "<unix epoch in nanoseconds>", "<log line>" ]
      ]
    }
  ]
}

You can set Content-Encoding: gzip request header and post gzipped JSON.

Loki can be configured to accept out-of-order writes.

In microservices mode, /loki/api/v1/push is exposed by the distributor.

Examples

$ curl -v -H "Content-Type: application/json" -XPOST -s "http://localhost:3100/loki/api/v1/push" --data-raw \
  '{"streams": [{ "stream": { "foo": "bar2" }, "values": [ [ "1570818238000000000", "fizzbuzz" ] ] }]}'

Identify ready Loki instance

GET /ready

/ready returns HTTP 200 when the Loki instance is ready to accept traffic. If running Loki on Kubernetes, /ready can be used as a readiness probe.

In microservices mode, the /ready endpoint is exposed by all components.

Flush in-memory chunks to backing store

POST /flush

/flush triggers a flush of all in-memory chunks held by the ingesters to the backing store. Mainly used for local testing.

In microservices mode, the /flush endpoint is exposed by the ingester.

Flush in-memory chunks and shut down

POST /ingester/shutdown

/ingester/shutdown triggers a shutdown of the ingester and notably will always flush any in memory chunks it holds. This is helpful for scaling down WAL-enabled ingesters where we want to ensure old WAL directories are not orphaned, but instead flushed to our chunk backend.

It accepts three URL query parameters flush, delete_ring_tokens, and terminate.

URL query parameters:

flush=<bool>: Flag to control whether to flush any in-memory chunks the ingester holds. Defaults to true.
delete_ring_tokens=<bool>: Flag to control whether to delete the file that contains the ingester ring tokens of the instance if the -ingester.token-file-path is specified.
terminate=<bool>: Flag to control whether to terminate the Loki process after service shutdown. Defaults to true.

This handler, in contrast to the deprecated /ingester/flush_shutdown handler, terminates the Loki process by default. This behaviour can be changed by setting the terminate query parameter to false.

In microservices mode, the /ingester/shutdown endpoint is exposed by the ingester.

Display distributor consistent hash ring status

GET /distributor/ring

Displays a web page with the distributor hash ring status, including the state, healthy and last heartbeat time of each distributor.

Return exposed Prometheus metrics

GET /metrics

/metrics returns exposed Prometheus metrics. See Observing Loki for a list of exported metrics.

In microservices mode, the /metrics endpoint is exposed by all components.

List current configuration

GET /config

/config exposes the current configuration. The optional mode query parameter can be used to modify the output. If it has the value diff only the differences between the default configuration and the current are returned. A value of defaults returns the default configuration.

In microservices mode, the /config endpoint is exposed by all components.

List running services

GET /services

/services returns a list of all running services and their current states.

Services can have the following states:

New: Service is new, not running yet (initial state)
Starting: Service is starting; if starting succeeds, service enters Running state
Running: Service is fully running now; when service stops running, it enters Stopping state
Stopping: Service is shutting down
Terminated: Service has stopped successfully (terminal state)
Failed: Service has failed in Starting, Running or Stopping state (terminal state)

List build information

GET /loki/api/v1/status/buildinfo

/loki/api/v1/status/buildinfo exposes the build information in a JSON object. The fields are version, revision, branch, buildDate, buildUser, and goVersion.

List series

The Series API is available under the following:

GET /loki/api/v1/series
POST /loki/api/v1/series
GET /api/prom/series
POST /api/prom/series

This endpoint returns the list of time series that match a certain label set.

URL query parameters:

match[]=<series_selector>: Repeated log stream selector argument that selects the streams to return. At least one match[] argument must be provided.
start=<nanosecond Unix epoch>: Start timestamp.
end=<nanosecond Unix epoch>: End timestamp.

You can URL-encode these parameters directly in the request body by using the POST method and Content-Type: application/x-www-form-urlencoded header. This is useful when specifying a large or dynamic number of stream selectors that may breach server-side URL character limits.

In microservices mode, these endpoints are exposed by the querier.

Examples

$ curl -s "http://localhost:3100/loki/api/v1/series" --data-urlencode 'match[]={container_name=~"prometheus.*", component="server"}' --data-urlencode 'match[]={app="loki"}' | jq '.'
{
  "status": "success",
  "data": [
    {
      "container_name": "loki",
      "app": "loki",
      "stream": "stderr",
      "filename": "/var/log/pods/default_loki-stack-0_50835643-1df0-11ea-ba79-025000000001/loki/0.log",
      "name": "loki",
      "job": "default/loki",
      "controller_revision_hash": "loki-stack-757479754d",
      "statefulset_kubernetes_io_pod_name": "loki-stack-0",
      "release": "loki-stack",
      "namespace": "default",
      "instance": "loki-stack-0"
    },
    {
      "chart": "prometheus-9.3.3",
      "container_name": "prometheus-server-configmap-reload",
      "filename": "/var/log/pods/default_loki-stack-prometheus-server-696cc9ddff-87lmq_507b1db4-1df0-11ea-ba79-025000000001/prometheus-server-configmap-reload/0.log",
      "instance": "loki-stack-prometheus-server-696cc9ddff-87lmq",
      "pod_template_hash": "696cc9ddff",
      "app": "prometheus",
      "component": "server",
      "heritage": "Tiller",
      "job": "default/prometheus",
      "namespace": "default",
      "release": "loki-stack",
      "stream": "stderr"
    },
    {
      "app": "prometheus",
      "component": "server",
      "filename": "/var/log/pods/default_loki-stack-prometheus-server-696cc9ddff-87lmq_507b1db4-1df0-11ea-ba79-025000000001/prometheus-server/0.log",
      "release": "loki-stack",
      "namespace": "default",
      "pod_template_hash": "696cc9ddff",
      "stream": "stderr",
      "chart": "prometheus-9.3.3",
      "container_name": "prometheus-server",
      "heritage": "Tiller",
      "instance": "loki-stack-prometheus-server-696cc9ddff-87lmq",
      "job": "default/prometheus"
    }
  ]
}

Index Stats

The /loki/api/v1/index/stats endpoint can be used to query the index for the number of streams, chunks, entries, and bytes that a query resolves to.

URL query parameters:

query: The LogQL matchers to check (i.e. {job="foo", env!="dev"})
start=<nanosecond Unix epoch>: Start timestamp.
end=<nanosecond Unix epoch>: End timestamp.

Response:

{
  "streams": 100,
  "chunks": 1000,
  "entries": 5000,
  "bytes": 100000,
}

It is an approximation with the following caveats:

It does not include data from the ingesters
It is a probabilistic technique
streams/chunks which span multiple period configurations may be counted twice.

These make it generally more helpful for larger queries. It can be used for better understanding the throughput requirements and data topology for a list of matchers over a period of time.

Statistics

Query endpoints such as /api/prom/query, /loki/api/v1/query and /loki/api/v1/query_range return a set of statistics about the query execution. Those statistics allow users to understand the amount of data processed and at which speed.

The example belows show all possible statistics returned with their respective description.

{
  "status": "success",
  "data": {
    "resultType": "streams",
    "result": [],
    "stats": {
     "ingester" : {
        "compressedBytes": 0, // Total bytes of compressed chunks (blocks) processed by ingesters
        "decompressedBytes": 0, // Total bytes decompressed and processed by ingesters
        "decompressedLines": 0, // Total lines decompressed and processed by ingesters
        "headChunkBytes": 0, // Total bytes read from ingesters head chunks
        "headChunkLines": 0, // Total lines read from ingesters head chunks
        "totalBatches": 0, // Total batches sent by ingesters
        "totalChunksMatched": 0, // Total chunks matched by ingesters
        "totalDuplicates": 0, // Total of duplicates found by ingesters
        "totalLinesSent": 0, // Total lines sent by ingesters
        "totalReached": 0 // Amount of ingesters reached.
      },
      "store": {
        "compressedBytes": 0, // Total bytes of compressed chunks (blocks) processed by the store
        "decompressedBytes": 0,  // Total bytes decompressed and processed by the store
        "decompressedLines": 0, // Total lines decompressed and processed by the store
        "chunksDownloadTime": 0, // Total time spent downloading chunks in seconds (float)
        "totalChunksRef": 0, // Total chunks found in the index for the current query
        "totalChunksDownloaded": 0, // Total of chunks downloaded
        "totalDuplicates": 0 // Total of duplicates removed from replication
      },
      "summary": {
        "bytesProcessedPerSecond": 0, // Total of bytes processed per second
        "execTime": 0, // Total execution time in seconds (float)
        "linesProcessedPerSecond": 0, // Total lines processed per second
        "queueTime": 0, // Total queue time in seconds (float)
        "totalBytesProcessed":0, // Total amount of bytes processed overall for this request
        "totalLinesProcessed":0 // Total amount of lines processed overall for this request
      }
    }
  }
}

Ruler

The ruler API endpoints require to configure a backend object storage to store the recording rules and alerts. The ruler API uses the concept of a “namespace” when creating rule groups. This is a stand-in for the name of the rule file in Prometheus. Rule groups must be named uniquely within a namespace.

Ruler ring status

GET /ruler/ring

Displays a web page with the ruler hash ring status, including the state, healthy and last heartbeat time of each ruler.

List rule groups

GET /loki/api/v1/rules

List all rules configured for the authenticated tenant. This endpoint returns a YAML dictionary with all the rule groups for each namespace and 200 status code on success.

Example response

---
<namespace1>:
- name: <string>
  interval: <duration;optional>
  rules:
  - alert: <string>
      expr: <string>
      for: <duration>
      annotations:
      <annotation_name>: <string>
      labels:
      <label_name>: <string>
- name: <string>
  interval: <duration;optional>
  rules:
  - alert: <string>
      expr: <string>
      for: <duration>
      annotations:
      <annotation_name>: <string>
      labels:
      <label_name>: <string>
<namespace2>:
- name: <string>
  interval: <duration;optional>
  rules:
  - alert: <string>
      expr: <string>
      for: <duration>
      annotations:
      <annotation_name>: <string>
      labels:
      <label_name>: <string>

Get rule groups by namespace

GET /loki/api/v1/rules/{namespace}

Returns the rule groups defined for a given namespace.

Example response

name: <string>
interval: <duration;optional>
rules:
  - alert: <string>
    expr: <string>
    for: <duration>
    annotations:
      <annotation_name>: <string>
    labels:
      <label_name>: <string>

Get rule group

GET /loki/api/v1/rules/{namespace}/{groupName}

Returns the rule group matching the request namespace and group name.

Set rule group

POST /loki/api/v1/rules/{namespace}

Creates or updates a rule group. This endpoint expects a request with Content-Type: application/yaml header and the rules YAML definition in the request body, and returns 202 on success.

Example request

Request headers:

Content-Type: application/yaml

Request body:

name: <string>
interval: <duration;optional>
rules:
  - alert: <string>
    expr: <string>
    for: <duration>
    annotations:
      <annotation_name>: <string>
    labels:
      <label_name>: <string>

Delete rule group

DELETE /loki/api/v1/rules/{namespace}/{groupName}

Deletes a rule group by namespace and group name. This endpoints returns 202 on success.

Delete namespace

DELETE /loki/api/v1/rules/{namespace}

Deletes all the rule groups in a namespace (including the namespace itself). This endpoint returns 202 on success.

List rules

GET /prometheus/api/v1/rules

Prometheus-compatible rules endpoint to list alerting and recording rules that are currently loaded.

For more information, refer to the Prometheus rules documentation.

List alerts

GET /prometheus/api/v1/alerts

Prometheus-compatible rules endpoint to list all active alerts.

For more information, refer to the Prometheus alerts documentation.

Compactor

Compactor ring status

GET /compactor/ring

Displays a web page with the compactor hash ring status, including the state, health, and last heartbeat time of each compactor.

Request log deletion

POST /loki/api/v1/delete
PUT /loki/api/v1/delete

Create a new delete request for the authenticated tenant. The log entry deletion documentation has configuration details.

Log entry deletion is supported only when the BoltDB Shipper is configured for the index store.

Query parameters:

query=<series_selector>: query argument that identifies the streams from which to delete with optional line filters.
start=<rfc3339 | unix_timestamp>: A timestamp that identifies the start of the time window within which entries will be deleted. If not specified, defaults to 0, the Unix Epoch time.
end=<rfc3339 | unix_timestamp>: A timestamp that identifies the end of the time window within which entries will be deleted. If not specified, defaults to the current time.

A 204 response indicates success.

The query parameter can also include filter operations. For example query={foo="bar"} |= "other" will filter out lines that contain the string “other” for the streams matching the stream selector {foo="bar"}.

Examples

URL encode the query parameter. This sample form of a cURL command URL encodes query={foo="bar"}:

curl -g -X POST \
  'http://127.0.0.1:3100/loki/api/v1/delete?query={foo="bar"}&start=1591616227&end=1591619692' \
  -H 'X-Scope-OrgID: 1'

The same example deletion request for Grafana Enterprise Logs uses Basic Authentication and specifies the tenant name as a user; Tenant1 is the tenant name in this example. The password in this example is an access policy token that has been defined in the API_TOKEN environment variable. The token must be for an access policy with logs:delete scope for the tenant specified in the user field:

curl -u "Tenant1:$API_TOKEN" \
  -g -X POST \
  'http://127.0.0.1:3100/loki/api/v1/delete?query={foo="bar"}&start=1591616227&end=1591619692'

List log deletion requests

GET /loki/api/v1/delete

List the existing delete requests for the authenticated tenant. The log entry deletion documentation has configuration details.

Log entry deletion is supported only when the BoltDB Shipper is configured for the index store.

List the existing delete requests using the following API:

GET /loki/api/v1/delete

This endpoint returns both processed and unprocessed deletion requests. It does not list canceled requests, as those requests will have been removed from storage.

Examples

Example cURL command:

curl -X GET \
  <compactor_addr>/loki/api/v1/delete \
  -H 'X-Scope-OrgID: <orgid>'

curl -u "Tenant1:$API_TOKEN" \
  -X GET \
  <compactor_addr>/loki/api/v1/delete

Request cancellation of a delete request

DELETE /loki/api/v1/delete

Remove a delete request for the authenticated tenant. The log entry deletion documentation has configuration details.

Loki allows cancellation of delete requests until the requests are picked up for processing. It is controlled by the delete_request_cancel_period YAML configuration or the equivalent command line option when invoking Loki.

Log entry deletion is supported only when the BoltDB Shipper is configured for the index store.

Cancel a delete request using this compactor endpoint:

DELETE /loki/api/v1/delete

Query parameters:

request_id=<request_id>: Identifies the delete request to cancel; IDs are found using the delete endpoint.

A 204 response indicates success.

Examples

Example cURL command:

curl -X DELETE \
  '<compactor_addr>/loki/api/v1/delete?request_id=<request_id>' \
  -H 'X-Scope-OrgID: <tenant-id>'

The same example deletion cancellation request for Grafana Enterprise Logs uses Basic Authentication and specifies the tenant name as a user; Tenant1 is the tenant name in this example. The password in this example is an access policy token that has been defined in the API_TOKEN environment variable. The token must be for an access policy with logs:delete scope for the tenant specified in the user field.

curl -u "Tenant1:$API_TOKEN" \
  -X DELETE \
  '<compactor_addr>/loki/api/v1/delete?request_id=<request_id>'

Deprecated endpoints

`GET /api/prom/tail`

DEPRECATED: /api/prom/tail is deprecated. Use /loki/api/v1/tail instead.

/api/prom/tail is a WebSocket endpoint that will stream log messages based on a query. It accepts the following query parameters in the URL:

query: The LogQL query to perform
delay_for: The number of seconds to delay retrieving logs to let slow loggers catch up. Defaults to 0 and cannot be larger than 5.
limit: The max number of entries to return
start: The start time for the query as a nanosecond Unix epoch. Defaults to one hour ago.

In microservices mode, /api/prom/tail is exposed by the querier.

Response (streamed):

{
  "streams": [
    {
      "labels": "<LogQL label key-value pairs>",
      "entries": [
        {
          "ts": "<RFC3339Nano timestamp>",
          "line": "<log line>"
        }
      ]
    }
  ],
  "dropped_entries": [
    {
      "Timestamp": "<RFC3339Nano timestamp>",
      "Labels": "<LogQL label key-value pairs>"
    }
  ]
}

dropped_entries will be populated when the tailer could not keep up with the amount of traffic in Loki. When present, it indicates that the entries received in the streams is not the full amount of logs that are present in Loki. Note that the keys in dropped_entries will be sent as uppercase Timestamp and Labels instead of labels and ts like in the entries for the stream.

As the response is streamed, the object defined by the response format above will be sent over the WebSocket multiple times.

`GET /api/prom/query`

WARNING: /api/prom/query is DEPRECATED; use /loki/api/v1/query_range instead.

/api/prom/query supports doing general queries. The URL query parameters support the following values:

query: The LogQL query to perform
limit: The max number of entries to return
start: The start time for the query as a nanosecond Unix epoch. Defaults to one hour ago.
end: The end time for the query as a nanosecond Unix epoch. Defaults to now.
direction: Determines the sort order of logs. Supported values are forward or backward. Defaults to backward.
regexp: a regex to filter the returned results

In microservices mode, /api/prom/query is exposed by the querier and the frontend.

Note that the larger the time span between start and end will cause additional load on Loki and the index store, resulting in slower queries.

Response:

{
  "streams": [
    {
      "labels": "<LogQL label key-value pairs>",
      "entries": [
        {
          "ts": "<RFC3339Nano string>",
          "line": "<log line>"
        },
        ...
      ],
    },
    ...
  ],
  "stats": [<statistics>]
}

See statistics for information about the statistics returned by Loki.

Examples

$ curl -G -s "http://localhost:3100/api/prom/query" --data-urlencode 'query={foo="bar"}' | jq
{
  "streams": [
    {
      "labels": "{filename=\"/var/log/myproject.log\", job=\"varlogs\", level=\"info\"}",
      "entries": [
        {
          "ts": "2019-06-06T19:25:41.972739Z",
          "line": "foo"
        },
        {
          "ts": "2019-06-06T19:25:41.972722Z",
          "line": "bar"
        }
      ]
    }
  ],
  "stats": {
    ...
  }
}

`GET /api/prom/label/<name>/values`

WARNING: /api/prom/label/<name>/values is DEPRECATED; use /loki/api/v1/label/<name>/values

/api/prom/label/<name>/values retrieves the list of known values for a given label within a given time span. It accepts the following query parameters in the URL:

start: The start time for the query as a nanosecond Unix epoch. Defaults to 6 hours ago.
end: The end time for the query as a nanosecond Unix epoch. Defaults to now.

In microservices mode, /api/prom/label/<name>/values is exposed by the querier.

Response:

{
  "values": [
    <label value>,
    ...
  ]
}

Examples

$ curl -G -s  "http://localhost:3100/api/prom/label/foo/values" | jq
{
  "values": [
    "cat",
    "dog",
    "axolotl"
  ]
}

`GET /api/prom/label`

WARNING: /api/prom/label is DEPRECATED; use /loki/api/v1/label

/api/prom/label retrieves the list of known labels within a given time span. It accepts the following query parameters in the URL:

start: The start time for the query as a nanosecond Unix epoch. Defaults to 6 hours ago.
end: The end time for the query as a nanosecond Unix epoch. Defaults to now.

In microservices mode, /api/prom/label is exposed by the querier.

Response:

{
  "values": [
    <label string>,
    ...
  ]
}

Examples

$ curl -G -s  "http://localhost:3100/api/prom/label" | jq
{
  "values": [
    "foo",
    "bar",
    "baz"
  ]
}

`POST /api/prom/push`

WARNING: /api/prom/push is DEPRECATED; use /loki/api/v1/push instead.

/api/prom/push is the endpoint used to send log entries to Loki. The default behavior is for the POST body to be a snappy-compressed protobuf message:

Alternatively, if the Content-Type header is set to application/json, a JSON post body can be sent in the following format:

{
  "streams": [
    {
      "labels": "<LogQL label key-value pairs>",
      "entries": [
        {
          "ts": "<RFC3339Nano string>",
          "line": "<log line>"
        }
      ]
    }
  ]
}

Loki can be configured to accept out-of-order writes.

In microservices mode, /api/prom/push is exposed by the distributor.

Examples

$ curl -H "Content-Type: application/json" -XPOST -s "https://localhost:3100/api/prom/push" --data-raw \
  '{"streams": [{ "labels": "{foo=\"bar\"}", "entries": [{ "ts": "2018-12-18T08:28:06.801064-04:00", "line": "fizzbuzz" }] }]}'

`POST /ingester/flush_shutdown`

WARNING: /ingester/flush_shutdown is DEPRECATED; use /ingester/shutdown?flush=true instead.

/ingester/flush_shutdown triggers a shutdown of the ingester and notably will always flush any in memory chunks it holds. This is helpful for scaling down WAL-enabled ingesters where we want to ensure old WAL directories are not orphaned, but instead flushed to our chunk backend.

In microservices mode, the /ingester/flush_shutdown endpoint is exposed by the ingester.

Storage

Mon, 14 Apr 2025 21:05:47 +0000

Storage

Unlike other logging systems, Grafana Loki is built around the idea of only indexing metadata about your logs: labels (just like Prometheus labels). Log data itself is then compressed and stored in chunks in object stores such as S3 or GCS, or even locally on the filesystem. A small index and highly compressed chunks simplifies the operation and significantly lowers the cost of Loki.

Until Loki 2.0, index data was stored in a separate index.

Loki 2.0 brings an index mechanism named ‘boltdb-shipper’ and is what we now call Single Store Loki. This index type only requires one store, the object store, for both the index and chunks. More detailed information can be found on the operations page.

Some more storage details can also be found in the operations section.

Implementations - Chunks

Cassandra

Cassandra is a popular database and one of Loki’s possible chunk stores and is production safe.

GCS

GCS is a hosted object store offered by Google. It is a good candidate for a managed object store, especially when you’re already running on GCP, and is production safe.

File System

The file system is the simplest backend for chunks, although it’s also susceptible to data loss as it’s unreplicated. This is common for single binary deployments though, as well as for those trying out loki or doing local development on the project. It is similar in concept to many Prometheus deployments where a single Prometheus is responsible for monitoring a fleet.

S3

S3 is AWS’s hosted object store. It is a good candidate for a managed object store, especially when you’re already running on AWS, and is production safe.

Notable Mentions

You may use any substitutable services, such as those that implement the S3 API like MinIO.

Implementations - Index

Single-Store

Also known as “boltdb-shipper” during development (and is still the schema store name). The single store configurations for Loki utilize the chunk store for both chunks and the index, requiring just one store to run Loki.

As of 2.0, this is the recommended index storage type, performance is comparable to a dedicated index type while providing a much less expensive and less complicated deployment.

Cassandra

Cassandra can also be utilized for the index store and aside from the boltdb-shipper, it’s the only non-cloud offering that can be used for the index that’s horizontally scalable and has configurable replication. It’s a good candidate when you already run Cassandra, are running on-prem, or do not wish to use a managed cloud offering.

BigTable

Bigtable is a cloud database offered by Google. It is a good candidate for a managed index store if you’re already using it (due to it’s heavy fixed costs) or wish to run in GCP.

DynamoDB

DynamoDB is a cloud database offered by AWS. It is a good candidate for a managed index store, especially if you’re already running in AWS.

Rate Limiting

DynamoDB is susceptible to rate limiting, particularly due to overconsuming what is called provisioned capacity. This can be controlled via the provisioning configs in the table manager.

BoltDB

BoltDB is an embedded database on disk. It is not replicated and thus cannot be used for high availability or clustered Loki deployments, but is commonly paired with a filesystem chunk store for proof of concept deployments, trying out Loki, and development. The boltdb-shipper aims to support clustered deployments using boltdb as an index.

Azure Storage Account

An Azure storage account contains all of your Azure Storage data objects: blobs, file shares, queues, tables, and disks.

Schema Configs

Loki aims to be backwards compatible and over the course of its development has had many internal changes that facilitate better and more efficient storage/querying. Loki allows incrementally upgrading to these new storage schemas and can query across them transparently. This makes upgrading a breeze. For instance, this is what it looks like when migrating from the v10 -> v11 schemas starting 2020-07-01:

schema_config:
  configs:
    - from: 2019-07-01
      store: boltdb
      object_store: filesystem
      schema: v10
      index:
        prefix: index_
        period: 168h
    - from: 2020-07-01
      store: boltdb
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 168h

For all data ingested before 2020-07-01, Loki used the v10 schema and then switched after that point to the more effective v11. This dramatically simplifies upgrading, ensuring it’s simple to take advantages of new storage optimizations. These configs should be immutable for as long as you care about retention.

Table Manager

One of the subcomponents in Loki is the table-manager. It is responsible for pre-creating and expiring index tables. This helps partition the writes and reads in loki across a set of distinct indices in order to prevent unbounded growth.

table_manager:
  # The retention period must be a multiple of the index / chunks
  # table "period" (see period_config).
  retention_deletes_enabled: true
  # This is 15 weeks retention, based on the 168h (1week) period durations used in the rest of the examples.
  retention_period: 2520h

For more information, see the table manager documentation.

Provisioning

In the case of AWS DynamoDB, you’ll likely want to tune the provisioned throughput for your tables as well. This is to prevent your tables being rate limited on one hand and assuming unnecessary cost on the other. By default Loki uses a provisioned capacity strategy for DynamoDB tables like so:

table_manager:
  index_tables_provisioning:
    # Read/write throughput requirements for the current table
    # (the table which would handle writes/reads for data timestamped at the current time)
    provisioned_write_throughput: <int> | default = 3000
    provisioned_read_throughput: <int> | default = 300

    # Read/write throughput requirements for non-current tables
    inactive_write_throughput: <int> | default = 1
    inactive_read_throughput: <int> | Default = 300

Note, there are a few other DynamoDB provisioning options including DynamoDB autoscaling and on-demand capacity. See the provisioning configuration documentation for more information.

Upgrading Schemas

When a new schema is released and you want to gain the advantages it provides, you can! Loki can transparently query & merge data from across schema boundaries so there is no disruption of service and upgrading is easy.

First, you’ll want to create a new period_config entry in your schema_config. The important thing to remember here is to set this at some point in the future and then roll out the config file changes to Loki. This allows the table manager to create the required table in advance of writes and ensures that existing data isn’t queried as if it adheres to the new schema.

As an example, let’s say it’s 2020-07-14 and we want to start using the v11 schema on the 20th:

schema_config:
  configs:
    - from: 2019-07-14
      store: boltdb
      object_store: filesystem
      schema: v10
      index:
        prefix: index_
        period: 168h
    - from: 2020-07-20
      store: boltdb
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 168h

It’s that easy; we just created a new entry starting on the 20th.

Retention

With the exception of the filesystem chunk store, Loki will not delete old chunk stores. This is generally handled instead by configuring TTLs (time to live) in the chunk store of your choice (bucket lifecycles in S3/GCS, and TTLs in Cassandra). Neither will Loki currently delete old data when your local disk fills when using the filesystem chunk store – deletion is only determined by retention duration.

We’re interested in adding targeted deletion in future Loki releases (think tenant or stream level granularity) and may include other strategies as well.

For more information, see the retention configuration documentation.

Examples

Single machine/local development (boltdb+filesystem)

The repo contains a working example, you may want to checkout a tag of the repo to make sure you get a compatible example.

GCP deployment (GCS Single Store)

storage_config:
  boltdb_shipper:
    active_index_directory: /loki/boltdb-shipper-active
    cache_location: /loki/boltdb-shipper-cache
    cache_ttl: 24h         # Can be increased for faster performance over longer query periods, uses more disk space
    shared_store: gcs
  gcs:
      bucket_name: <bucket>

schema_config:
  configs:
    - from: 2020-07-01
      store: boltdb-shipper
      object_store: gcs
      schema: v11
      index:
        prefix: index_
        period: 24h

AWS deployment (S3 Single Store)

storage_config:
  boltdb_shipper:
    active_index_directory: /loki/boltdb-shipper-active
    cache_location: /loki/boltdb-shipper-cache
    cache_ttl: 24h         # Can be increased for faster performance over longer query periods, uses more disk space
    shared_store: s3
  aws:
    s3: s3://<access_key>:<uri-encoded-secret-access-key>@<region>
    bucketnames: <bucket1,bucket2>

schema_config:
  configs:
    - from: 2020-07-01
      store: boltdb-shipper
      object_store: aws
      schema: v11
      index:
        prefix: index_
        period: 24h

If you don’t wish to hard-code S3 credentials, you can also configure an EC2 instance role by changing the storage_config section:

storage_config:
  aws:
    s3: s3://region
    bucketnames: <bucket1,bucket2>
    dynamodb:
      dynamodb_url: dynamodb://region

On prem deployment (Cassandra+Cassandra)

Keeping this for posterity, but this is likely not a common config. Cassandra should work and could be faster in some situations but is likely much more expensive.

storage_config:
  cassandra:
    addresses: <comma-separated-IPs-or-hostnames>
    keyspace: <keyspace>
    auth: <true|false>
    username: <username> # only applicable when auth=true
    password: <password> # only applicable when auth=true

schema_config:
  configs:
    - from: 2020-07-01
      store: cassandra
      object_store: cassandra
      schema: v11
      index:
        prefix: index_
        period: 168h
      chunks:
        prefix: chunk_
        period: 168h

On prem deployment (MinIO Single Store)

We configure MinIO by using the AWS config because MinIO implements the S3 API:

storage_config:
  aws:
    # Note: use a fully qualified domain name, like localhost.
    # full example: http://loki:supersecret@localhost.:9000
    s3: http<s>://<username>:<secret>@<fqdn>:<port>
    s3forcepathstyle: true
  boltdb_shipper:
    active_index_directory: /loki/boltdb-shipper-active
    cache_location: /loki/boltdb-shipper-cache
    cache_ttl: 24h         # Can be increased for faster performance over longer query periods, uses more disk space
    shared_store: s3

schema_config:
  configs:
    - from: 2020-07-01
      store: boltdb-shipper
      object_store: aws
      schema: v11
      index:
        prefix: index_
        period: 24h

Azure Storage Account

schema_config:
  configs:
  - from: "2020-12-11"
    index:
      period: 24h
      prefix: index_
    object_store: azure
    schema: v11
    store: boltdb-shipper
storage_config:
  azure:
    # Your Azure storage account name
    account_name: <account-name>
    # For the account-key, see docs: https://docs.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage?tabs=azure-portal
    account_key: <account-key>
    # See https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction#containers
    container_name: <container-name>
    use_managed_identity: <true|false>
    # Providing a user assigned ID will override use_managed_identity
    user_assigned_id: <user-assigned-identity-id>
    request_timeout: 0    
  boltdb_shipper:
    active_index_directory: /data/loki/boltdb-shipper-active
    cache_location: /data/loki/boltdb-shipper-cache
    cache_ttl: 24h
    shared_store: azure
  filesystem:
    directory: /data/loki/chunks

Grafana Loki on Grafana Labs

LogCLI

LogCLI, Grafana Loki’s command-line interface

Installation

Binary (Recommended)

Build LogCLI from source

LogCLI usage

Grafana Cloud example

Batched queries

Configuration

LogCLI command reference

LogCLI query command reference

LogCLI labels command reference

LogCLI series command reference

LogCLI --stdin usage

Fundamentals

Grafana Loki Fundamentals

Configuration

Configuring Grafana Loki

Printing Loki Config At Runtime

Configuration File Reference

Use environment variables in the configuration

Generic placeholders

Supported contents and default values of loki.yaml

server

distributor

querier

query_scheduler

frontend

query_range

ruler

bos_storage_config

azure_storage_config

gcs_storage_config

s3_storage_config

swift_storage_config

hedging

local_storage_config

frontend_worker

ingester_client

ingester

consul_config

etcd_config

memberlist_config

storage_config

chunk_store_config

cache_config

schema_config

period_config

compactor

limits_config

grpc_client_config

index_gateway

table_manager

provision_config

auto_scaling_config

tracing

common

analytics

storage

filesystem

ring

Runtime Configuration file

Accept out-of-order writes

Clients

Grafana Loki clients

Third-party clients

Alerting and Recording Rules

Rules and the Ruler

Alerting Rules

Example

Recording Rules

Example

Remote-Write

Operations

Use cases

Black box monitoring

Event alerting

Alerting on high-cardinality sources

Interacting with the Ruler

LogCLI `--stdin` usage

Supported contents and default values of `loki.yaml`