Important: This documentation is about an older version. It's relevant only to the release noted, many of the features and functions have been updated or replaced. Please view the current version.
Configuring Grafana Loki
Grafana Loki is configured in a YAML file (usually referred to as loki.yaml
)
which contains information on the Loki server and its individual components,
depending on which mode Loki is launched in.
Configuration examples can be found in the Configuration Examples document.
Printing Loki Config At Runtime
If you pass Loki the flag -print-config-stderr
or -log-config-reverse-order
, (or -print-config-stderr=true
)
Loki will dump the entire config object it has created from the built in defaults combined first with
overrides from config file, and second by overrides from flags.
The result is the value for every config object in the Loki config struct, which is very large…
Many values will not be relevant to your install such as storage configs which you are not using and which you did not define, this is expected as every option has a default value if it is being used or not.
This config is what Loki will use to run, it can be invaluable for debugging issues related to configuration and is especially useful in making sure your config files and flags are being read and loaded properly.
-print-config-stderr
is nice when running Loki directly e.g. ./loki
as you can get a quick output of the entire Loki config.
-log-config-reverse-order
is the flag we run Loki with in all our environments, the config entries are reversed so
that the order of configs reads correctly top to bottom when viewed in Grafana’s Explore.
Configuration File Reference
To specify which configuration file to load, pass the -config.file
flag at the
command line. The value can be a list of comma separated paths, then the first
file that exists will be used.
If no -config.file
argument is specified, Loki will look up the config.yaml
in the
current working directory and the config/
sub-directory and try to use that.
The file is written in YAML format, defined by the scheme below. Brackets indicate that a parameter is optional. For non-list parameters the value is set to the specified default.
Use environment variables in the configuration
Note: This feature is only available in Loki 2.1+.
You can use environment variable references in the configuration file to set values that need to be configurable during deployment.
To do this, pass -config.expand-env=true
and use:
${VAR}
Where VAR is the name of the environment variable.
Each variable reference is replaced at startup by the value of the environment variable. The replacement is case-sensitive and occurs before the YAML file is parsed. References to undefined variables are replaced by empty strings unless you specify a default value or custom error text.
To specify a default value, use:
${VAR:-default_value}
Where default_value is the value to use if the environment variable is undefined.
Pass the -config.expand-env
flag at the command line to enable this way of setting configs.
Generic placeholders
<boolean>
: a boolean that can take the valuestrue
orfalse
<int>
: any integer matching the regular expression[1-9]+[0-9]*
<duration>
: a duration matching the regular expression[0-9]+(ns|us|µs|ms|[smh])
<labelname>
: a string matching the regular expression[a-zA-Z_][a-zA-Z0-9_]*
<labelvalue>
: a string of unicode characters<filename>
: a valid path relative to current working directory or an absolute path.<host>
: a valid string consisting of a hostname or IP followed by an optional port number<string>
: a string<secret>
: a string that represents a secret, such as a password
Supported contents and default values of loki.yaml
server
The server
block
configures the HTTP and gRPC server communication of the launched service(s).
distributor
The distributor
block configures the distributor component.
# Configures the distributors ring, used when the "global" ingestion rate
# strategy is enabled.
ring:
kvstore:
# The backend storage to use for the ring. Supported values are
# consul, etcd, inmemory, memberlist
# CLI flag: -distributor.ring.store
store: <string>
# The prefix for the keys in the store. Should end with a /.
# CLI flag: -distributor.ring.prefix
[prefix: <string> | default = "collectors/"]
# Configuration for a Consul client. Only applies if store is "consul"
# The CLI flags prefix for this block config is: distributor.ring
[consul: <consul_config>]
# Configuration for an ETCD v3 client. Only applies if store is "etcd"
# The CLI flags prefix for this block config is: distributor.ring
[etcd: <etcd_config>]
# The heartbeat timeout after which ingesters are skipped for
# reading and writing.
# CLI flag: -distributor.ring.heartbeat-timeout
[heartbeat_timeout: <duration> | default = 1m]
querier
The querier
block configures the Loki Querier.
query_scheduler
The query_scheduler
block configures the Loki query scheduler.
# Maximum number of outstanding requests per tenant per query-scheduler.
# In-flight requests above this limit will fail with HTTP response status code
# 429.
# CLI flag: -query-scheduler.max-outstanding-requests-per-tenant
[max_outstanding_requests_per_tenant: <int> | default = 100]
# If a querier disconnects without sending notification about graceful shutdown,
# the query-scheduler will keep the querier in the tenant's shard until the forget delay has passed.
# This feature is useful to reduce the blast radius when shuffle-sharding is enabled.
# CLI flag: -query-scheduler.querier-forget-delay
[querier_forget_delay: <duration> | default = 0]
# This configures the gRPC client used to report errors back to the
# query-frontend.
[grpc_client_config: <grpc_client_config>]
# Set to true to have the query schedulers create and place themselves in a ring.
# If no frontend_address or scheduler_address are present
# anywhere else in the configuration, Loki will toggle this value to true.
[use_scheduler_ring: <boolean> | default = false]
# The hash ring configuration. This option is required only if use_scheduler_ring is true
# The CLI flags prefix for this block config is scheduler.ring
[scheduler_ring: <ring>]
frontend
The frontend
block configures the Loki query-frontend.
query_range
The query_range
block configures query splitting and caching in the Loki query-frontend.
# Deprecated: Split queries by day and execute in parallel.
# Use -querier.split-queries-by-interval instead.
# CLI flag: -querier.split-queries-by-day
[split_queries_by_day: <boolean> | default = false]
# Mutate incoming queries to align their start and end with their step.
# CLI flag: -querier.align-querier-with-step
[align_queries_with_step: <boolean> | default = false]
results_cache:
# The CLI flags prefix for this block config is: frontend
cache: <cache_config>
# Cache query results.
# CLI flag: -querier.cache-results
[cache_results: <boolean> | default = false]
# Maximum number of retries for a single request; beyond this, the downstream
# error is returned.
# CLI flag: -querier.max-retries-per-request
[max_retries: <int> | default = 5]
# Perform query parallelisations based on storage sharding configuration and
# query ASTs. This feature is supported only by the chunks storage engine.
# CLI flag: -querier.parallelise-shardable-queries
[parallelise_shardable_queries: <boolean> | default = true]
ruler
The ruler
block configures the Loki ruler.
bos_storage_config
The bos_storage_config
block configures Baidu Object Storage (BOS) as general storage for data generated by Loki.
# Name of BOS bucket.
# CLI flag: <prefix>.baidubce.bucket-name
[ bucket_name: <string> | default = "" ]
# BOS endpoint to connect to.
# CLI flag: <prefix>.baidubce.endpoint
[ endpoint: <string> | default = "bj.bcebos.com" ]
# Baidu Cloud Engine (BCE) Access Key ID
# CLI flag: <prefix>.baidubce.access-key-id
[ access_key_id: <string> | default = "" ]
# BCE Secret Access Key
# CLI flag: <prefix>.baidubce.secret-access-key
[ secret_access_key: <string> | default = "" ]
azure_storage_config
The azure_storage_config
configures Azure as a general storage for different data generated by Loki.
gcs_storage_config
The gcs_storage_config
configures GCS as a general storage for different data generated by Loki.
# Name of GCS bucket to put chunks in.
# CLI flag: -<prefix>.gcs.bucketname
[bucket_name: <string> | default = ""]
# The size of the buffer that GCS client for each PUT request. 0 to disable
# buffering.
# CLI flag: -<prefix>.gcs.chunk-buffer-size
[chunk_buffer_size: <int> | default = 0]
# The duration after which the requests to GCS should be timed out.
# CLI flag: -<prefix>.gcs.request-timeout
[request_timeout: <duration> | default = 0s]
# Enable HTTP/2 when connecting to GCS. This configuration only applies to GET operations.
# CLI flag: -<prefix>.gcs.enable-http2
[enable_http2: <boolean> | default = true]
s3_storage_config
The s3_storage_config
configures S3 as a general storage for different data generated by Loki.
swift_storage_config
The swift_storage_config
configures Swift as a general storage for different data generated by Loki.
hedging
The hedging
block configures how to hedge storage requests.
The hedging implementation sends a second storage request once a first request has been outstanding for more than a configured expected latency for this class of requests. Calculate your latency to be the 99th percentile of object storage response times.
# An optional duration that sets the quantity of time after a first storage request
# is sent and before a second request is sent, when no response is received for the first
# storage request. The recommended duration is the measured 99th percentile of object
# storage response times, to reduce long tail latency. This option is most impactful
# when used with queriers, and has minimal to no impact on other components.
# The default value of 0 disables the hedging of storage requests.
# Example: "at: 500ms"
[at: <duration> | default = 0]
# An optional maximum quantity of hedged requests to be issued for a given request.
[up_to: <int> | default = 2]
# Caps the rate of hedged requests by optionally defining the maximum quantity of
# hedged requests issued per second.
[max_per_second: <int> | default = 5]
local_storage_config
The local_storage_config
configures a (local) file system as a general storage for different data generated by Loki.
# Filesystem directory to be used as storage.
# CLI flag: -<prefix>.local.directory
[directory: <filename> | default = ""]
frontend_worker
The frontend_worker
configures the worker - running within the Loki querier - picking up and executing queries enqueued by the query-frontend.
# Address of query frontend service, in host:port format.
# CLI flag: -querier.frontend-address
[frontend_address: <string> | default = ""]
# Number of simultaneous queries to process.
# CLI flag: -querier.worker-parallelism
[parallelism: <int> | default = 10]
# Force worker concurrency to match the -querier.max-concurrent option. Overrides querier.worker-parallelism.
# CLI flag: -querier.worker-match-max-concurrent
[match_max_concurrent: <boolean> | default = true]
# How often to query the frontend_address DNS to resolve frontend addresses.
# Also used to determine how often to poll the scheduler-ring for addresses if configured.
# CLI flag: -querier.dns-lookup-period
[dns_lookup_duration: <duration> | default = 3s]
# The CLI flags prefix for this block config is: querier.frontend-client
[grpc_client_config: <grpc_client_config>]
# DNS hostname used for finding query-schedulers.
# CLI flag: -querier.scheduler-address
[scheduler_address: <string> | default = ""]
ingester_client
The ingester_client
block configures how connections to ingesters
operate.
# Configures how connections are pooled
pool_config:
# Whether or not to do health checks.
# CLI flag: -distributor.health-check-ingesters
[health_check_ingesters: <boolean> | default = false]
# How frequently to clean up clients for servers that have gone away after
# a health check.
# CLI flag: -distributor.client-cleanup-period
[client_cleanup_period: <duration> | default = 15s]
# How quickly a dead client will be removed after it has been detected
# to disappear. Set this to a value to allow time for a secondary
# health check to recover the missing client.
# CLI flag: -ingester.client.healthcheck-timeout
[remote_timeout: <duration> | default = 1s]
# The remote request timeout on the client side.
# CLI flag: -ingester.client.timeout
[remote_timeout: <duration> | default = 5s]
# Configures how the gRPC connection to ingesters work as a client
# The CLI flags prefix for this block config is: ingester.client
[grpc_client_config: <grpc_client_config>]
ingester
The ingester
block configures the Loki Ingesters.
consul_config
The consul_config
configures the consul client. The supported CLI flags
# The hostname and port of Consul.
# CLI flag: -<prefix>.consul.hostname
[host: <string> | default = "localhost:8500"]
# The ACL Token used to interact with Consul.
# CLI flag: -<prefix>.consul.acl-token
[acl_token: <string>]
# The HTTP timeout when communicating with Consul
# CLI flag: -<prefix>.consul.client-timeout
[http_client_timeout: <duration> | default = 20s]
# Whether or not consistent reads to Consul are enabled.
# CLI flag: -<prefix>.consul.consistent-reads
[consistent_reads: <boolean> | default = true]
etcd_config
The etcd_config
configures the etcd client. The supported CLI flags
# The etcd endpoints to connect to.
# CLI flag: -<prefix>.etcd.endpoints
[endpoints: <list of string> | default = []]
# The dial timeout for the etcd connection.
# CLI flag: -<prefix>.etcd.dial-timeout
[dial_timeout: <duration> | default = 10s]
# The maximum number of retries to do for failed ops.
# CLI flag: -<prefix>.etcd.max-retries
[max_retries: <int> | default = 10]
memberlist_config
The memberlist_config
block configures the gossip ring to discover and connect
between distributors, ingesters and queriers. The configuration is unique for all
three components to ensure a single shared ring.
When a memberlist_config
with least 1 join_members
is defined, a kvstore
of type memberlist
is
automatically configured for the distributor
, ingester
, and ruler
rings unless otherwise specified in
those components specific configuration sections.
storage_config
The storage_config
block configures one of many possible stores for both the
index and chunks. Which configuration to be picked should be defined in schema_config
block.
chunk_store_config
The chunk_store_config
block configures how chunks will be cached and how long
to wait before saving them to the backing store.
# The cache configuration for storing chunks
# The CLI flags prefix for this block config is: store.chunks-cache
[chunk_cache_config: <cache_config>]
# The cache configuration for deduplicating writes
# The CLI flags prefix for this block config is: store.index-cache-write
[write_dedupe_cache_config: <cache_config>]
# The minimum time between a chunk update and being saved
# to the store.
[min_chunk_age: <duration>]
# Cache index entries older than this period. Default is disabled.
# CLI flag: -store.cache-lookups-older-than
[cache_lookups_older_than: <duration>]
# Limit how long back data can be queried. Default is disabled.
# This should always be set to a value less than or equal to
# what is set in `table_manager.retention_period` .
# CLI flag: -store.max-look-back-period
[max_look_back_period: <duration>]
cache_config
The cache_config
block configures how Loki will cache requests, chunks, and
the index to a backing cache store.
The memcached configuration variable addresses is experimental.
schema_config
The schema_config
block configures schemas from given dates.
# The configuration for chunk index schemas.
configs:
- [<period_config>]
period_config
The period_config
block configures what index schemas should be used
for from specific time periods.
compactor
The compactor
block configures the compactor component. This component periodically
compacts index shards to more performant forms.
limits_config
The limits_config
block configures global and per-tenant limits in Loki.
grpc_client_config
The grpc_client_config
block configures a client connection to a gRPC service.
index_gateway
The index_gateway
block configures the Loki index gateway server, responsible for serving index queries
without the need to constantly interact with the object store.
# Defines in which mode the index gateway server will operate (default to 'simple').
# It supports two modes:
# 'simple': an index gateway server instance is responsible for handling,
# storing and returning requests for all indices for all tenants.
# 'ring': an index gateway server instance is responsible for a subset of tenants instead
# of all tenants.
[mode: <string> | default = simple]
# Defines the ring to be used by the index gateway servers and clients in case the servers
# are configured to run in 'ring' mode. In case this isn't configured, this block supports
# inheriting configuration from the common ring section.
[ring: <ring>]
table_manager
The table_manager
block configures the Loki table-manager.
# Master 'off-switch' for table capacity updates, e.g. when troubleshooting.
# CLI flag: -table-manager.throughput-updates-disabled
[throughput_updates_disabled: <boolean> | default = false]
# Master 'on-switch' for table retention deletions.
# CLI flag: -table-manager.retention-deletes-enabled
[retention_deletes_enabled: <boolean> | default = false]
# How far back tables will be kept before they are deleted. 0s disables
# deletion. The retention period must be a multiple of the index / chunks
# table "period" (see period_config).
# CLI flag: -table-manager.retention-period
[retention_period: <duration> | default = 0s]
# Period with which the table manager will poll for tables.
# CLI flag: -table-manager.poll-interval
[poll_interval: <duration> | default = 2m]
# Duration a table will be created before it is needed.
# CLI flag: -table-manager.periodic-table.grace-period
[creation_grace_period: <duration> | default = 10m]
# Configures management of the index tables for DynamoDB.
# The CLI flags prefix for this block config is: table-manager.index-table
index_tables_provisioning: <provision_config>
# Configures management of the chunk tables for DynamoDB.
# The CLI flags prefix for this block config is: table-manager.chunk-table
chunk_tables_provisioning: <provision_config>
provision_config
The provision_config
block configures provisioning capacity for DynamoDB.
auto_scaling_config
The auto_scaling_config
block configures autoscaling for DynamoDB.
# Whether or not autoscaling should be enabled.
# CLI flag: -<prefix>.scale.enabled
[enabled: <boolean>: default = false]
# AWS AutoScaling role ARN.
# CLI flag: -<prefix>.scale.role-arn
[role_arn: <string>]
# DynamoDB minimum provision capacity.
# CLI flag: -<prefix>.scale.min-capacity
[min_capacity: <int> | default = 3000]
# DynamoDB maximum provision capacity.
# CLI flag: -<prefix>.scale.max-capacity
[max_capacity: <int> | default = 6000]
# DynamoDB minimum seconds between each autoscale up.
# CLI flag: -<prefix>.scale.out-cooldown
[out_cooldown: <int> | default = 1800]
# DynamoDB minimum seconds between each autoscale down.
# CLI flag: -<prefix>.scale.in-cooldown
[in_cooldown: <int> | default = 1800]
# DynamoDB target ratio of consumed capacity to provisioned capacity.
# CLI flag: -<prefix>.scale.target-value
[target: <float> | default = 80]
tracing
The tracing
block configures tracing for Jaeger. Currently limited to disable auto-configuration per environment variables only.
# Whether or not tracing should be enabled.
# CLI flag: -tracing.enabled
[enabled: <boolean>: default = true]
common
The common
block sets common definitions to be shared by different components.
This way, one doesn’t have to replicate configuration in multiple places.
analytics
The analytics
block configures the reporting of Loki analytics to grafana.com.
# By default, Loki will send anonymous, but uniquely-identifiable usage and configuration
# analytics to Grafana Labs. These statistics are sent to https://stats.grafana.org/
#
# Statistics help us better understand how Loki is used, and they show us performance
# levels for most users. This helps us prioritize features and documentation.
# For more information on what's sent, look at
# https://github.com/grafana/loki/blob/main/pkg/usagestats/stats.go
# Refer to the buildReport method to see what goes into a report.
#
# When true, enables usage reporting.
# CLI flag: -reporting.enabled
[reporting_enabled: <boolean>: default = true]
storage
The common storage
block defines a common storage to be reused by different
components as a way to facilitate storage configuration.
If any specific configuration for an object storage client have been provided elsewhere in the configuration file, the specific configuration will supersede the common storage configuration.
# Configures Azure as the common storage.
[azure: <azure_storage_config>]
# Configures GCS as the common storage.
[gcs: <gcs_storage_config>]
# Configures S3 as the common storage.
[s3: <s3_storage_config>]
# Configures Swift as the common storage.
[swift: <swift_storage_config>]
# Configures a (local) file system as the common storage.
[filesystem: <filesystem>]
# Configures Baidu Object Storage (BOS) as the common storage.
[bos: <bos_storage_config>]
# The `hedging_config` configures how to hedge requests for the storage.
[hedging: <hedging_config>]
filesystem
The common filesystem
block configures a local file system as a general
storage for various types of data generated by Loki.
# File system directory to be used for chunks storage.
[chunks_directory: <filename> | default = ""]
# File system directory to be used for rules storage.
[rules_directory: <filename> | default = ""]
ring
The common ring
block defines a ring configuration used by a Loki component.
Runtime Configuration file
Loki has a concept of “runtime config” file, which is simply a file that is reloaded while Loki is running. It is used by some Loki components to allow operator to change some aspects of Loki configuration without restarting it. File is specified by using -runtime-config.file=<filename>
flag and reload period (which defaults to 10 seconds) can be changed by -runtime-config.reload-period=<duration>
flag. Previously this mechanism was only used by limits overrides, and flags were called -limits.per-user-override-config=<filename>
and -limits.per-user-override-period=10s
respectively. These are still used, if -runtime-config.file=<filename>
is not specified.
At the moment, two components use runtime configuration: limits and multi KV store.
Options for runtime configuration reload can also be configured via YAML:
# Configuration file to periodically check and reload.
[file: <string>: default = empty]
# How often to check the file.
[period: <duration>: default 10s]
Example runtime configuration file:
overrides:
tenant1:
ingestion_rate_mb: 10
max_streams_per_user: 100000
max_chunks_per_query: 100000
tenant2:
max_streams_per_user: 1000000
max_chunks_per_query: 1000000
multi_kv_config:
mirror-enabled: false
primary: consul
Accept out-of-order writes
Since the beginning of Loki, log entries had to be written to Loki in order by time. This limitation has been lifted. Out-of-order writes are enabled globally by default, but can be disabled/enabled on a cluster or per-tenant basis.
To disable out-of-order writes for all tenants, place in the
limits_config
section:limits_config: unordered_writes: false
To disable out-of-order writes for specific tenants, configure a runtime configuration file:
runtime_config: overrides.yaml
In the
overrides.yaml
file, addunordered_writes
for each tenant permitted to have out-of-order writes:overrides: "tenantA": unordered_writes: false
How far into the past accepted out-of-order log entries may be
is configurable with max_chunk_age
.
max_chunk_age
defaults to 1 hour.
Loki calculates the earliest time that out-of-order entries may have
and be accepted with
time_of_most_recent_line - (max_chunk_age/2)
Log entries with timestamps that are after this earliest time are accepted. Log entries further back in time return an out-of-order error.
For example, if max_chunk_age
is 2 hours
and the stream {foo="bar"}
has one entry at 8:00
,
Loki will accept data for that stream as far back in time as 7:00
.
If another log line is written at 10:00
,
Loki will accept data for that stream as far back in time as 9:00
.