Grafana Loki configuration parameters on Grafana Labs

Configuration best practices

Thu, 09 Apr 2026 02:28:18 +0000

Configuration best practices

Grafana Loki is under active development, and the Loki team is constantly working to improve performance. But here are some of the most current best practices for configuration that will give you the best experience with Loki.

Configure authentication

Grafana Loki does not come with any included authentication layer. You must run an authenticating reverse proxy in front of your services to prevent unauthorized access to Loki (for example, nginx). Refer to Manage authentication for a list of open-source reverse proxies you can use.

Configure caching

Loki can cache data at many levels, which can drastically improve performance. Details of this will be in a future post.

Time ordering of logs

Loki accepts out-of-order writes by default. This section identifies best practices when Loki is not configured to accept out-of-order writes.

One issue many people have with Loki is their client receiving errors for out of order log entries. This happens because of this hard and fast rule within Loki:

For any single log stream, logs must always be sent in increasing time order. If a log is received with a timestamp older than the most recent log received for that stream, that log will be dropped.

There are a few things to dissect from that statement. The first is this restriction is per stream. Let’s look at an example:

{job="syslog"} 00:00:00 i'm a syslog!
{job="syslog"} 00:00:01 i'm a syslog!

If Loki received these two lines which are for the same stream, everything would be fine. But what about this case:

{job="syslog"} 00:00:00 i'm a syslog!
{job="syslog"} 00:00:02 i'm a syslog!
{job="syslog"} 00:00:01 i'm a syslog!  <- Rejected out of order!

What can you do about this? What if this was because the sources of these logs were different systems? You can solve this with an additional label which is unique per system:

{job="syslog", instance="host1"} 00:00:00 i'm a syslog!
{job="syslog", instance="host1"} 00:00:02 i'm a syslog!
{job="syslog", instance="host2"} 00:00:01 i'm a syslog!  <- Accepted, this is a new stream!
{job="syslog", instance="host1"} 00:00:03 i'm a syslog!  <- Accepted, still in order for stream 1
{job="syslog", instance="host2"} 00:00:02 i'm a syslog!  <- Accepted, still in order for stream 2

But what if the application itself generated logs that were out of order? Well, I’m afraid this is a problem. If you are extracting the timestamp from the log line with something like the Alloy stage.timestamp block, you could instead not do this and let your log shipping agent assign a timestamp to the log lines. Or you can hopefully fix it in the application itself.

It’s also worth noting that the batching nature of the Loki push API can lead to some instances of out of order errors being received which are really false positives. (Perhaps a batch partially succeeded and was present; or anything that previously succeeded would return an out of order entry; or anything new would be accepted.)

Use `snappy` compression algorithm

Snappy is currently the Loki compression algorithm of choice. It performs much better than gzip for speed, but it is not as efficient in storage. This was an acceptable tradeoff for us.

Grafana Labs has found that gzip was very good for compression but was very slow, and this was causing slow query responses.

LZ4 is a good compromise of speed and compression performance. While compression is slightly slower than snappy, the compression ratio is higher, resulting in smaller chunks in object storage.

Use `chunk_target_size`

Using chunk_target_size instructs Loki to try to fill all chunks to a target compressed size of 1.5MB. These larger chunks are more efficient for Loki to process.

Other configuration variables affect how full a chunk can get. Loki has a default max_chunk_age of 2h and chunk_idle_period of 30m to limit the amount of memory used as well as the exposure of lost logs if the process crashes.

Depending on the compression used (Loki has been using snappy which has less compressibility but faster performance), you need 5-10x or 7.5-10MB of raw log data to fill a 1.5MB chunk. Remembering that a chunk is per stream, the more streams you break up your log files into, the more chunks that sit in memory, and the higher likelihood they get flushed by hitting one of those timeouts mentioned above before they are filled.

Lots of small, unfilled chunks negatively affect Loki. The team is always working to improve this and may consider a compactor to improve this in some situations. But, in general, the guidance should stay about the same: try your best to fill chunks.

If you have an application that can log fast enough to fill these chunks quickly (much less than max_chunk_age), then it becomes more reasonable to use dynamic labels to break that up into separate streams.

Use `-print-config-stderr` or `-log-config-reverse-order`

Loki has flags which will dump the entire config object to stderr or the log file when it starts.

-print-config-stderr works well when invoking Loki from the command line, as you can get a quick output of the entire Loki configuration.

-log-config-reverse-order is the flag Grafana runs Loki with in all our environments. The configuration entries are reversed, so that the order of the configuration reads correctly top to bottom when viewed in Grafana’s Explore.

Recommended production limits

limits_config:
  # Rate limits
  ingestion_rate_strategy: global
  ingestion_rate_mb: 10
  ingestion_burst_size_mb: 20
  per_stream_rate_limit: 3MB
  per_stream_rate_limit_burst: 15MB
  
  # Stream limits
  max_global_streams_per_user: 10000
  max_streams_per_user: 0
  
  # Validation
  max_line_size: 256KB
  max_line_size_truncate: false
  max_label_name_length: 1024
  max_label_value_length: 2048
  max_label_names_per_series: 15
  
  # Time constraints
  reject_old_samples: true
  reject_old_samples_max_age: 168h  # 7 days
  creation_grace_period: 10m
  unordered_writes: true

Recommended Ingester configuration

ingester:
  # Chunk settings
  chunk_idle_period: 30m
  chunk_target_size: 1572864  # 1.5 MB
  chunk_encoding: snappy
  max_chunk_age: 2h
  
  # WAL settings
  wal:
    enabled: true
    dir: /loki/wal
    checkpoint_duration: 5m
    flush_on_shutdown: true

Storage

Thu, 09 Apr 2026 02:28:18 +0000

Storage

Unlike other logging systems, Grafana Loki is built around the idea of only indexing metadata about your logs: labels (just like Prometheus labels). Log data itself is then compressed and stored in chunks in object stores such as S3 or GCS, or even locally on the filesystem. A small index and highly compressed chunks simplifies the operation and significantly lowers the cost of Loki.

Loki 2.8 introduced TSDB as a new mode for the Single Store and is now the recommended way to persist data in Loki. More detailed information about TSDB can be found under the manage section.

Loki 2.0 introduced an index mechanism named ‘boltdb-shipper’ and is what we now call Single Store. This type only requires one store, the object store, for both the index and chunks. More detailed information about ‘boltdb-shipper’ can be found under the manage section.

Prior to Loki 2.0, chunks and index data were stored in separate backends: object storage (or filesystem) for chunk data and NoSQL/Key-Value databases for index data. These “multistore” backends have been deprecated, as noted below.

You can find more detailed information about all of the storage options in the manage section.

Single Store

Single Store refers to using object storage as the storage medium for both the Loki index as well as its data (“chunks”). There are two supported modes:

TSDB (recommended)

Starting in Loki 2.8, the TSDB index store improves query performance, reduces TCO and has the same feature parity as “boltdb-shipper”. TSDB is the recommended index store for Loki 2.8 and newer.

BoltDB (deprecated)

Also known as “boltdb-shipper” during development (and is still the schema store name). The single store configurations for Loki utilize the chunk store for both chunks and the index, requiring just one store to run Loki. BoldDB is the recommended index store for Loki v2.0.0 through v2.7x.

Performance is comparable to a dedicated index type while providing a much less expensive and less complicated deployment. When using Single Store, no extra Chunk storage and Index storage are necessary.

Supported storage backends

See Object Storage for supported backends.

Chunk storage

File system

The file system is the simplest backend for chunks, although it’s also susceptible to data loss as it’s unreplicated. This is common for single binary deployments though, as well as for those trying out loki or doing local development on the project. It is similar in concept to many Prometheus deployments where a single Prometheus is responsible for monitoring a fleet.

Object storage

Google Cloud Storage (GCS)

GCS is a hosted object store offered by Google. It is a good candidate for a managed object store, especially when you’re already running on GCP, and is production safe.

Amazon Simple Storage Storage (S3)

S3 is AWS’s hosted object store. It is a good candidate for a managed object store, especially when you’re already running on AWS, and is production safe.

Azure Blob Storage

Blob Storage is Microsoft Azure’s hosted object store. It is a good candidate for a managed object store, especially when you’re already running on Azure, and is production safe. You can authenticate Blob Storage access by using a storage account name and key or by using a Service Principal.

IBM Cloud Object Storage (COS)

COS is IBM Cloud hosted object store. It is a good candidate for a managed object store, especially when you’re already running on IBM Cloud, and is production safe.

Baidu Object Storage (BOS)

BOS is the Baidu Cloud hosted object storage.

Alibaba Object Storage Service (OSS)

OSS is the Alibaba Cloud hosted object storage.

Other notable mentions

You may use any substitutable services, such as those that implement the S3 API like MinIO.

Cassandra (deprecated)

Cassandra is a popular database and one of the possible chunk stores for Loki and is production safe.

Note
This storage type for chunks is deprecated and may be removed in future major versions of Loki.

Index storage

Cassandra (deprecated)

Cassandra can also be utilized for the index store and aside from the boltdb-shipper, it’s the only non-cloud offering that can be used for the index that’s horizontally scalable and has configurable replication. It’s a good candidate when you already run Cassandra, are running on-prem, or do not wish to use a managed cloud offering.

Note
This storage type for indexes is deprecated and may be removed in future major versions of Loki.

BigTable (deprecated)

Bigtable is a cloud database offered by Google. It is a good candidate for a managed index store if you’re already using it (due to its heavy fixed costs) or wish to run in GCP.

Note
This storage type for indexes is deprecated and may be removed in future major versions of Loki.

DynamoDB (deprecated)

DynamoDB is a cloud database offered by AWS. It is a good candidate for a managed index store, especially if you’re already running in AWS.

Note
This storage type for indexes is deprecated and may be removed in future major versions of Loki.

Rate limiting

DynamoDB is susceptible to rate limiting, particularly due to overconsuming what is called provisioned capacity. This can be controlled via the provisioning configs in the table manager.

BoltDB (deprecated)

BoltDB is an embedded database on disk. It is not replicated and thus cannot be used for high availability or clustered Loki deployments, but is commonly paired with a filesystem chunk store for proof of concept deployments, trying out Loki, and development. The boltdb-shipper aims to support clustered deployments using boltdb as an index.

Note
This storage type for indexes is deprecated and may be removed in future major versions of Loki.

Schema Config

Loki aims to be backwards compatible and over the course of its development has had many internal changes that facilitate better and more efficient storage/querying. Loki allows incrementally upgrading to these new storage schemas and can query across them transparently. This makes upgrading a breeze. For instance, this is what it looks like when migrating from BoltDB with v11 schema to TSDB with v13 schema starting 2023-07-01:

schema_config:
  configs:
    - from: 2019-07-01
      store: boltdb
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h
    - from: 2023-07-01
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

For all data ingested before 2023-07-01, Loki used BoltDB with the v11 schema, and then switched after that point to the more effective TSDB with the v13 schema. This dramatically simplifies upgrading, ensuring it’s simple to take advantage of new storage optimizations. These configs should be immutable for as long as you care about retention.

Table Manager (deprecated)

One of the subcomponents in Loki is the table-manager. It is responsible for pre-creating and expiring index tables. This helps partition the writes and reads in Loki across a set of distinct indices in order to prevent unbounded growth.

table_manager:
  # The retention period must be a multiple of the index / chunks
  # table "period" (see period_config).
  retention_deletes_enabled: true
  # This is 15 weeks retention, based on the 168h (1week) period durations used in the rest of the examples.
  retention_period: 2520h

For more information, see the table manager documentation.

Provisioning

In the case of AWS DynamoDB, you’ll likely want to tune the provisioned throughput for your tables as well. This is to prevent your tables being rate limited on one hand and assuming unnecessary cost on the other. By default Loki uses a provisioned capacity strategy for DynamoDB tables like so:

table_manager:
  index_tables_provisioning:
    # Read/write throughput requirements for the current table
    # (the table which would handle writes/reads for data timestamped at the current time)
    provisioned_write_throughput: <int> | default = 3000
    provisioned_read_throughput: <int> | default = 300

    # Read/write throughput requirements for non-current tables
    inactive_write_throughput: <int> | default = 1
    inactive_read_throughput: <int> | Default = 300

Note, there are a few other DynamoDB provisioning options including DynamoDB autoscaling and on-demand capacity. See the provisioning configuration in the table_manager block documentation for more information.

Upgrading Schemas

When a new schema is released and you want to gain the advantages it provides, you can! Loki can transparently query and merge data from across schema boundaries so there is no disruption of service and upgrading is easy.

First, you’ll want to create a new period_config entry in your schema_config. The important thing to remember here is to set this at some point in the future and then roll out the config file changes to Loki. This allows the table manager to create the required table in advance of writes and ensures that existing data isn’t queried as if it adheres to the new schema.

As an example, let’s say it’s 2023-07-14 and you want to start using the v13 schema on the 20th:

schema_config:
  configs:
    - from: 2019-07-14
      store: tsdb
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h
    - from: 2023-07-20
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

It’s that easy; you just created a new entry starting on the 20th.

Retention

Loki manages retention through the Compactor when using TSDB or the BoltDB Shipper. When retention is enabled, the Compactor identifies data that falls outside of the configured retention period, removes the corresponding index entries, and deletes the underlying chunk objects asynchronously.

For object storage backends (S3, GCS, Azure Blob) Loki no longer relies solely on external time to live (TTL) or bucket lifecycle rules; these may still be used as an additional safeguard, but Loki itself performs retention-driven deletion when configured.

When using the filesystem chunk store, Loki does not delete data based on disk usage or free-space conditions. Deletion is determined only by the retention settings, and disk-full scenarios must be handled operationally outside of Loki.

Loki also supports targeted deletion at the tenant or stream level.

For more information, see the retention configuration documentation.

Examples

Single machine/local development (tsdb+filesystem)

The repo contains a working example, you may want to checkout a tag of the repo to make sure you get a compatible example.

GCP deployment (GCS Single Store)

storage_config:
  tsdb_shipper:
    active_index_directory: /loki/index
    cache_location: /loki/index_cache
    cache_ttl: 24h # Can be increased for faster performance over longer query periods, uses more disk space
  gcs:
    bucket_name: <bucket>
    service_account: |
      {
        "type": "service_account",
        ...
      }

schema_config:
  configs:
    - from: 2020-07-01
      store: tsdb
      object_store: gcs
      schema: v13
      index:
        prefix: index_
        period: 24h

service_account should contain JSON from either a GCP Console client_credentials.json file or a GCP service account key. If this value is blank, most services will fall back to GCP’s Application Default Credentials (ADC) strategy. For more information about ADC, refer to How Application Default Credentials works.

The pre-defined storage.objectUser role (or a custom role modeled after it) contains sufficient permissions for Loki to operate.

Note
GCP recommends Workload Identity Federation instead of a service account key.

AWS deployment (S3 Single Store)

storage_config:
  tsdb_shipper:
    active_index_directory: /loki/index
    cache_location: /loki/index_cache
    cache_ttl: 24h         # Can be increased for faster performance over longer query periods, uses more disk space
  aws:
    s3: s3://<access_key>:<uri-encoded-secret-access-key>@<region>
    bucketnames: <bucket1,bucket2>

schema_config:
  configs:
    - from: 2020-07-01
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h

If you don’t wish to hard-code S3 credentials, you can also configure an EC2 instance role by changing the storage_config section:

storage_config:
  aws:
    s3: s3://region
    bucketnames: <bucket1,bucket2>

The role should have a policy with the following permissions attached.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "LokiStorage",
            "Effect": "Allow",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::<account_ID>"
                ]
            },
            "Action": [
                "s3:ListBucket",
                "s3:PutObject",
                "s3:GetObject",
                "s3:DeleteObject"
            ],
            "Resource": [
                "arn:aws:s3:::<bucket_name>",
                "arn:aws:s3:::<bucket_name>/*"
            ]
        }
    ]
}

To setup an S3 bucket and an IAM role and policy:

This guide assumes a provisioned EKS cluster.

Checkout the Loki repository and navigate to production/terraform/modules/s3.
Initialize Terraform terraform init.

Export the AWS profile and region if not done so:

export AWS_PROFILE=<profile in ~/.aws/config>
export AWS_REGION=<region of EKS cluster>

Save the OIDC provider in an environment variable:

oidc_provider=$(aws eks describe-cluster --name <EKS cluster> --query "cluster.identity.oidc.issuer" --output text | sed -e "s/^https:\/\///")

See the IAM OIDC provider guide for a guide for creating a provider.

Apply the Terraform module terraform -var region="$AWS_REGION" -var cluster_name=<EKS cluster> -var oidc_id="$oidc_provider"

Note, the bucket name defaults to loki-data but can be changed via the bucket_name variable.

Azure deployment (Azure Blob Storage Single Store)

Using account name and key

schema_config:
  configs:
    - from: "2020-12-11"
      index:
        period: 24h
        prefix: index_
      object_store: azure
      schema: v13
      store: tsdb
storage_config:
  azure:
    # Your Azure storage account name
    account_name: <account-name>
    # For the account-key, see docs: https://docs.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage?tabs=azure-portal
    account_key: <account-key>
    # See https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction#containers
    container_name: <container-name>
    use_managed_identity: <true|false>
    # Providing a user assigned ID will override use_managed_identity
    user_assigned_id: <user-assigned-identity-id>
    request_timeout: 0
    # Configure this if you are using private azure cloud like azure stack hub and will use this endpoint suffix to compose container and blob storage URL. Ex: https://account_name.endpoint_suffix/container_name/blob_name
    endpoint_suffix: <endpoint-suffix>
    # If `connection_string` is set, the values of `account_name` and `endpoint_suffix` values will not be used. Use this method over `account_key` if you need to authenticate via a SAS token. Or if you use the Azurite emulator.
    connection_string: <connection-string>
  tsdb_shipper:
    active_index_directory: /loki/index
    cache_location: /loki/index_cache
    cache_ttl: 24h
  filesystem:
    directory: /loki/chunks

Using a service principal

schema_config:
  configs:
    - from: "2020-12-11"
      index:
        period: 24h
        prefix: index_
      object_store: azure
      schema: v13
      store: tsdb
storage_config:
  azure:
    use_service_principal: true
    # Azure tenant ID used to authenticate through Azure OAuth
    tenant_id : <tenant-id>
    # Azure Service Principal ID
    client_id: <client-id>
    # Azure Service Principal secret key
    client_secret: <client-secret>
    # See https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction#containers
    container_name: <container-name>
    request_timeout: 0
  tsdb_shipper:
    active_index_directory: /loki/index
    cache_location: /loki/index_cache
    cache_ttl: 24h
  filesystem:
    directory: /loki/chunks

IBM Deployment (COS Single Store)

schema_config:
  configs:
    - from: 2020-10-01
      index:
        period: 24h
        prefix: loki_index_
      object_store: cos
      schema: v13
      store: tsdb

storage_config:
  tsdb_shipper:
    active_index_directory: /loki/index
    cache_location: /loki/index_cache
  cos:
    bucketnames: <bucket1, bucket2>
    endpoint: <endpoint>
    api_key: <api_key_to_authenticate_with_cos>
    region: <region>
    service_instance_id: <cos_service_instance_id>
    auth_endpoint: <iam_endpoint_for_authentication>

On premise deployment (Cassandra+Cassandra)

Note
Cassandra as storage backend for chunks and indexes is deprecated.

Keeping this for posterity, but this is likely not a common config. Cassandra should work and could be faster in some situations but is likely much more expensive.

storage_config:
  cassandra:
    addresses: <comma-separated-IPs-or-hostnames>
    keyspace: <keyspace>
    auth: <true|false>
    username: <username> # only applicable when auth=true
    password: <password> # only applicable when auth=true

schema_config:
  configs:
    - from: 2020-07-01
      store: cassandra
      object_store: cassandra
      schema: v11
      index:
        prefix: index_
        period: 168h
      chunks:
        prefix: chunk_
        period: 168h

On premise deployment (MinIO Single Store)

You configure MinIO by using the AWS config because MinIO implements the S3 API:

storage_config:
  aws:
    # Note: use a fully qualified domain name (fqdn), like localhost.
    # full example: http://loki:supersecret@localhost.:9000
    s3: http<s>://<username>:<secret>@<fqdn>:<port>
    s3forcepathstyle: true
  tsdb_shipper:
    active_index_directory: /loki/index
    cache_location: /loki/index_cache
    cache_ttl: 24h         # Can be increased for faster performance over longer query periods, uses more disk space

schema_config:
  configs:
    - from: 2020-07-01
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h

Examples

Thu, 09 Apr 2026 02:28:18 +0000

Examples

The following pages contain examples of how to configure Grafana Loki.

Grafana Loki configuration parameters on Grafana Labs

Configuration best practices

Configuration best practices

Configure authentication

Configure caching

Time ordering of logs

Use snappy compression algorithm

Use chunk_target_size

Use -print-config-stderr or -log-config-reverse-order

Recommended production limits

Recommended Ingester configuration

Storage

Storage

Single Store

TSDB (recommended)

BoltDB (deprecated)

Supported storage backends

Chunk storage

File system

Object storage

Google Cloud Storage (GCS)

Amazon Simple Storage Storage (S3)

Azure Blob Storage

IBM Cloud Object Storage (COS)

Baidu Object Storage (BOS)

Alibaba Object Storage Service (OSS)

Other notable mentions

Cassandra (deprecated)

Index storage

Cassandra (deprecated)

BigTable (deprecated)

DynamoDB (deprecated)

Rate limiting

BoltDB (deprecated)

Schema Config

Table Manager (deprecated)

Provisioning

Upgrading Schemas

Retention

Examples

Single machine/local development (tsdb+filesystem)

GCP deployment (GCS Single Store)

AWS deployment (S3 Single Store)

Azure deployment (Azure Blob Storage Single Store)

Using account name and key

Using a service principal

IBM Deployment (COS Single Store)

On premise deployment (Cassandra+Cassandra)

On premise deployment (MinIO Single Store)

Examples

Examples

Use `snappy` compression algorithm

Use `chunk_target_size`

Use `-print-config-stderr` or `-log-config-reverse-order`