Manage storage on Grafana Labs

Single Store TSDB (tsdb)

Thu, 09 Apr 2026 02:28:18 +0000

Single Store TSDB (tsdb)

Starting with Loki v2.8, TSDB is the recommended Loki index. It is heavily inspired by the Prometheus’s TSDB sub-project. For a deeper explanation you can read Loki maintainer Owen’s blog post. The short version is that this new index is more efficient, faster, and more scalable. It also resides in object storage like the boltdb-shipper index which preceded it.

Example Configuration

To get started using TSDB, add the following configurations to your config.yaml:

schema_config:
  configs:
    # Old boltdb-shipper schema. Included in example for reference but does not need changing.
    - from: "2023-01-03" # <---- A date in the past
      index:
        period: 24h
        prefix: index_
      object_store: gcs
      schema: v12
      store: boltdb-shipper
    # New TSDB schema below
    - from: "2023-01-05" # <---- A date in the future
      index:
        period: 24h
        prefix: index_
      object_store: gcs
      schema: v13
      store: tsdb

storage_config:
  # Old boltdb-shipper configuration. Included in example for reference but does not need changing.
  boltdb_shipper:
    active_index_directory: /data/index
    build_per_tenant_index: true
    cache_location: /data/boltdb-cache
    index_gateway_client:
      # only applicable if using microservices where index-gateways are independently deployed.
      # This example is using kubernetes-style naming.
      server_address: dns:///index-gateway.<namespace>.svc.cluster.local:9095
  # New tsdb-shipper configuration
  tsdb_shipper:
    active_index_directory: /data/tsdb-index
    cache_location: /data/tsdb-cache
    index_gateway_client:
      # only applicable if using microservices where index-gateways are independently deployed.
      # This example is using kubernetes-style naming.
      server_address: dns:///index-gateway.<namespace>.svc.cluster.local:9095

query_scheduler:
  # the TSDB index dispatches many more, but each individually smaller, requests. 
  # We increase the pending request queue sizes to compensate.
  max_outstanding_requests_per_tenant: 32768

querier:
  # Each `querier` component process runs a number of parallel workers to process queries simultaneously.
  # You may want to adjust this up or down depending on your resource usage
  # (more available cpu and memory can tolerate higher values and vice versa),
  # but we find the most success running at around `16` with tsdb
  max_concurrent: 16

Operations

Limits

We’ve added a user per-tenant limit called tsdb_max_query_parallelism in the limits_config. This functions the same as the prior max_query_parallelism configuration but applies to tsdb queries instead. Since the TSDB index will create many more smaller queries compared to the other index types before it, we’ve added a separate configuration so they can coexist. This is helpful when transitioning between index types. The default parallelism is 128 which should work well for most cases, but you can extend it globally in the limits_config or per-tenant in the overrides file as needed.

Dynamic Query Sharding

Previously we would statically shard queries based on the index row shards configured here. TSDB does Dynamic Query Sharding based on how much data a query is going to be processing. We additionally store size(KB) and number of lines for each chunk in the TSDB index which is then used by the Query Frontend for planning the query. Based on our experience from operating many Loki clusters, we have configured TSDB to aim for processing 300-600 MBs of data per query shard. This means with TSDB we will be running more, smaller queries.

Index Caching not required

TSDB is a compact and optimized format. Loki does not currently use an index cache for TSDB. If you are already using Loki with other index types, it is recommended to keep the index caching until all of your existing data falls out of retention) or your configured max_query_lookback under limits_config. After that, we suggest running without an index cache (it isn’t used in TSDB).

Single Store BoltDB (boltdb-shipper)

Thu, 09 Apr 2026 02:28:18 +0000

Single Store BoltDB (boltdb-shipper)

Note
Single store BoltDB Shipper is a legacy storage option recommended for Loki 2.0 through 2.7.x and is not recommended for new deployments. The TSDB is the recommended index for Loki 2.8 and newer.

BoltDB Shipper lets you run Grafana Loki without any dependency on NoSQL stores for storing index. It locally stores the index in BoltDB files instead and keeps shipping those files to a shared object store i.e the same object store which is being used for storing chunks. It also keeps syncing BoltDB files from shared object store to a configured local directory for getting index entries created by other services of same Loki cluster. This helps run Loki with one less dependency and also saves costs in storage since object stores are likely to be much cheaper compared to cost of a hosted NoSQL store or running a self hosted instance of Cassandra.

Note
BoltDB shipper works best with 24h periodic index files. It is a requirement to have the index period set to 24h for either active or upcoming usage of boltdb-shipper. If boltdb-shipper already has created index files with 7 days period, and you want to retain previous data, add a new schema config using boltdb-shipper with a future date and index files period set to 24h.

Example Configuration

Example configuration with GCS:

schema_config:
  configs:
    - from: 2018-04-15
      store: boltdb-shipper
      object_store: gcs
      schema: v11
      index:
        prefix: loki_index_
        period: 24h

storage_config:
  gcs:
    bucket_name: GCS_BUCKET_NAME

  boltdb_shipper:
    active_index_directory: /loki/index
    cache_location: /loki/boltdb-cache

This would run Loki with BoltDB Shipper storing BoltDB files locally at /loki/index and chunks at configured GCS_BUCKET_NAME. It would also keep shipping BoltDB files periodically to same configured bucket. It would also keep downloading BoltDB files from shared bucket uploaded by other ingesters to /loki/boltdb-cache folder locally.

Operational Details

Loki can be configured to run as just a single vertically scaled instance or as a cluster of horizontally scaled single binary(running all Loki services) instances or in micro-services mode running just one of the services in each instance. When it comes to reads and writes, Ingesters are the ones which writes the index and chunks to stores and Queriers are the ones which reads index and chunks from the store for serving requests.

Before we get into more details, it is important to understand how Loki manages index in stores. Loki shards index as per configured period which defaults to seven days i.e when it comes to table based stores like Bigtable/Cassandra/DynamoDB there would be separate table per week containing index for that week. In the case of BoltDB Shipper, a table is defined by a collection of many smaller BoltDB files, each file storing just 15 mins worth of index. Tables created per day are identified by a configured prefix_ + <period-number-since-epoch>. Here <period-number-since-epoch> in case of boltdb-shipper would be day number since epoch. For example, if you have a prefix set to loki_index_ and a write request comes in on 20th April 2020, it would be stored in a table named loki_index_18372 because it has been 18371 days since the epoch, and we are in 18372th day. Since sharding of index creates multiple files when using BoltDB, BoltDB Shipper would create a folder per day and add files for that day in that folder and names those files after ingesters which created them.

To reduce the size of files which help with faster transfer speeds and reduced storage costs, they are stored after compressing them with gzip.

To show how BoltDB files in shared object store would look like, let us consider 2 ingesters named ingester-0 and ingester-1 running in a Loki cluster, and they both having shipped files for day 18371 and 18372 with prefix loki_index_, here is how the files would look like:

└── index
    ├── loki_index_18371
    │   ├── ingester-0-1587254400.gz
    │   └── ingester-1-1587255300.gz
    |   ...
    └── loki_index_18372
        ├── ingester-0-1587254400.gz
        └── ingester-1-1587254400.gz
        ...

Note
Loki also adds a timestamp to names of the files to randomize the names to avoid overwriting files when running Ingesters with same name and not have a persistent storage. Timestamps not shown here for simplification.

Let us talk about more in depth about how both Ingesters and Queriers work when running them with BoltDB Shipper.

Ingesters

Ingesters write the index to BoltDB files in active_index_directory, and the BoltDB Shipper looks for new and updated files in that directory at 1 minute intervals, to upload them to the shared object store. When running Loki in microservices mode, there could be multiple ingesters serving write requests. Each ingester generates BoltDB files locally.

Note
To avoid any loss of index when an ingester crashes, we recommend running ingesters as a StatefulSet (when using Kubernetes) with a persistent storage for storing index files.

When chunks are flushed, they are available for reads in the object store instantly. The index is not available instantly, since we upload every 15 minutes with the BoltDB shipper. Ingesters expose a new RPC for letting queriers query the ingester’s local index for chunks which were recently flushed, but its index might not be available yet with queriers. For all the queries which require chunks to be read from the store, queriers also query ingesters over RPC for IDs of chunks which were recently flushed. This avoids missing any logs from queries.

Queriers

To avoid running Queriers as a StatefulSet with persistent storage, we recommend running an Index Gateway. An Index Gateway will download and synchronize the index, and it will serve it over gRPC to Queriers and Rulers.

Queriers lazily loads BoltDB files from shared object store to configured cache_location. When a querier receives a read request, the query range from the request is resolved to period numbers and all the files for those period numbers are downloaded to cache_location, if not already. Once we have downloaded files for a period we keep looking for updates in shared object store and download them every 5 Minutes by default. Frequency for checking updates can be configured with resync_interval config.

To avoid keeping downloaded index files forever there is a ttl for them which defaults to 24 hours, which means if index files for a period are not used for 24 hours they would be removed from cache location. ttl can be configured using cache_ttl config.

Within Kubernetes, if you are not using an Index Gateway, we recommend running Queriers as a StatefulSet with persistent storage for downloading and querying index files. This will obtain better read performance, and it will avoid using node disk.

Index Gateway

An Index Gateway downloads and synchronizes the BoltDB index from the Object Storage in order to serve index queries to the Queriers and Rulers over gRPC. This avoids running Queriers and Rulers with a disk for persistence. Disks can become costly in a big cluster.

To run an Index Gateway, configure StorageConfig and set the -target CLI flag to index-gateway. To connect Queriers and Rulers to the Index Gateway, set the address (with gRPC port) of the Index Gateway with the -boltdb.shipper.index-gateway-client.server-address CLI flag or its equivalent YAML value under StorageConfig.

When using the Index Gateway within Kubernetes, we recommend using a StatefulSet with persistent storage for downloading and querying index files. This can obtain better read performance, avoids noisy neighbor problems by not using the node disk, and avoids the time consuming index downloading step on startup after rescheduling to a new node.

Write Deduplication disabled

Loki does write deduplication of chunks and index using Chunks and WriteDedupe cache respectively, configured with ChunkStoreConfig. The problem with write deduplication when using boltdb-shipper though is ingesters only keep uploading boltdb files periodically to make them available to all the other services which means there would be a brief period where some of the services would not have received updated index yet. The problem due to that is if an ingester which first wrote the chunks and index goes down and all the other ingesters which were part of replication scheme skipped writing those chunks and index due to deduplication, we would end up missing those logs from query responses since only the ingester which had the index went down. This problem would be faced even during rollouts which is quite common.

To avoid this, Loki disables deduplication of index when the replication factor is greater than 1 and boltdb-shipper is an active or upcoming index type. While using boltdb-shipper avoid configuring WriteDedupe cache since it is used purely for the index deduplication, so it would not be used anyways.

Compactor

Compactor is a BoltDB Shipper specific service that reduces the index size by deduping the index and merging all the files to a single file per table. We recommend running a Compactor since a single Ingester creates 96 files per day which include a lot of duplicate index entries and querying multiple files per table adds up the overall query latency.

Note
There should be only one compactor instance running at a time that otherwise could create problems and may lead to data loss.

Example compactor configuration with GCS:

Delete Permissions

The compactor is an optional but suggested component that combines and deduplicates the boltdb-shipper index files. When compacting index files, the compactor writes a new file and deletes unoptimized files. Ensure that the compactor has appropriate permissions for deleting files, for example, s3:DeleteObject permission for AWS S3.

compactor:
  working_directory: /loki/compactor

storage_config:
  gcs:
    bucket_name: GCS_BUCKET_NAME

Filesystem object store

Thu, 09 Apr 2026 02:28:18 +0000

Filesystem object store

The filesystem object store is the easiest to get started with Grafana Loki but there are some pros/cons to this approach.

Very simply it stores all the objects (chunks) in the specified directory:

storage_config:
  filesystem:
    directory: /tmp/loki/

A folder is created for every tenant all the chunks for one tenant are stored in that directory.

If Loki is run in single-tenant mode, all the chunks are put in a folder named fake which is the synthesized tenant name used for single tenant mode.

See multi-tenancy for more information.

Pros

Very simple, no additional software required to use Loki when paired with the BoltDB index store.

Great for low volume applications, proof of concepts, and just playing around with Loki.

Cons

The filesystem is not supported by Grafana Labs for production environments (for those customers who have purchased a support contract).

Scaling

At some point there is a limit to how many chunks can be stored in a single directory, for example see issue #1502 which explains how a Loki user ran into a strange error with about 5.5 million chunk files in their file store (and also a workaround for the problem).

However, if you keep your streams low (remember loki writes a chunk per stream) and use configs like chunk_target_size (around 1MB), max_chunk_age (increase beyond 1h), chunk_idle_period (increase to match max_chunk_age) can be tweaked to reduce the number of chunks flushed (although they will trade for more memory consumption).

It’s still very possible to store terabytes of log data with the filestore, but realize there are limitations to how many files a filesystem will want to store in a single directory.

Durability

The durability of the objects is at the mercy of the filesystem itself where other object stores like S3/GCS do a lot behind the scenes to offer extremely high durability to your data.

High Availability

Running Loki clustered is not possible with the filesystem store unless the filesystem is shared in some fashion (NFS for example). However using shared filesystems is likely going to be a bad experience with Loki just as it is for almost every other application.

Storage schema

Thu, 09 Apr 2026 02:28:18 +0000

Storage schema

To support iterations over the storage layer contents, Loki has a configurable storage schema. The schema is defined to apply over periods of time. A from value marks the starting point of that schema. The schema is active until another entry defines a new schema with a new from date.

Loki uses the defined schemas to determine which format to use when storing and querying the data.

Use of a schema allows Loki to iterate over the storage layer without requiring migration of existing data.

New Loki installs

For a new Loki install with no previous data, here is an example schema configuration with recommended values

schema_config:
  configs:
    - from: 2024-04-01
      object_store: s3
      store: tsdb
      schema: v13
      index:
        prefix: index_
        period: 24h

Property	Description
from	for a new install, this must be a date in the past, use a recent date. Format is YYYY-MM-DD.
object_store	s3, azure, gcs, alibabacloud, bos, cos, swift, filesystem, or a named_store (see StorageConfig).
store	`tsdb` is the current and only recommended value for store.
schema	`v13` is the most recent schema and recommended value.
prefix:	any value without spaces is acceptable.
period:	must be `24h`.

Note
For a new install, the from date must be in the past so the schema is immediately active when Loki starts. If you set it to a future date, Loki will have no valid schema for the current time and will not be able to store incoming data.

This is different from adding a new schema entry to an existing install, where the from date must be in the future. See Changing the schema below.

Changing the schema

Note
The guidance in this section applies when you are adding a new schema entry to an existing Loki install that already has data. Setting the from date to a future date gives Loki time to transition to the new schema and ensures that existing data continues to be read using the old schema. If the from date is not in the future, data written just before the cutover may become unreadable because Loki would try to query it using the wrong schema.

For a brand new install with no previous data, the from date should be in the past instead. See New Loki installs above.

Here are items to consider when changing the schema; if schema changes are not done properly, a scenario can be created which prevents data from being read.

Always set the from date in the new schema to a date in the future.

The from date is interpreted by Loki to start at 00:00:00 UTC. Therefore, Loki must have a date in the future to be able to transition to the new schema when that date and time arrives.

Be aware of your relation to UTC when using the current date. Make sure that UTC 00:00:00 has not already passed for your current date.

As an example, assume that the current date is 2022-04-10, and you want to update to the v13 schema, so you restart Loki with 2022-04-11 as the from date for the new schema. If you forget to take into account that your timezone is UTC -5:00 and it’s currently 20:00 hours in your local timezone, that is actually 2022-04-11T01:00:00 UTC. When Loki starts it will see the new schema and begin to write and store objects following that new schema. If you then try to query data that was written between 00:00:00 and 01:00:00 UTC, Loki will use the new schema and the data will be unreadable, because it was created with the previous schema.
You cannot undo or roll back a schema change.

Any data written with an active schema can only be read by that schema. If you wish to return to the previous schema; you can add another new entry with the previous schema settings.

Schema configuration example

schema_config:
  configs:
    - from: "2020-07-31"
      index:
        period: 24h
        prefix: loki_ops_index_
      object_store: gcs
      schema: v11
      store: tsdb
    - from: "2022-01-20"
      index:
        period: 24h
        prefix: loki_ops_index_
      object_store: gcs
      schema: v13
      store: tsdb

Write Ahead Log

Thu, 09 Apr 2026 02:28:18 +0000

Write Ahead Log

Ingesters temporarily store data in memory. In the event of a crash, there could be data loss. The Write Ahead Log (WAL) helps fill this gap in reliability.

The WAL in Grafana Loki records incoming data and stores it on the local file system in order to guarantee persistence of acknowledged data in the event of a process crash. Upon restart, Loki will “replay” all of the data in the log before registering itself as ready for subsequent writes. This allows Loki to maintain the performance and cost benefits of buffering data in memory and durability benefits (it won’t lose data once a write has been acknowledged).

This section will use Kubernetes as a reference deployment paradigm in the examples.

Disclaimer and WAL nuances

The Write Ahead Log in Loki takes a few particular tradeoffs compared to other WALs you may be familiar with. The WAL aims to add additional durability guarantees, but not at the expense of availability. Particularly, there are two scenarios where the WAL sacrifices these guarantees.

Corruption/Deletion of the WAL prior to replaying it

In the event the WAL is corrupted/partially deleted, Loki will not be able to recover all of its data. In this case, Loki will attempt to recover any data it can, but will not prevent Loki from starting.

You can use the Prometheus metric loki_ingester_wal_corruptions_total to track and alert when this happens.
No space left on disk

In the event the underlying WAL disk is full, Loki will not fail incoming writes, but neither will it log them to the WAL. In this case, the persistence guarantees across process restarts will not hold.

You can use the Prometheus metric loki_ingester_wal_disk_full_failures_total to track and alert when this happens.

Backpressure

The WAL also includes a backpressure mechanism to allow a large WAL to be replayed within a smaller memory bound. This is helpful after bad scenarios (i.e. an outage) when a WAL has grown past the point it may be recovered in memory. In this case, the ingester will track the amount of data being replayed and once it’s passed the ingester.wal-replay-memory-ceiling threshold, will flush to storage. When this happens, it’s likely that the Loki attempt to deduplicate chunks via content addressable storage will suffer. We deemed this efficiency loss an acceptable tradeoff considering how it simplifies operation and that it should not occur during regular operation (rollouts, rescheduling) where the WAL can be replayed without triggering this threshold.

Metrics

The following metrics are available for monitoring the WAL:

loki_ingester_wal_corruptions_total: Total number of WAL corruptions encountered
loki_ingester_wal_disk_full_failures_total: Total number of disk full failures
loki_ingester_wal_records_logged: Counter for WAL records logged
loki_ingester_wal_logged_bytes_total: Total bytes written to WAL

Changes to deployment

Since ingesters need to have the same persistent volume across restarts/rollout, all the ingesters should be run on StatefulSet with fixed volumes.
Following flags needs to be set
- --ingester.wal-enabled to true which enables writing to WAL during ingestion.
- --ingester.wal-dir to the directory where the WAL data should be stored and/or recovered from. Note that this should be on the mounted volume.
- --ingester.checkpoint-duration to the interval at which checkpoints should be created.
- --ingester.wal-replay-memory-ceiling (default 4GB) may be set higher/lower depending on your resource settings. It handles memory pressure during WAL replays, allowing a WAL many times larger than available memory to be replayed. This is provided to minimize reconciliation time after very bad situations, i.e. an outage, and will likely not impact regular operations/rollouts at all. We suggest setting this to a high percentage (~75%) of available memory.

Changes in lifecycle when WAL is enabled

Flushing of data to chunk store during rollouts or scale down is disabled. This is because during a rollout of statefulset there are no ingesters that are simultaneously leaving and joining, rather the same ingester is shut down and brought back again with updated config. Hence flushing is skipped and the data is recovered from the WAL. If you need to ensure that data is always flushed to the chunk store when your pod shuts down, you can set the --ingester.flush-on-shutdown flag to true.

Disk space requirements

Based on tests in real world:

Numbers from an ingester with 5000 series ingesting ~5mb/s.
Checkpoint period was 5mins.
disk utilization on a WAL-only disk was steady at ~10-15GB.

You should not target 100% disk utilization.

Migrating from stateless deployments

The ingester Deployment without WAL and StatefulSet with WAL should be scaled down and up respectively in sync without transfer of data between them to ensure that any ingestion after migration is reliable immediately.

Let’s take an example of 4 ingesters. The migration would look something like this:

Bring up one stateful ingester ingester-0 and wait until it’s ready (accepting read and write requests).
Scale down the old ingester deployment to 3 and wait until the leaving ingester flushes all the data to chunk store.
Once that ingester has disappeared from kc get pods ..., add another stateful ingester and wait until it’s ready. Now you have ingester-0 and ingester-1.
Repeat step 2 to reduce remove another ingester from old deployment.
Repeat step 3 to add another stateful ingester. Now you have ingester-0 ingester-1 ingester-2.
Repeat step 4 and 5, and now you will finally have ingester-0 ingester-1 ingester-2 ingester-3.

How to scale up/down

Scale up

Scaling up is same as what you would do without WAL or StatefulSets. Nothing to change here.

Scale down

When scaling down, we must ensure existing data on the leaving ingesters are flushed to storage instead of just the WAL. This is because we won’t be replaying the WAL on an ingester that will no longer exist and we need to make sure the data is not orphaned.

Consider you have 4 ingesters ingester-0 ingester-1 ingester-2 ingester-3 and you want to scale down to 2 ingesters, the ingesters which will be shut down according to StatefulSet rules are ingester-3 and then ingester-2.

Hence before actually scaling down in Kubernetes, port forward those ingesters and hit the /ingester/shutdown?flush=true endpoint. This will flush the chunks and remove itself from the ring, after which it will register as unready and may be deleted.

After hitting the endpoint for ingester-2 ingester-3, scale down the ingesters to 2.

Also you can set the --ingester.flush-on-shutdown flag to true. This enables chunks to be flushed to long-term storage when the ingester is shut down.

Additional notes

Kubernetes hacking

StatefulSets are significantly more cumbersome to work with, upgrade, and so on. Much of this stems from immutable fields on the specification. For example, if one wants to start using the WAL with single store Loki and wants separate volume mounts for the WAL and the boltdb-shipper, you may see immutability errors when attempting updates the Kubernetes StatefulSets.

In this case, try kubectl -n <namespace> delete sts ingester --cascade=false. This will leave the Pods alive but delete the StatefulSet. Then you may recreate the (updated) StatefulSet and one-by-one start deleting the ingester-0 through ingester-n Pods in that order, allowing the StatefulSet to spin up new pods to replace them.

Scaling Down Using `/flush_shutdown` Endpoint and Lifecycle Hook

StatefulSets for Ordered Scaling Down: The Loki ingesters should be scaled down one by one, which is efficiently handled by Kubernetes StatefulSets. This ensures an ordered and reliable scaling process, as described in the Deployment and Scaling Guarantees documentation.
Using PreStop Lifecycle Hook: During the Pod scaling down process, the PreStop lifecycle hook triggers the /flush_shutdown endpoint on the ingester. This action flushes the chunks and removes the ingester from the ring, allowing it to register as unready and become eligible for deletion.
Using terminationGracePeriodSeconds: Provides time for the ingester to flush its data before being deleted, if flushing data takes more than 30 minutes, you may need to increase it.
Cleaning Persistent Volumes: Persistent volumes are automatically cleaned up by leveraging the enableStatefulSetAutoDeletePVC feature in Kubernetes.

By following the above steps, you can ensure a smooth scaling down process for the Loki ingesters while maintaining data integrity and minimizing potential disruptions.

Non-Kubernetes or baremetal deployments

When the ingester restarts for any reason (upgrade, crash, etc), it should be able to attach to the same volume in order to recover back the WAL and tokens.
2 ingesters should not be working with the same volume/directory for the WAL.
A rollout should bring down an ingester completely and then start the new ingester, not the other way around.

Log retention

Thu, 09 Apr 2026 02:28:18 +0000

Log retention

Retention in Grafana Loki is achieved through the Compactor. By default the compactor.retention-enabled flag is not set, so the logs sent to Loki live forever.

Note
If you have a lifecycle policy configured on the object store, please ensure that it is longer than the retention period.

Granular retention policies to apply retention at per tenant or per stream level are also supported by the Compactor.

Note
The Compactor does not support retention on legacy index types. Please use the Table Manager when using legacy index types. Both the Table manager and legacy index types are deprecated and may be removed in future major versions of Loki.

Compactor

The Compactor is responsible for compaction of index files and applying log retention.

Note
Run the Compactor as a singleton (a single instance).

The Compactor loops to apply compaction and retention at every compactor.compaction-interval, or as soon as possible if running behind. Both compaction and retention are idempotent, which means once the action has been performed, if the action is performed multiple times, it has no further effect on logs after the first time it is performed. If the Compactor restarts, it continues from where it left off.

Note
Changes to your retention period are not retroactive, that is, they are not applied to logs that have already been ingested.

The Compactor’s algorithm to apply retention is as follows:

For each day or table (one table per day with 24h index period):
- Compact multiple index files in the table into per-tenant index files. Result of compaction is a single index file per tenant per day.
- Traverse the per-tenant index. Use the tenant configuration to identify the chunks that need to be removed.
- Remove the references to the matching chunks from the index and add the chunk references to a marker file on disk.
- Upload the new modified index files.

Chunks are not deleted while applying the retention algorithm on the index. They are deleted asynchronously by a sweeper process and this delay can be configured by setting -compactor.retention-delete-delay. Marker files are used to keep track of the chunks pending for deletion.

Chunks cannot be deleted immediately for the following reasons:

Index Gateway downloads a copy of the index files to serve queries and refreshes them at a regular interval. Having a delay allows the index gateways to pull the modified index file which would not contain any reference to the chunks marked for deletion. Without the delay, index files (that are stale) on the gateways could refer to already deleted chunks leading to query failures.
It provides a short window of time in which to cancel chunk deletion in the case of a configuration mistake.

Marker files should be stored on a persistent disk to ensure that the chunks pending for deletion are processed even if the Compactor process restarts.

Note
Grafana Labs recommends running Compactor as a stateful deployment (StatefulSet when using Kubernetes) with a persistent storage for storing marker files.

Retention Configuration

This Compactor configuration example activates retention.

compactor:
  working_directory: /data/retention
  compaction_interval: 10m
  retention_enabled: true
  retention_delete_delay: 2h
  retention_delete_worker_count: 150
  delete_request_store: gcs
schema_config:
  configs:
    - from: "2020-07-31"
      index:
        period: 24h
        prefix: index_
      object_store: gcs
      schema: v13
      store: tsdb
storage_config:
  tsdb_shipper:
    active_index_directory: /data/index
    cache_location: /data/index_cache
  gcs:
    bucket_name: loki

Note
Retention is only available if the index period is 24h. Single store TSDB and single store BoltDB require 24h index period.

retention_enabled should be set to true. Without this, the Compactor will only compact tables.

delete_request_store should be set to configure the store for delete requests. This is required when retention is enabled.

working_directory is the directory where marked chunks and temporary tables will be saved.

compaction_interval dictates how often compaction and/or retention is applied. If the Compactor falls behind, compaction and/or retention occur as soon as possible.

retention_delete_delay is the delay after which the Compactor will delete marked chunks.

retention_delete_worker_count specifies the maximum quantity of goroutine workers instantiated to delete chunks.

Configuring the retention period

Retention period is configured within the limits_config configuration section.

There are two ways of setting retention policies:

retention_period which is applied globally for all log streams.
retention_stream which is only applied to log streams matching the selector.

Note
The minimum retention period is 24h.

This example configures global retention that applies to all tenants (unless overridden by configuring per-tenant overrides):

...
limits_config:
  retention_period: 744h
  retention_stream:
  - selector: '{namespace="dev"}'
    priority: 1
    period: 24h
  per_tenant_override_config: /etc/overrides.yaml
...

Note
You can only use label matchers in the selector field of a retention_stream definition. Arbitrary LogQL expressions are not supported.

Per tenant retention can be defined by configuring runtime overrides. For example:

overrides:
    "29":
        retention_period: 168h
        retention_stream:
        - selector: '{namespace="prod"}'
          priority: 2
          period: 336h
        - selector: '{container="loki"}'
          priority: 1
          period: 72h
    "30":
        retention_stream:
        - selector: '{container="nginx", level="debug"}'
          priority: 1
          period: 24h

Retention period for a given stream is decided based on the first match in this list:

If multiple per-tenant retention_stream selectors match the stream, retention period with the highest priority is picked.
If multiple global retention_stream selectors match the stream, retention period with the highest priority is picked. This value is not considered if per-tenant retention_stream is set.
If a per-tenant retention_period is specified, it will be applied.
The global retention_period will be applied if none of the above match.
If no global retention_period is specified, the default value of 0s is used, which means logs are kept indefinitely.

Note
The larger the priority value, the higher the priority.

Stream matching uses the same syntax as Prometheus label matching:

=: Select labels that are exactly equal to the provided string.
!=: Select labels that are not equal to the provided string.
=~: Select labels that regex-match the provided string.
!~: Select labels that do not regex-match the provided string.

The example configurations defined above will result in the following retention periods:

For tenant 29:
- Streams that have the namespace label prod will have a retention period of 336h (2 weeks), even if the container label is loki, since the priority of the prod rule is higher.
- Streams that have the container label loki but are not in the namespace prod will have a 72h retention period.
- For the rest of the streams in this tenant, per-tenant override retention_period value of 168h is applied.
For tenant 30:
- Streams that have the label nginx and level debug will have a retention period of 24h.
- For the rest of the streams in this tenant the global retention period of 744h, since there is no override specified.
All tenants except 29 and 30:
- Streams that have the namespace label dev will have a retention period of 24h hours.
- Streams except those with the namespace label dev will have the retention period of 744h.

Note
If you are a Grafana Cloud customer, you can use the config self-serve API to configure your tenant retention.

Table Manager (deprecated)

Retention through the Table Manager is achieved by relying on the object store TTL feature, and will work for both boltdb-shipper store and chunk/index stores.

In order to enable the retention support, the Table Manager needs to be configured to enable deletions and a retention period. Please refer to the table_manager section of the Loki configuration reference for all available options. Alternatively, the table-manager.retention-period and table-manager.retention-deletes-enabled command line flags can be used. The provided retention period needs to be a duration represented as a string that can be parsed using the Prometheus common model ParseDuration. Examples: 7d, 1w, 168h.

Warning
The retention period must be a multiple of the index and chunks table period, configured in the period_config block. See the Table Manager documentation for more information.

Note
To avoid querying of data beyond the retention period,max_query_lookback config in limits_config must be set to a value less than or equal to what is set in table_manager.retention_period.

When using S3 or GCS, the bucket storing the chunks needs to have the expiry policy set correctly. For more details check S3’s documentation or GCS’s documentation.

If you must delete ingested logs, you can delete old chunks in your object store. Note, however, that this only deletes the log content and keeps the label index intact; you will still be able to see related labels but will be unable to retrieve the deleted log content.

For further details on the Table Manager internals, refer to the Table Manager documentation.

Alternatively, if the BoltDB Shipper is configured for the index store, you can enable Log entry deletion to delete log entries from a specific stream.

Example Configuration

Example configuration with GCS with a 28 day retention:

schema_config:
  configs:
    - from: 2018-04-15
      store: tsdb
      object_store: gcs
      schema: v13
      index:
        prefix: loki_index_
        period: 24h

storage_config:
  tsdb_shipper:
    active_index_directory: /loki/index
    cache_location: /loki/index_cache
  gcs:
    bucket_name: GCS_BUCKET_NAME

limits_config:
  max_query_lookback: 672h # 28 days
  retention_period: 672h   # 28 days

compactor:
  working_directory: /data/retention
  delete_request_store: gcs
  retention_enabled: true

Log entry deletion

Thu, 09 Apr 2026 02:28:18 +0000

Log entry deletion

Grafana Loki supports the deletion of log entries from a specified stream. Log entries that fall within a specified time window and match an optional line filter are those that will be deleted.

Log entry deletion is supported only when TSDB or BoltDB shipper is configured as the index store.

The compactor component exposes REST endpoints that process delete requests. Hitting the endpoint specifies the streams and the time window. The deletion of the log entries takes place after a configurable cancellation time period expires.

Log entry deletion relies on configuration of the custom logs retention workflow as defined for the compactor. The compactor looks at unprocessed requests which are past their cancellation period to decide whether a chunk is to be deleted or not.

Configuration

Enable log entry deletion by setting retention_enabled to true in the compactor’s configuration and setting and deletion_mode to filter-only or filter-and-delete in the runtime config. delete_request_store also needs to be configured when retention is enabled to process delete requests, this determines the storage bucket that stores the delete requests.

Warning
Be very careful when enabling retention. It is strongly recommended that you also enable versioning on your objects in object storage to allow you to recover from accidental misconfiguration of a retention setting. If you want to enable deletion but do not want to enforce retention, configure the retention_period setting with a value of 0s.

Because it is a runtime configuration, deletion_mode can be set per-tenant, if desired.

With filter-only, log lines matching the query in the delete request are filtered out when querying Loki. They are not removed from storage. With filter-and-delete, log lines matching the query in the delete request are filtered out when querying Loki, and they are also removed from storage.

A delete request may be canceled within a configurable cancellation period. Set the delete_request_cancel_period in the compactor’s YAML configuration or on the command line when invoking Loki. Its default value is 24h.

As long as the compactor.retention_enabled setting is true, the API endpoints will be available. Afterwards, access to the deletion API can be enabled per tenant via the deletion_mode tenant override.

Horizontal scaling of Compactor

Thu, 09 Apr 2026 02:28:18 +0000

Introduction

Caution
Compactor horizontal scaling is an experimental feature. Use it with caution in your production environments.

Grafana Loki saw a major improvement in its operational complexity and cost-effectiveness with the introduction of object-storage-based indexes. This change also led to the addition of a singleton Compactor service, initially responsible only for index compaction. However, as new features like Custom Retention and Deletion of Logs with line filters were introduced, the Compactor’s responsibilities grew. With increasing scale and more demanding features, especially log deletion with line filters, the singleton Compactor began to show its scaling limits.

Now, you can run the Loki Compactor in a horizontally scalable mode. Since log deletion with line filters is the compactor’s most operationally intensive work, initially, this horizontally scalable architecture will be leveraged to speed up and distribute the workload specifically for its log deletion with line filters functionality.

How it works

We have introduced two new modes to the Compactor for operating in horizontally scalable mode:

Main:
- Runs all Compactor functions and distributes chunk processing for log line deletion with filters to the workers.
- Should be deployed as a singleton with access to a disk, like the current singleton deployment.
Worker:
- Connects to the Main Compactor over gRPC to get and execute the jobs.
- Multiple replicas can be deployed to achieve higher job processing throughput.
- Only needs access to Object Storage for reading/writing chunks.

Implementation details

Although, Horizontally Scalable Compactor currently only supports distributing the work of log line deletion with filters, in the future, we might add support for distributing more kind of work to the Workers. We are going to use the current functionality to see in detail some of the implementation details.

Working of Main mode

For Distribution of Chunk processing work from Main compactor, there are following core components involved:

Deletion Manifest Builder: The manifest builder works on a batch of delete request to discover all the chunks covered by them based on their labels filters and time range. Using the discovered chunks, it creates structured manifests and stores it to the Object Storage. Here is what comprises Deletion Manifest:
- Segments: Groups of up to 100K chunks per segment. Also acts as partition for chunks by Loki Tenant/Table.
- Manifest: Complete metadata about all segments and requests to process.
Job Builder: The job builder converts manifests into discrete jobs and manages their lifecycle:
- Job Creation: Breaks segments into jobs of up to 1K chunks each. Also includes line filters to apply on the chunks to remove the log lines requested for deletion.
- Progress Tracking: Monitors job completion and stops processing of manifest when any of the jobs fail.
- Job Queueing: Sends jobs to the Job Queue for processing.
- Storage Updates: Collects storage updates suggested by Workers and stores them to Object Storage for each segment.
- Post-processing Cleanup: Marks requests as processed and removes all the files from the storage.
Job Queue: The job queue manages job distribution and Worker-Job Builder communication.
- Job Distribution: Sends jobs to available workers via gRPC streaming.
- Retry Logic: Automatically retries failed or timed-out jobs until allowed number of attempts.
- Job Response: Sends the job processing response to Job Builder. Also notifies about failed jobs after running out of retries.

Working of Worker mode

Workers connect to the Main Compactor via gRPC to fetch and execute jobs, returning results on the same gRPC stream. It uses compactor_grpc_address under the common config to connect to the Main Compactor.

When handling Deletion jobs, a worker downloads the listed chunks, applies the specified filters to rebuild them without the filtered lines, and then provides a comprehensive storage update as job execution response. The storage update details which chunks to delete from Object Storage and which newly created chunks to index.

Sequence diagram

The sequence diagram depicts Deletion Manifest Builder, Job Builder and Job Queue as separate entities than the Main Compactor to show how the components are interlinked. In reality, all three components run within the Main Compactor.

Configuration

The horizontal_scaling_mode configuration option in the compactor controls the enablement of horizontally scalable compactor deployment. It supports setting the following modes:

disabled (default): Keeps the horizontal scaling mode disabled. Locally runs all the functions of the compactor.
main: Runs all functions of the compactor. Distributes work to workers where possible.
worker: Runs the compactor in worker mode, only working on jobs built by the main compactor.

Config for Main mode

To run Compactor in Main mode, the Horizontal Scaling Mode needs to be set to “main”:

compactor:
  # CLI flag: -compactor.horizontal-scaling-mode="main"
  horizontal_scaling_mode: "main"

Additionally, there are the below config options available for the Main compactor to configure some aspects of Job Building:

compactor:
  jobs_config:
    deletion:
      # Object storage path prefix for storing deletion manifests.
      # CLI flag: -compactor.jobs.deletion.deletion-manifest-store-prefix
      [deletion_manifest_store_prefix: <string> | default = "__deletion_manifest__/"]
      
      # Maximum time to wait for a job before considering it failed and retrying.
      # CLI flag: -compactor.jobs.deletion.timeout
      [timeout: <duration> | default = 15m]
      
      # Maximum number of times to retry a failed or timed out job.
      # CLI flag: -compactor.jobs.deletion.max-retries
      [max_retries: <int> | default = 3]

Config for Worker mode

To run Compactor in Worker mode, the Horizontal Scaling Mode needs to be set to “worker” and Main compactor’s GRPC address needs to be set:

common:
  compactor_grpc_address: <HOST>:<GRPC_PORT>
compactor:
  # CLI flag: -compactor.horizontal-scaling-mode="worker"
  horizontal_scaling_mode: "worker"

Additionally, there are the below config options available for the Worker to configure some aspects of Job execution:

compactor:
  jobs_config:
    deletion:
      # Maximum number of chunks to process concurrently in each worker.
      # CLI flag: -compactor.jobs.deletion.chunk-processing-concurrency
      [chunk_processing_concurrency: <int> | default = 3]

worker_config:
  # Number of sub-workers to run for concurrent processing of jobs. Setting it to 0
  # will run a subworker per available CPU core.
  # CLI flag: -compactor.worker.num-sub-workers
  [num_sub_workers: <int> | default = 0]

Legacy storage

Thu, 09 Apr 2026 02:28:18 +0000

Legacy storage

Warning
The concepts described on this page are considered legacy and pre-date the single store storage introduced in Loki 2.0. The usage of legacy storage for new installations is highly discouraged and documentation is meant for informational purposes in case of upgrade to a single store.

The chunk store is the Loki long-term data store, designed to support interactive querying and sustained writing without the need for background maintenance tasks. It consists of:

An index for the chunks. This index can be backed by:
A key-value (KV) store for the chunk data itself, which can be:

Note
Unlike the other core components of Loki, the chunk store is not a separate service, job, or process, but rather a library embedded in the two services that need to access Loki data: the ingester and querier.

The chunk store relies on a unified interface to the “NoSQL” stores (DynamoDB, Bigtable, and Cassandra) that can be used to back the chunk store index. This interface assumes that the index is a collection of entries keyed by:

A hash key. This is required for all reads and writes.
A range key. This is required for writes and can be omitted for reads, which can be queried by prefix or range.

The interface works somewhat differently across the supported databases:

DynamoDB supports range and hash keys natively. Index entries are thus modelled directly as DynamoDB entries, with the hash key as the distribution key and the range as the DynamoDB range key.
For Bigtable and Cassandra, index entries are modelled as individual column values. The hash key becomes the row key and the range key becomes the column key.

A set of schemas are used to map the matchers and label sets used on reads and writes to the chunk store into appropriate operations on the index. Schemas have been added as Loki has evolved, mainly in an attempt to better load balance writes and improve query performance.

Table manager

Thu, 09 Apr 2026 02:28:18 +0000

Table manager

Note
Table manager is only needed if you are using a multi-store backend. If you are using either TSDB (recommended), or BoltDB (deprecated) you do not need the Table Manager.

Grafana Loki supports storing indexes and chunks in table-based data storages. When such a storage type is used, multiple tables are created over the time: each table - also called periodic table - contains the data for a specific time range.

This design brings two main benefits:

Schema config changes: each table is bounded to a schema config and version, so that changes can be introduced over the time and multiple schema configs can coexist
Retention: the retention is implemented deleting an entire table, which allows to have fast delete operations

The Table Manager is a Loki component which takes care of creating a periodic table before its time period begins, and deleting it once its data time range exceeds the retention period.

The Table Manager supports the following backends:

Index store
- Single Store (boltdb-shipper)
- Amazon DynamoDB
- Google Bigtable
- Apache Cassandra
- BoltDB (primarily used for local environments)
Chunk store
- Filesystem (primarily used for local environments)

Loki does support the following backends for both index and chunk storage, but they are deprecated and will be removed in a future release:

The object storages - like Amazon S3 and Google Cloud Storage - supported by Loki to store chunks, are not managed by the Table Manager, and a custom bucket policy should be set to delete old data.

For detailed information on configuring the Table Manager, refer to the table_manager section in the Loki configuration document.

Tables and schema config

A periodic table stores the index or chunk data relative to a specific period of time. The duration of the time range of the data stored in a single table and its storage type is configured in the schema_config configuration block.

The schema_config can contain one or more configs. Each config, defines the storage used between the day set in from (in the format yyyy-mm-dd) and the next config, or “now” in the case of the last schema config entry.

This allows to have multiple non-overlapping schema configs over the time, in order to perform schema version upgrades or change storage settings (including changing the storage type).

The write path hits the table where the log entry timestamp falls into (usually the last table, except short periods close to the end of a table and the beginning of the next one), while the read path hits the tables containing data for the query time range.

Schema config example

For example, the following schema_config defines two configurations: the first one using the schema v10 and the current one using the v11.

The first config stores data between 2019-01-01 and 2019-04-14 (included), then a new config has been added - to upgrade the schema version to v11 - storing data using the v11 schema from 2019-04-15 on.

For each config, multiple tables are created, each one storing data for period time (168 hours = 7 days).

schema_config:
  configs:
    - from:   2019-01-01
      store:  dynamo
      schema: v10
      index:
        prefix: loki_
        period: 168h
    - from:   2019-04-15
      store:  dynamo
      schema: v11
      index:
        prefix: loki_
        period: 168h

Table creation

The Table Manager creates new tables slightly ahead of their start period, in order to make sure that the new table is ready once the current table end period is reached.

The creation_grace_period property - in the table_manager configuration block - defines how long before a table should be created.

Retention

The retention - managed by the Table Manager - is disabled by default, due to its destructive nature. You can enable the data retention explicitly enabling it in the configuration and setting a retention_period greater than zero:

table_manager:
  retention_deletes_enabled: true
  retention_period: 336h

The Table Manager implements the retention deleting the entire tables whose data exceeded the retention_period. This design allows to have fast delete operations, at the cost of having a retention granularity controlled by the table’s period.

Given each table contains data for period of time and that the entire table is deleted, the Table Manager keeps the last tables alive using this formula:

number_of_tables_to_keep = floor(retention_period / table_period) + 1

Note
It’s important to note that - due to the internal implementation - the table period and retention_period must be multiples of 24h in order to get the expected behavior.

For detailed information on configuring the retention, refer to the Loki Storage Retention documentation.

Active / inactive tables

A table can be active or inactive.

A table is considered active if the current time is within the range:

Table start period - creation_grace_period
Table end period + max chunk age (hardcoded to 12h)

Currently, the difference between an active and inactive table only applies to the DynamoDB storage settings: capacity mode (on-demand or provisioned), read/write capacity units and autoscaling.

DynamoDB	Active table	Inactive table
Capacity mode	`enable_ondemand_throughput_mode`	`enable_inactive_throughput_on_demand_mode`
Read capacity unit	`provisioned_read_throughput`	`inactive_read_throughput`
Write capacity unit	`provisioned_write_throughput`	`inactive_write_throughput`
Autoscaling	Enabled (if configured)	Always disabled

DynamoDB Provisioning

When configuring DynamoDB with the Table Manager, the default on-demand provisioning capacity units for reads are set to 300 and writes are set to 3000. The defaults can be overwritten:

table_manager:
  index_tables_provisioning:
    provisioned_write_throughput: 10
    provisioned_read_throughput: 10
  chunk_tables_provisioning:
    provisioned_write_throughput: 10
    provisioned_read_throughput: 10

If Table Manager is not automatically managing DynamoDB, old data cannot easily be erased and the index will grow indefinitely. Manual configurations should ensure that the primary index key is set to h (string) and the sort key is set to r (binary). The “period” attribute in the configuration YAML should be set to 0.

Table Manager deployment mode

The Table Manager can be executed in two ways:

Implicitly executed when Loki runs in monolithic mode (single process)
Explicitly executed when Loki runs in microservices mode

Monolithic mode

When Loki runs in monolithic mode, the Table Manager is also started as component of the entire stack.

Microservices mode

When Loki runs in microservices mode, the Table Manager should be started as separate service named table-manager.

You can check out a production grade deployment example at table-manager.libsonnet.

Manage storage on Grafana Labs

Single Store TSDB (tsdb)

Single Store TSDB (tsdb)

Example Configuration

Operations

Limits

Dynamic Query Sharding

Index Caching not required

Single Store BoltDB (boltdb-shipper)

Single Store BoltDB (boltdb-shipper)

Example Configuration

Operational Details

Ingesters

Queriers

Index Gateway

Write Deduplication disabled

Compactor

Delete Permissions

Filesystem object store

Filesystem object store

Pros

Cons

Scaling

Durability

High Availability

Storage schema

Storage schema

New Loki installs

Changing the schema

Schema configuration example

Write Ahead Log

Write Ahead Log

Disclaimer and WAL nuances

Backpressure

Metrics

Changes to deployment

Changes in lifecycle when WAL is enabled

Disk space requirements

Migrating from stateless deployments

How to scale up/down

Scale up

Scale down

Additional notes

Kubernetes hacking

Scaling Down Using /flush_shutdown Endpoint and Lifecycle Hook

Non-Kubernetes or baremetal deployments

Log retention

Log retention

Compactor

Retention Configuration

Configuring the retention period

Table Manager (deprecated)

Example Configuration

Log entry deletion

Log entry deletion

Configuration

Horizontal scaling of Compactor

Introduction

How it works

Implementation details

Working of Main mode

Working of Worker mode

Sequence diagram

Configuration

Config for Main mode

Config for Worker mode

Legacy storage

Legacy storage

Table manager

Table manager

Tables and schema config

Schema config example

Table creation

Retention

Active / inactive tables

DynamoDB Provisioning

Table Manager deployment mode

Monolithic mode

Microservices mode

Scaling Down Using `/flush_shutdown` Endpoint and Lifecycle Hook