Important: This documentation is about an older version. It's relevant only to the release noted, many of the features and functions have been updated or replaced. Please view the current version.

Operations

Storage

Retention

Open source

Loki Storage Retention

Retention in Loki is achieved through the Table Manager. In order to enable the retention support, the Table Manager needs to be configured to enable deletions and a retention period. Please refer to the table_manager_config section of the Loki configuration reference for all available options. Alternatively, the table-manager.retention-period and table-manager.retention-deletes-enabled command line flags can be used. The provided retention period needs to be a duration represented as a string that can be parsed using Go’s time.Duration.

WARNING: The retention period must be a multiple of the index and chunks table period, configured in the period_config block. See the Table Manager documentation for more information.

NOTE: To avoid querying of data beyond the retention period, max_look_back_period config in chunk_store_config must be set to a value less than or equal to what is set in table_manager.retention_period.

When using S3 or GCS, the bucket storing the chunks needs to have the expiry policy set correctly. For more details check S3’s documentation or GCS’s documentation.

Currently, the retention policy can only be set globally. A per-tenant retention policy with an API to delete ingested logs is still under development.

Since a design goal of Loki is to make storing logs cheap, a volume-based deletion API is deprioritized. Until this feature is released, if you suddenly must delete ingested logs, you can delete old chunks in your object store. Note, however, that this only deletes the log content and keeps the label index intact; you will still be able to see related labels but will be unable to retrieve the deleted log content.

For further details on the Table Manager internals, refer to the Table Manager documentation.

Example Configuration

Example configuration with GCS with a 28 day retention:

schema_config:
  configs:
  - from: 2018-04-15
    store: bigtable
    object_store: gcs
    schema: v11
    index:
      prefix: loki_index_
      period: 168h

storage_config:
  bigtable:
    instance: BIGTABLE_INSTANCE
    project: BIGTABLE_PROJECT
  gcs:
    bucket_name: GCS_BUCKET_NAME

chunk_store_config:
  max_look_back_period: 672h

table_manager:
  retention_deletes_enabled: true
  retention_period: 672h