This is documentation for the next version of Loki. For the latest stable release, go to the latest version.
Single Store TSDB (tsdb)
Starting with Loki v2.8, TSDB is the recommended Loki index. It is heavily inspired by the Prometheus’s TSDB sub-project. For a deeper explanation you can read Loki maintainer Owen’s blog post. The short version is that this new index is more efficient, faster, and more scalable. It also resides in object storage like the boltdb-shipper index which preceded it.
Example Configuration
To get started using TSDB, add the following configurations to your config.yaml
:
Operations
Limits
We’ve added a user per-tenant limit called tsdb_max_query_parallelism
in the limits_config
. This functions the same as the prior max_query_parallelism
configuration but applies to tsdb queries instead. Since the TSDB index will create many more smaller queries compared to the other index types before it, we’ve added a separate configuration so they can coexist. This is helpful when transitioning between index types. The default parallelism is 128
which should work well for most cases, but you can extend it globally in the limits_config
or per-tenant in the overrides
file as needed.
Dynamic Query Sharding
Previously we would statically shard queries based on the index row shards configured here. TSDB does Dynamic Query Sharding based on how much data a query is going to be processing. We additionally store size(KB) and number of lines for each chunk in the TSDB index which is then used by the Query Frontend for planning the query. Based on our experience from operating many Loki clusters, we have configured TSDB to aim for processing 300-600 MBs of data per query shard. This means with TSDB we will be running more, smaller queries.
Index Caching not required
TSDB is a compact and optimized format. Loki does not currently use an index cache for TSDB. If you are already using Loki with other index types, it is recommended to keep the index caching until all of your existing data falls out of retention) or your configured max_query_lookback
under limits_config. After that, we suggest running without an index cache (it isn’t used in TSDB).