It has been nearly 6 months since Loki 2.4 was released, and we’ve been busy making Loki better than ever. Here’s a summary of new enhancements and important fixes.
Features and enhancements
- Thanks to @bboreham for taking a deep dive on the Go
regexplibrary, and for creating a fork which greatly improves the performance of common regex uses cases in Loki.
- Binary operations are now significantly faster, taking full advantage of Loki’s parallelism.
- A new schema is available, which uses more path prefixes to avoid rate limits on S3.
- That same schema change was also added to the filesystem store, which avoids using one directory to store every chunk.
- A new capability for hedging requests to storage improves performance on highly parallelized queries. Refer to the hedging configuration under the
storage_configblock for more information.
- Promtail has several new ways to ingest logs:
- Promtail permits client-side global rate limiting and pipeline-configurable rate limiting.
For a full list of all changes please look at the CHANGELOG.
As always, please read the upgrade guide before upgrading Loki.
Changes to the config
The most likely impact many people will see is Loki failing to start because of a change in the YAML configuration for
split_queries_by_interval. It was previously possible to define this value in two places.
query_range: split_queries_by_interval: 10m
limits_config: split_queries_by_interval: 10m
In 2.5.0 it can only be defined in the
Loki will fail to start if you do not remove the
split_queries_by_interval configuration parameter from the
Additionally, it has a new default value of
30m rather than
The CLI flag is not changed and remains
More parallelism by default
We continue to move defaults in Loki to leverage parallelism in all configurations, including the single binary.
In v2.5, all queries will be split and sharded by default. This will likely result in increased memory and CPU usage for Loki processes during queries, if you didn’t previously have these values enabled.
Loki 2.5 includes code we added to report anonymous usage statistics back to Grafana Labs, an issue was created to outline the intent, and what went into the final implementation can be seen here in the source.
Usage reporting helps provide anonymous information on how people use Loki and what the Loki team should focus on for features and documentation. No private information is collected, and all reports are completely anonymous.
If possible, we ask you to leave the usage reporting feature enabled and help us understand more about Loki! We are also working to figure out how we can share this info with the community so everyone can watch Loki grow.
If you would rather not participate in usage stats reporting, the feature can be disabled in config
analytics: reporting_enabled: false
2.5.0 bug fixes
V2.5.0 fixes numerous bugs. The CHANGELOG has the complete list.
A summary of some of the more important fixes:
- Query correctness:
- PR 5474 cyriltovena: Disable sharding of count/avg when labels are mutated
- PR 5444 cyriltovena: Do not insert missing point when sharding
- PR 5423 cyriltovena: Correctly sets hash value for headblock iterator
- PR 5289 cyriltovena: Fixes log deduplication when mutating Labels using LogQL
- PR 5006 sandeepsukhani: Fix splitting of queries when step is larger than split interval
- PR 5413 owen-d: A deadlock in the Azure blob client
- PR 5334 kavirajk: A fix in live tailing that could lead to exploding memory usage
- PR 5144 owen-d: A fix for the ruler when trying to use Basic Auth for remote write
- Query cancellation:
- PR 4741 sandeepsukhani: An issue where retention didn’t cleanup the index completely