Open source


Grafana Labs is excited to announce the release of Loki 2.7. Here’s a summary of new enhancements and important fixes:

Features and enhancements

  • New Internal Server and TLS TLS can now be configured everywhere and there is an internal server to allow ring, memberlist, and readiness handlers to continue to work without TLS.
  • Better Support for Azure Blob Storage thanks to the ability to use Azure’s Service Principal Credentials.
  • Logs can now be pushed from the Loki canary so you don’t have to rely on a scraping service to use the canary.
  • Additional label_format fields __timestamp__ and __line__.
  • fifocache has been renamed The in-memory fifocache has been renamed to embedded-cache. Check upgrade guide for more details
  • New HTTP endpoint for Ingester shutdown that will also delete the ring token.
  • Faster label queries thanks to new parallization.
  • Introducing Stream Sharding an experimental new feature to help deal with very large streams.
  • Promtail
    • support for max stream limit
    • config reload endpoint / signal
    • compressed file support
    • lambda-promtail now supports Kinesis data stream events
    • matches for the journal reader
    • basic tracing support

For a full list of all, look at the CHANGELOG.

Upgrade Considerations

As always, please read the upgrade guide before upgrading Loki.

Bug fixes

2.7.5 (2023-03-28)

  • Flush buffered logger on exit: this makes sure logs are printed if Loki crashes on startup.

2.7.4 (2023-02-24)

  • Fixed different streams for cri tags ending on the same stream.
  • Fixed the userdata field (from Windows Event Log) being scraped incorrectly.
  • Fixed vector() function producing wrong timestamp.
  • Fixed behavior for overlapping chunks with multiple stores.
  • Fixed logs results caching causing query-frontend to return logs outside of query window.
  • Fixed panics when:
    • /scheduler/ring endpoint is requested with scheduler ring disabled.
    • LogQL clones a specific query.
    • Promtail deals with invalid calls to Details().

2.7.3 (2023-02-01)

  • Fixed a bug in compactor that caused divide-by-zero panics when startTime and endTime of a delete request were equal.
  • Fixed the output of the --version command that showed an incorrect version information.

2.7.2 (2023-01-25)

  • Fixed bug in validation of pattern and regexp parsers where missing or empty parameters for these parsers caused panics.
  • Fixed bugs in processing delete requests with line filters:
    • Whole chunks covered by delete requests with a line filter were incorrectly skipped.
    • Chunks partially covered by delete requests with a line filter were not properly processed. The part that was not covered by the delete request was rewritten with a line filter, while the part that was covered by the delete request was completely deleted.
  • Fixed bug in multi-tenant querying that caused HTTP 400 responses when multiple tenants where used in X-Scope-OrgID header like so tenant-a|tenant-b.
  • Upgraded Go build version and Docker container base images to 1.19.5 to mitigate GO-2022-1144 vulnerability.

2.7.1 (2022-12-09)

  • Add single compactor http client for delete and gennumber clients. This fixes a bug caused by the accidental introduction of different HTTP clients for compactor and gennumber operations that resulted in 404s when only the gennumber middlewares were enabled:

    level=error ts=2022-10-18T14:13:41.598649348Z caller=delete_requests_client.go:211 msg="error getting delete requests from the store" err="unexpected status code: 404"
    ts=2022-10-18T14:13:41.598697295Z caller=spanlogger.go:80 user=application level=error msg="failed loading deletes for user" err="unexpected status code: 404"
  • Change default deletion mode to filter-only on account of the bug (see below).

Known Issues

  • There is a bug with the experimental deletion mode in 2.6.x and 2.7.x that in some cases can cause the wrong chunks to be deleted. As a result we have changed the default deletion mode to filter-only until this bug has been addressed.