Documentation Grafana Loki Release notes v3.3

v3.3

Grafana Labs and the Loki team are excited to announce the release of Loki 3.3. Here’s a summary of new enhancements and important fixes.

For a full list of all changes and fixes, refer to the CHANGELOG.

Features and enhancements

Key features in Loki 3.3.0 include the following:

• Query acceleration with Bloom filters (experimental): The Bloom filters feature introduced in Loki 3.0 remains experimental. There has been a major shift in how Loki leverages bloom filters — pivoting from free-text search to harnessing the power of structured metadata (#14061) (a2fbaa8). This change leverages the growing adoption of OpenTelemetry in observability stacks. The Bloom Compactor component has been removed and replaced by the Bloom Planner and Bloom Builder components (#13997) (bf60455) and blooms has a new block schema (V3) (#14038) (5395daf) which is a breaking change.

• Explore Logs: Starting with Grafana v11.3, the plugin for the Explore Logs app is installed in Grafana by default. This release includes enhancements to improve pattern ingester tracing (#14707) (80aec25), to apply patterns line length limit to json message keys (#14296) (41fafd8), add configuration to set max-allowed-line-length in the pattern ingester (#14070) (0780456) and move metric aggregation to a per-tenant configuration (#14709) (c1fde26).

• helm: Multiple improvements to the Helm charts, including: Updating the chart with Loki version 3.2.0 (#14281) (11b92ee), replacing bloom compactor with bloom planner and builder (#14003) (08e61ca), adding the ability to configure client_max_body_size (#12924) (809a024), adding additional service annotations for components in distributed mode (#14131) (5978f13), adding configurable extraEnvFrom to admin-api and enterprise-gateway (#14533) (5d78a3a), adding kubeVersionOverride for Helm chart (#14434) (0935d77), adding persistence option to memcached on Helm chart (#13619) (ef1df0e), adding tolerations to pattern-ingester statefulset (#13605) (09530c0), allowing setting node attributes to tokengen and provisioner (#14311) (c708ae6), updating Loki Helm chart for restricted environments (#14440) (adc7538), and supporting ruler sidecar in singleBinary mode (#13572) (684baf7).

• mixins: Multiple improvements to the Loki mixins, including: adding a missing cluster label to mixins (#12870) (547ca70), adding support for partition ingester in dashboards (#14498) (70deebf), adding a Loki compaction not successful alert (#14239) (da04f50), allowing overriding of some labels by parameterizing mixin recording/alert rules (#11495) (f1425b6), allowing disabling of bloom dashboards (#14177) (ce2e6d5), allowing hiding of unused rows in loki-operational (#13646) (3aa4f22), and merging read-resources and write-resources dashboards for SSD into one dashboard that also includes the backend-resources(#13471) (45b8719).

• operator: Multiple updates to the Operator, including: Adding support for Loki OTLP limits configuration (#13446) (d02f435), providing default OTLP attribute configuration (#14410) (1b52387), adding a User-guide for OTLP configuration (#14620) (27b4071), updating the Loki operand to v3.1.1 (#14042) (7ae1588), updating the Loki operand to v3.2.1 (#14526) (5e970e5), renaming the Loki API go module (#14568) (976d8ab), introducing 1x.pico size (#14407) (57de81d), migrating the project layout to kubebuilder go/v4 (#14447) (dbb3b6e), declaring feature FIPS support for OpenShift only (#14308) (720c303).

• docs: New videos and tutorials, including a New Cloud deployment guide for Deploying Loki on AWS, a new Fluent Bit tutorial and a Getting Started with OpenTelemetry Collector tutorial. Also there are improvements to the Query documentation, including a new Query Best Practices topic, a reorganized Query Template, and a revised LogQL Simulator topic.

Other improvements include the following:

• api: Added endpoint to return detected field values (#14350) (7983f94).
• blooms: Add disk-backed queue for the bloom-planner (backport k227) (#14927) (1f6828b).
• blooms: Only write key and key=value to blooms (#14686) (3af0004).
• blooms: Do not add empty blooms to offsets (#14577) (51c42e8).
• blooms: Extract task computing into a strategy interface (#13690) (ab5e6ea).
• blooms: New bloom planning using chunk size TSDB stats (#14547) (673ede1).
• blooms: Remove compression of .tar archived bloom blocks (#14159) (cdf084f).
• compactor: Add backoff mechanism to the retention process (#14182) (3136880).
• distributor: Track discarded data by usageTracker (#14081) (c65721e).
• distributor: Add ability to log stream selectors before service name detection (#14154) (d7ff426).
• distributors: Use a pool of worker to push to ingesters. (#14245) (f80d68a).
• fluentd: Support custom http headers (#14299) (e59035e).
• ingester: Implement owned streams calculation using Partition Ring (#14282) (3c36ba9).
• ingester: Implement partition shuffle sharding for ingester (#14304) (1a4436c).
• jsonnet: Allow to name prefix zoned ingesters (#14260) (fac3177).
• labels: Add app_name as a service label (#13660) (f2a16f4).
• logcli: Add gzip compression option (#14598) (4d3f9f5).
• logql: Introduce shardable probabilistic topk for instant queries. (backport k227) (#14765) (02eb024).
• logql: Micro-optimizations for IP filter (#14072) (c5083c7).
• logql: Add query user and query source to “executing query” log lines (#14320) (4d69929).
• promtail: Add structured metadata to the promtail push API (#14153) (66cffcb).
• querier: Move detected field logic to query frontend (#14212) (36ace66).
• ruler: Enable ruler store that uses clients from thanos-io/objstore pkg (#11713) (8bca2e7).
• storage: Add retries for s3 ObjectExists calls (#14062) (73cbbb0).
• storage: Implement IsRetryableErr for S3ObjectClient (#14174) (fc90a63).
• storage: Introduce new ObjectExistsWithSize API to (#14268) (ac422b3).
• storage: AWS backend using thanos.io/objstore (#11221) (b872246).
• storage: Azure backend using thanos.io/objstore (#11315) (5824e3d).
• storage: GCS backend using thanos.io/objstore (#11132) (c059ace).
• structured metadata: Include structured_metadata size while asserting rate limit (#14571) (a962edb).

Deprecations

One of the focuses of Loki 3.0 was cleaning up unused code and old features that had been previously deprecated but not removed. Loki 3.0 removed a number of previous deprecations and introduces some new deprecations. Some of the main areas with changes include:

• Deprecated storage options including the deprecation of the BoltDB store.

• Deprecated configuration options.

• API endpoint deprecations.

To learn more about breaking changes in this release, refer to the Upgrade guide.

Upgrade Considerations

For important upgrade guidance, refer to the Upgrade Guide.

• BREAKING CHANGE - blooms: Introduce a new block schema (V3) (#14038).
• BREAKING CHANGE - blooms: Index structured metadata into blooms (#14061).
• BREAKING CHANGE - operator: Migrate project layout to kubebuilder go/v4 (#14447).
• BREAKING CHANGE - operator: Rename Loki API go module (#14568).
• BREAKING CHANGE - operator: Provide default OTLP attribute configuration (#14410).

Bug fixes

3.3.0 (2024-11-19)

• blooms: Add tenant limits as dependency to pattern ingester (#14665) (31eea90).
• blooms: Check length of tasks before accessing first element in slice (#14634) (601f549).
• blooms: Copy chunks from ForSeries (backport k227) (#14864) (d10f79c).
• blooms: Do not restart builders when planner disconnects (backport k227) (#14922) (213e8ee).
• blooms: Exclude label filters where label name is part of the series labels. (#14661) (d1668f6).
• blooms: Fix panic in initialization of the bloom planner and builder (#14110) (8307c42).
• blooms: Fix strategy logger and add task test (backport k227) (#14921) (dc36a1e).
• blooms: Fix tenants slice on loadTenantTables (backport k227) (#14901) (540dd5a).
• blooms: Skip multi-tenant TSDBs during bloom planning (backport k227) (#14888) (631cff3).
• blooms: Improve docs for min and max table offsets (backport k227) (#14929) (3161fdc).
• blooms: Fix lint errors (#14574) (99ef900).
• build: Use Debian Bullseye base image for build image (#14368) (3beb8ff).
• canary: Reconnect immediately upon tail max duration (#14287) (9267ee3).
• ci: Fix Publish Rendered Helm Chart Diff workflow (#14365) (6de6420).
• ci: Update helm diff rendering workflow (#14424) (916e511).
• config: Copy Alibaba and IBM object storage configuration from common (#14297) (59ff1ec).
• config: Migrate renovate configuration (#14646) (a67d8ef).
• deps: Update aws-sdk-go-v2 monorepo (#13986) (6f49123).
• deps: update aws-sdk-go-v2 monorepo (#14742) (53a1ab7).
• deps: update github.com/grafana/dskit digest to 687ec48 (#14395) (c2f38e1).
• deps: update github.com/grafana/dskit digest to 7c41a40 (#14277) (f39cdbd).
• deps: update github.com/grafana/dskit digest to 931a021 (#14032) (7c18642).
• deps: update github.com/grafana/dskit digest to b69ac1b (#14355) (9d7a6ea).
• deps: update github.com/grafana/dskit digest to f52de24 (#14319) (a4f3edf).
• deps: update github.com/twmb/franz-go/pkg/kfake digest to cea7aa5 (#14590) (688c42a).
• deps: update k8s.io/utils digest to 702e33f (#14033) (b7eecc7).
• deps: Update module cloud.google.com/go/bigtable to v1.33.0 (#14580) (a0920ed).
• deps: Update module cloud.google.com/go/pubsub to v1.45.0 (#14361) (4351238).
• deps: Update module cloud.google.com/go/pubsub to v1.45.1 (#14650) (f173708).
• deps: Update module cloud.google.com/go/storage to v1.46.0 (#14744) (8e45116).
• deps: Update module github.com/alicebob/miniredis/v2 to v2.33.0 (#14721) (7bfda25).
• deps: Update module github.com/aws/aws-sdk-go to v1.55.5 (#14715) (03f0f5a).
• deps: Update module github.com/axiomhq/hyperloglog to v0.2.0 (#14722) (0167b22).
• deps: Update module github.com/baidubce/bce-sdk-go to v0.9.189 (#14044) (7fb34b4).
• deps: Update module github.com/baidubce/bce-sdk-go to v0.9.192 (#14337) (6f7cae2).
• deps: Update module github.com/baidubce/bce-sdk-go to v0.9.196 (#14651) (478085a).
• deps: Update module github.com/baidubce/bce-sdk-go to v0.9.197 (#14682) (b898294).
• deps: Update module github.com/coder/quartz to v0.1.2 (#14652) (7459e07).
• deps: Update module github.com/felixge/fgprof to v0.9.5 (#14338) (a2ad3aa).
• deps: Update module github.com/fsouza/fake-gcs-server to v1.50.2 (#14313) (275c97c).
• deps: Update module github.com/hashicorp/raft to v1.7.1 (#14005) (e9cec1d).
• deps: Update module github.com/ibm/go-sdk-core/v5 to v5.17.5 (#14045) (677d217).
• deps: Update module github.com/ibm/go-sdk-core/v5 to v5.18.1 (#14716) (8395acd).
• deps: Update module github.com/ibm/ibm-cos-sdk-go to v1.11.1 (#14342) (aa82a7c).
• deps: Update module github.com/klauspost/compress to v1.17.10 (#14352) (e23c5ed).
• deps: Update module github.com/minio/minio-go/v7 to v7.0.76 (#14006) (51f9376).
• deps: Update module github.com/minio/minio-go/v7 to v7.0.77 (#14353) (d0e3ef7).
• deps: Update module github.com/minio/minio-go/v7 to v7.0.80 (#14654) (eec2513).
• deps: Update module github.com/ncw/swift/v2 to v2.0.3 (#14356) (c843288).
• deps: Update module github.com/prometheus/client_golang to v1.20.5 (#14655) (e12f843).
• deps: Update module github.com/schollz/progressbar/v3 to v3.17.0 (#14720) (4419d0f).
• deps: Update module github.com/shirou/gopsutil/v4 to v4.24.10 (#14719) (3280376).
• deps: Update module github.com/shirou/gopsutil/v4 to v4.24.9 (#14357) (c8e6a9d).
• deps: Update module github.com/shopify/sarama to v1.43.3 (#14059) (1cf4813).
• deps: Update module github.com/spf13/afero to v1.11.0 (#14060) (bbbd82b).
• deps: Update module go.etcd.io/bbolt to v1.3.11 (#14358) (b7bccfc).
• deps: Update module golang.org/x/net to v0.29.0 (#14341) (1b6b9da).
• deps: Downgrade grpc to fix regression (#14065) (8c38d46).
• deps: Fix missing dep PartitionRing for Ingester (#14292) (6354ded).
• deps: nix build, downgrade toolchain to go1.23.1 (#14442) (26dfd62).
• deps: Revert “fix(deps): Update module github.com/shirou/gopsutil/v4 to v4.24.9 (#14357)” (#14437) (d53955b).
• detected_fields: Return parsed labels when parsers are passed (#14047) (aa1ac99).
• detected_fields: Always return empty array as null (#14112) (93009d4).
• docs: Correct OTLP documentation typo (#14602) (063c590).
• distributor: Always write detected_level when enabled, even if unknown (#14464) (41c6b6c).
• distributor: Validate partition ring is kafka is enabled (#14303) (8438d41).
• distributor: Level detection for warning level (#14444) (242a852).
• explore logs: Correctly create logfmt string (#14124) (63e84b4).
• explore logs: Allow any level for aggregated metrics (#14255) (c001a1d).
• helm: Add missing loki.storage.azure.chunkDelimiter parameter to Helm chart (#14011) (08c70cc).
• helm: Check for rbac.namespaced condition before creating roles (#14201) (3f47f09).
• helm: Fix persistence configuration for Memcached (#14049) (ee6e1cf).
• helm: Fix wrong port name referenced for ingress NetworkPolicy (#12907) (963a25b).
• helm: Various fixes and enhancements for bloom components (#14128) (dc0cbd4).
• ingester: Report PSRL error message correctly (#14187) (a475153).
• ingester: Wait for OwnedStreams service in Ingester startup (#14208) (a4aee4f).
• lambda-promtail: Add s3 principal to iam policy attached to sqs in lambda-promtail terraform code (#14619) (db0889e).
• logcli: Check for errors before checking for exists when fetching data (backport k227) (#14906) (31b2a63).
• logcli: Create new tail response for every line (#14525) (bcfd0d1).
• logql: Fix panic in json parsing when using empty array index (#14393) (833bf0d).
• logql: Update JSONExpressionParser to not unescape extracted values if it is JSON object. (#14499) (08b1a90).
• logql: Allow rename of structured metadata labels (#13955) (2d4792a).
• logql: Renamed label should use ParsedLabel category (#14515) (82fb2f0).
• logql: Skipping label if it contains special symbol (#14068) (55e374e).
• mixin: Remove pod label from disk usage aggregation (#14180) (5d45c96).
• mixins: Fix loki-resources-overview panel layout (#14178) (8f54ec6).
• mixins: Add backend path section in loki-operational for single scalable deployment (#13023) (16881ab).
• mixins: Disk space utilization panels with latest KSM versions (#13486) (0ea7431).
• mixins: Retention dashboards fix metric name (#14617) (c762b9b).
• oltp: Enable service detection for otlp endpoint (#14036) (4f962ef).
• operator: Add 1x.pico OpenShift UI dropdown menu (#14660) (4687f37).
• operator: Add missing groupBy label for all rules on OpenShift (#14279) (ce7b2e8).
• operator: Correctly ignore again BlotDB dashboards (#14587) (4879d10).
• operator: Disable automatic discovery of service name (#14506) (3834c74).
• operator: Disable log level discovery for OpenShift tenancy modes (#14613) (5034d34).
• operator: Fix building the size-calculator image (#14573) (a79b8fe).
• operator: Fix make build target for size-calculator (#14551) (e727187).
• operator: Move OTLP attribute for statefulset name to stream labels (#14630) (5df3594).
• operator: Use empty initialized pod status map when no pods (#14314) (6f533ed).
• operator: Update renovate ignore for operator API with new module path (#14581) (c9b2907).
• pattern: Fixes latency metric namespace for tee to pattern (#14241) (ae955ed).
• promtail: Fix configuration unmarshalling (#14408) (a05431f).
• promtail: Fix parser for azureeventhubs message without time field (#14218) (2e62abb).
• promtail: Validate scrape_config job name, do not allow duplicate job names (#13719) (f2d3499).
• querier: Propagate query stats from quantile & topk queries (#13831) (78b275b).
• querier: Correct _extracted logic in detected fields (#14064) (1b3ba53).
• querier: Timestamp. Time should be called with milliseconds (#14196) (f8d9143).
• querier: Report correct status code for metric and log queries in metrics.go (#12102) (900751c).
• query: Fix bug in query result marshaling for invalid utf8 characters (#14585) (f411a07).
• Remove usage of unsafe string in label adapter unmarshall (#14216) (758364c).
• Rename misspelled filename (#14237) (cf1d4a3).
• sharding: Apply offset to both from and through in shard request (#14256) (17c472d).
• storage: Add additional validation for timeout while retrieving headers (#14217) (8322e51).
• storage: Do not retain span logger created with index set initialized at query time (#14027) (4e41744).
• storage: Expand matching for additional variations (#14221) (71d7291).
• storage: Disable client retries when congestion control is enabled in S3(#14588) (cff9f43).
• storage: Have GetObject check for canceled context (#14420) (5f325aa).
• storage: Transform ObjectExistsWithSize into GetAttributes (#14329) (2f56f50).
• storage: Update AWS storage timeout error for Go 1.23 behavior (#14226) (a4642b5).