Documentation Grafana Loki Set up Upgrade

Upgrade Loki

Every attempt is made to keep Grafana Loki backwards compatible, such that upgrades should be low risk and low friction.

Unfortunately Loki is software and software is hard and sometimes we are forced to make decisions between ease of use and ease of maintenance.

If we have any expectation of difficulty upgrading, we will document it here.

As more versions are released it becomes more likely unexpected problems arise moving between multiple versions at once. If possible try to stay current and do sequential updates. If you want to skip versions, try it in a development environment before attempting to upgrade production.

Checking for config changes

Using docker you can check changes between 2 versions of Loki with a command like this:

Bash Copy

export OLD_LOKI=2.9.4
export NEW_LOKI=3.0.0
export CONFIG_FILE=local-config.yaml
diff --color=always --side-by-side <(docker run --rm -t -v "${PWD}":/config grafana/loki:${OLD_LOKI} -config.file=/etc/loki/${CONFIG_FILE} -print-config-stderr 2>&1 | sed '/Starting Loki/q' | tr -d '\r') <(docker run --rm -t -v "${PWD}":/config grafana/loki:${NEW_LOKI} -config.file=/etc/loki/${CONFIG_FILE} -print-config-stderr 2>&1 | sed '/Starting Loki/q' | tr -d '\r') | less -R

The tr -d '\r' is likely not necessary for most people, seems like WSL2 was sneaking in some windows newline characters…

The output is incredibly verbose as it shows the entire internal config struct used to run Loki, you can play around with the diff command if you prefer to only show changes or a different style output.

Main / Unreleased

Distributor Max Receive Limits for uncompressed bytes

The next Loki release introduces a new configuration option (i.e. -distibutor.max-recv-msg-size) for the distributors to control the max receive size of uncompressed stream data. The new options’s default value is set to 100MB.

Supported clients should check the configuration options for max send message size if applicable.

Helm Chart Upgrades

Migrating from the Loki Repository Helm Chart to the Community Helm Chart

If you are upgrading from the Helm chart previously hosted in the Loki repository (chart version 6.x) to the Grafana Community Helm chart (chart version 18.x), refer to the dedicated migration guide: Upgrade to the Community Helm chart.

Helm Chart 6.50.0 - Respect the global registry in the sidecar image

If you prefixed the sidecar container with a private registry (sidecar.image.repository), this is no longer necessary and is deprecated as the global registry is used starting with Helm chart 6.46.1. Therefore please use global.imageRegistry or alternatively, sidecar.image.registry for more fine-grained control.

Helm Chart 6.50.0 - Uniform naming for image digest also in the sidecar image

For most images used in the helm chart, a .digest is available to pin an image to a specific hash. The sidecar images diverges from this convention by introducing a .tag. Starting with Helm chart 6.46.1, the .tag is deprecated and .digest should be used.

Helm Chart 6.46.0 - Default service account name change

Starting with Helm chart 6.46.0 (#19590), when serviceAccount.create is true and serviceAccount.name is not set, the default service account name is now derived from the chart’s fullname template instead of the chart name. For example:

If an external identity, such as an AWS IAM role through EKS Pod Identity or IRSA, is bound to the previous service account name, your pods lose access to that identity after the upgrade. For object-storage backends, this typically results in an outage because Loki components can no longer read or write to the bucket.

Recommended action:

To preserve the previous behavior and avoid the rename, set the service account name explicitly in your values.yaml:

YAML Copy

serviceAccount:
  name: enterprise-logs   # use "loki" for open source deployments

Setting serviceAccount.name explicitly is forward-compatible and works on both pre- and post-6.46.0 chart versions, so it is also the recommended setting going forward if you want the service account name to be independent of the Helm release name.

If you have already upgraded and your pods have lost cloud-provider IAM access, you have two options:

• Set serviceAccount.name to the previous default (for example, enterprise-logs) and run helm upgrade again. The previously bound external identity will resume working.
• Update the external identity binding (for example, the EKS Pod Identity association or the IRSA trust policy) to reference the new service account name.

Helm Chart 6.34.0 - Zone-aware Ingester Breaking Change

If you are using zone-aware ingesters (ingester.zoneAwareReplication.enabled: true), upgrading to Helm chart 6.34.0 requires manual StatefulSet deletion before the upgrade. This is due to a fix for the serviceName field in zone-aware ingester StatefulSets, which is an immutable field in Kubernetes.

For detailed upgrade instructions, see: Helm Chart 6.x Upgrade Guide - Zone-aware Ingester Breaking Change

Key points:

• Only affects deployments with ingester.zoneAwareReplication.enabled: true
• Requires manual StatefulSet deletion with --cascade=orphan
• No data loss - PersistentVolumeClaims and data are preserved
• New StatefulSets will be created with correct service references

3.6.0

Loki 3.6.0

Upgraded AWS SDK to v2

Loki uses the official AWS SDK for configuring and communication with S3 object storage. Version 1 of the SDK reached its end of life on 31st, 2025, and therefore had to be replaced with Version 2. While the user-facing configuration in Loki did not change, internal functionality of the object store client did change, without affecting functionality of Loki.

Please refer to the full release notes of v2 https://github.com/aws/aws-sdk-go-v2/releases/tag/release-2025-01-15 for further information and whether you may be impacted by any of the changes.

3.5.0

Loki 3.5.8

Removal of BusyBox Shell from Docker Images

Starting in Loki version 3.5.8, the busybox utility was removed from the official Loki Docker images. This means that shell utilities like /bin/sh are no longer available inside the container by default.

Impact

• You cannot exec into the Loki container to use a shell as before. Commands like kubectl exec -it podname -- sh or docker exec -it containername sh will fail because /bin/sh does not exist in the image.
• Common utilities provided by BusyBox (e.g., ls, cat, ps) are also not available inside the container.

Why was BusyBox removed?

Removing BusyBox addresses the following CVEs:

• CVE-2023-42364
• CVE-2023-42365
• CVE-2023-42363
• CVE-2023-42366
• CVE-2025-46394
• CVE-2024-58251

How do I troubleshoot or inspect a running Loki container now?

If you need to debug or inspect a Loki container:

1. Use ephemeral containers: Kubernetes allows you to attach an ephemeral container with a shell to a running Pod for debugging (if your cluster supports it).
   • Example:
     Copy

     kubectl debug -it <pod-name> --image=busybox --target=<container-name>

2. Copy files out/in instead of shelling in: Use kubectl cp or docker cp to move logs or config files for inspection.
3. Include shell utilities in your own derived image: If your operational process requires a shell, you can build a custom Docker image based on Loki and add BusyBox or another shell to it (not recommended for production).
   • Example Dockerfile snippet:
     Copy

     FROM grafana/loki:3.5.8
     USER root
     RUN apk add --no-cache busybox
     USER 10001

3.4.0

Loki 3.4.0

New Object Storage Clients

Loki release 3.4.0 introduces new object storage clients based on the Thanos Object Storage Client Go module, this is an opt-in feature. In a future release, this will become the default way of configuring storage and the existing storage clients will be deprecated.

The new storage configuration deviates from the existing format. Refer to the Thanos storage configuration reference to view the complete list of supported storage providers and their configuration options.

The documentation now also includes a migration guide and configuration examples for using Thanos-based storage clients.

3.3.0

Loki 3.3.0

Experimental Bloom Filters

With Loki 3.3.0, the bloom block format changed and any previously created block is incompatible with the new format. Before upgrading, we recommend deleting all the existing bloom blocks in the object store. We store bloom blocks and metas inside the bloom path in the configured object store. To get rid of all the bloom blocks, delete all the objects inside the bloom path in the object store.

3.2.0

Loki 3.2.0

HTTP API

The API endpoint for instant queries /api/v1/query now returns a HTTP status 400 (Bad Request) when the provided query parameter contains a log selector query instead of returning inconsistent results. Please use the range query endpoint /api/v1/query_range (Range type in Grafana Explore) instead.

Configuration

Loki changes the default value of -ruler.alertmanager-use-v2 from false to true. Alertmanager APIv1 was deprecated in Alertmanager 0.16.0 and is removed as of 0.27.0.

Experimental Bloom Filters

The bloom compactor component, which builds bloom filter blocks for query acceleration, has been removed in favor of two new components: bloom planner and bloom builder. Please consult the Query Acceleration with Blooms docs for more information.

CLI arguments (and their YAML counterparts) of per-tenant settings that have been removed as part of this change:

• -bloom-compactor.enable-compaction
• -bloom-compactor.shard-size
• -bloom-compactor.shard-size
• -bloom-compactor.shard-size

CLI arguments of per-tenant settings that have been moved to a different prefix as part of this change:

• -bloom-compactor.max-page-size changed to -bloom-builder.max-page-size
• -bloom-compactor.max-block-size changed to -bloom-builder.max-block-size
• -bloom-compactor.ngram-length changed to -bloom-builder.ngram-length
• -bloom-compactor.ngram-skip changed to -bloom-builder.ngram-skip
• -bloom-compactor.false-positive-rate changed to -bloom-builder.false-positive-rate
• -bloom-compactor.block-encoding changed to -bloom-builder.block-encoding

Their YAML counterparts in the limits_config block are kept identical.

All other CLI arguments (and their YAML counterparts) prefixed with -bloom-compactor. have been removed.

3.0.0

Loki 3.0 is a major version increase and comes with several breaking changes.

Here is the shortlist of things we think most people may encounter:

• Structured metadata is enabled by default and requires tsdb and v13 schema or Loki won’t start. Refer to Structured Metadata, Open Telemetry, Schemas and Indexes.
• The shared_store config is removed. Refer to Removed shared_store and shared_store_key_prefix from shipper configuration.
• Loki now enforces a max line size of 256KB by default (you can disable this or increase this but this is how Grafana Labs runs Loki). Refer to Changes to default configure values.
• Loki now enforces a max label limit of 15 labels per series, down from 30. Extra labels inflate the size of the index and reduce performance, you should almost never need more than 15 labels. Refer to Changes to default configure values.
• Loki will automatically attempt to populate a service_name label on ingestion. Refer to service_name label.
• There are many metric name changes. Refer to Distributor metric changes, Embedded cache metric changes, and Metrics namespace.

If you would like to see if your existing configuration will work with Loki 3.0:

1. In an empty directory on your computer, copy you configuration into a file named loki-config.yaml.
2. Run this command from that directory:

Bash Copy

docker run --rm -t -v "${PWD}":/config grafana/loki:3.0.0 -config.file=/config/loki-config.yaml -verify-config=true

The Helm chart has gone through some significant changes and has a separate upgrade guide: Upgrading to Helm 6.x.

Loki 3.0.0

Structured Metadata, Open Telemetry, Schemas and Indexes

A flagship feature of Loki 3.0 is native support for the Open Telemetry Protocol (OTLP). This is made possible by a new feature in Loki called Structured Metadata, a place for metadata which doesn’t belong in labels or log lines. OTel resources and attributes are often a great example of data which doesn’t belong in the index nor in the log line.

Structured Metadata is enabled by default in Loki 3.0, however, it requires your active schema be using both the tsdb index type AND the v13 storage schema. If you are not using both of these you have two options:

• Upgrade your index version and schema version before updating to 3.0, see schema config upgrade.
• Disable Structured Metadata (and therefore OTLP support) and upgrade to 3.0 and perform the schema migration after. This can be done by setting allow_structured_metadata: false in the limits_config section or set the command line argument -validation.allow-structured-metadata=false.

service_name label

Loki 3.0 will automatically assign a service_name label to all ingested logs by default. A service name is something required by Open Telemetry semantic conventions and is something Grafana Labs is building into our future user interface and query experiences.

Loki will attempt to create the service_name label by looking for the following labels in this order:

• service_name
• service
• app
• application
• name
• app_kubernetes_io_name
• container
• container_name
• component
• workload
• job

If no label is found matching the list, a value of unknown_service is applied.

You can change this list by providing a list of labels to discover_service_name in the limits_config block.

You can disable this by providing an empty value for discover_service_name.

Removed shared_store and shared_store_key_prefix from shipper configuration

The following CLI flags and the corresponding YAML settings to configure shared store for TSDB and BoltDB shippers are now removed:

• -boltdb.shipper.shared-store
• -tsdb.shipper.shared-store

Going forward the object_store setting in the period_config will be used to configure the store for the index. This enforces chunks and index files to reside together in the same storage bucket for a given period.

We are removing the shared store setting in an effort to simplify storage configuration and reduce the possibility for misconfiguration.

• If you have not configured -boltdb.shipper.shared-store,-tsdb.shipper.shared-store or their corresponding YAML setting before, no changes are required as part of the upgrade.
• If you have configured -boltdb.shipper.shared-store or its YAML setting:
  • If it matches the value of object_store for all the periods that use boltdb-shipper as index type, no additional changes are required besides removing the usage of the deleted configuration option.
  • If there is a mismatch, you lose access to the index for periods where -boltdb.shipper.shared-store does not match object_store.
    • To make these indexes queryable, index tables need to moved or copied to the store configured in object_store.
• If you have configured -tsdb.shipper.shared-store or its YAML setting:
  • If it matches the value of object_store for all the periods that use tsdb as index type, no additional changes are required besides removing the usage of the deleted configuration option.
  • If there is a mismatch, you lose access to the index for periods where -tsdb.shipper.shared-store does not match object_store.
    • To make these indexes queryable, index tables need to moved or copied to the store configured in object_store.

The following CLI flags and the corresponding YAML settings to configure a path prefix for TSDB and BoltDB shippers are now removed:

• -boltdb.shipper.shared-store.key-prefix
• -tsdb.shipper.shared-store.key-prefix

Path prefix for storing the index can now be configured by setting path_prefix under index key in period_config. This enables users to change the path prefix by adding a new period config.

YAML Copy

period_config:
  index:
    path_prefix: "index/"
    period: 24h

path_prefix defaults to index/ which is same as the default value of the removed configurations.

• No changes are required if you have not configured -boltdb.shipper.shared-store.key-prefix, -tsdb.shipper.shared-store.key-prefix or the corresponding YAML setting previously.
• If you have configured -boltdb.shipper.shared-store.key-prefix or its YAML setting to a value other than index/, ensure that all the existing period configs that use boltdb-shipper as the index have path_prefix set to the value previously configured.
• If you have configured -tsdb.shipper.shared-store.key-prefix or its YAML setting to a value other than index/, ensure that all the existing period configs that use tsdb as the index have the path_prefix set to the value previously configured.

Removed shared_store and shared_store_key_prefix from compactor configuration

The following CLI flags and the corresponding YAML settings to configure the shared store and path prefix for compactor are now removed:

• -boltdb.shipper.compactor.shared-store
• -boltdb.shipper.compactor.shared-store.key-prefix

Going forward compactor will run compaction and retention on all the object stores configured in period configs where the index type is either tsdb or boltdb-shipper.

delete_request_store should be explicitly configured

-compactor.delete-request-store or its YAML setting should be explicitly configured when retention is enabled, this is required for storing delete requests. The path prefix under which the delete requests are stored is decided by -compactor.delete-request-store.key-prefix, it defaults to index/.

Configuration async_cache_write_back_concurrency and async_cache_write_back_buffer_size have been removed

These configurations were redundant with the Background configuration in the cache-config.

async_cache_write_back_concurrency can be set with writeback_goroutines async_cache_write_back_buffer_size can be set with writeback_buffer

additionally the Background configuration also lest you set writeback_size_limit which can be used to set a maximum amount of memory to use for writeback objects vs a count of objects.

Legacy ingester shutdown handler is removed

The already deprecated handler /ingester/flush_shutdown is removed in favor of /ingester/shutdown?flush=true.

Ingester configuration max_transfer_retries is removed

The setting max_transfer_retries (-ingester.max-transfer-retries) is removed in favor of the Write Ahead log (WAL). It was used to allow transferring chunks to new ingesters when the old ingester was shutting down during a rolling restart. Alternatives to this setting are:

• A. (Preferred) Enable the WAL and rely on the new ingester to replay the WAL.
  • Optionally, you can enable flush_on_shutdown (-ingester.flush-on-shutdown) to flush to long-term storage on shutdowns.
• B. Manually flush during shutdowns via the ingester /shutdown?flush=true endpoint.

Removed the default section of the runtime overrides config file

This was introduced in 2.9 and likely not widely used. This only affects you if you run Loki with a runtime config file AND you had populated the new default block added in 2.9.

The default block was removed and instead a top level config now exists in the standard Loki config called operational_config, you can set default values here for runtime configs.

Configuration use_boltdb_shipper_as_backup is removed

The setting use_boltdb_shipper_as_backup (-tsdb.shipper.use-boltdb-shipper-as-backup) was a remnant from the development of the TSDB storage. It was used to allow writing to both TSDB and BoltDB when TSDB was still highly experimental. Since TSDB is now stable and the recommended index type, the setting has become irrelevant and therefore was removed. The previous default value false is applied.

Deprecated configuration options are removed

1. Removed already deprecated store.max-look-back-period CLI flag and the corresponding YAML settings. Use querier.max-query-lookback config instead.
2. Removes already deprecated -querier.engine.timeout CLI flag and the corresponding YAML setting.
3. Also removes the query_timeout from the querier YAML section. Instead of configuring query_timeout under querier, you now configure it in Limits Config.
4. s3.sse-encryption is removed. AWS now defaults encryption of all buckets to SSE-S3. Use sse.type to set SSE type.
5. ruler.wal-cleaer.period is removed. Use ruler.wal-cleaner.period instead.
6. experimental.ruler.enable-api is removed. Use ruler.enable-api instead.
7. split_queries_by_interval is removed from query_range YAML section. You can instead configure it in Limits Config.
8. frontend.forward-headers-list CLI flag and its corresponding YAML setting are removed.
9. frontend.cache-split-interval CLI flag is removed. Results caching interval is now determined by querier.split-queries-by-interval.
10. querier.worker-parallelism CLI flag and its corresponding yaml setting are now removed as it does not offer additional value to already existing querier.max-concurrent. We recommend configuring querier.max-concurrent to limit the max concurrent requests processed by the queriers.
11. ruler.evaluation-delay-duration CLI flag and the corresponding YAML setting are removed.
12. validation.enforce-metric-name CLI flag and the corresponding YAML setting are removed.
13. boltdb.shipper.compactor.deletion-mode CLI flag and the corresponding YAML setting are removed. You can instead configure the compactor.deletion-mode CLI flag or deletion_mode YAML setting in Limits Config.
14. Compactor CLI flags that use the prefix boltdb.shipper.compactor. are removed. You can instead use CLI flags with the compactor. prefix.

Distributor metric changes

The loki_distributor_ingester_append_failures_total metric has been removed in favour of loki_distributor_ingester_append_timeouts_total. This new metric will provide a more clear signal that there is an issue with ingesters, and this metric can be used for high-signal alerting.

Changes to default configuration values in 3.0

Automatic stream sharding is enabled by default

Automatic stream sharding helps keep the write load of high volume streams balanced across ingesters and helps to avoid hot-spotting. Check out the operations page for more information

More results caching is enabled by default

The TSDB index type has support for caching results for ‘stats’ and ‘volume’ queries which are now enabled by default.

’label’ and ‘series’ requests can be cached now too and this is enabled by default.

All of these are cached to the results_cache which is configured in the query_range config section. By default, an in memory cache is used.

Write dedupe cache is deprecated

Write dedupe cache is deprecated because it not required by the newer single store indexes ( TSDB and boltdb-shipper). If you using a legacy index type, consider migrating to TSDB (recommended).

Embedded cache metric changes

• The following embedded cache metrics are removed. Instead use loki_cache_fetched_keys, loki_cache_hits, loki_cache_request_duration_seconds which instruments requests made to the configured cache (embeddedcache, memcached or redis).

  • querier_cache_added_total
  • querier_cache_gets_total
  • querier_cache_misses_total

• The following embedded cache metrics are renamed:

  • querier_cache_added_new_total is renamed to loki_embeddedcache_added_new_total
  • querier_cache_evicted_total is renamed to loki_embeddedcache_evicted_total
  • querier_cache_entries is renamed to loki_embeddedcache_entries
  • querier_cache_memory_bytes is renamed to loki_embeddedcache_memory_bytes

• Already deprecated metric querier_cache_stale_gets_total is now removed.

Metrics namespace

Some Loki metrics started with the prefix cortex_. In this release they will be changed so they start with loki_. To keep them at cortex_ change the metrics_namespace from the default loki to cortex. These metrics will be changed:

• cortex_distributor_ingester_clients
• cortex_dns_failures_total
• cortex_dns_lookups_total
• cortex_dns_provider_results
• cortex_frontend_query_range_duration_seconds_bucket
• cortex_frontend_query_range_duration_seconds_count
• cortex_frontend_query_range_duration_seconds_sum
• cortex_ingester_flush_queue_length
• cortex_kv_request_duration_seconds_bucket
• cortex_kv_request_duration_seconds_count
• cortex_kv_request_duration_seconds_sum
• cortex_member_consul_heartbeats_total
• cortex_prometheus_last_evaluation_samples
• cortex_prometheus_notifications_alertmanagers_discovered
• cortex_prometheus_notifications_dropped_total
• cortex_prometheus_notifications_errors_total
• cortex_prometheus_notifications_latency_seconds
• cortex_prometheus_notifications_latency_seconds_count
• cortex_prometheus_notifications_latency_seconds_sum
• cortex_prometheus_notifications_queue_capacity
• cortex_prometheus_notifications_queue_length
• cortex_prometheus_notifications_sent_total
• cortex_prometheus_rule_evaluation_duration_seconds
• cortex_prometheus_rule_evaluation_duration_seconds_count
• cortex_prometheus_rule_evaluation_duration_seconds_sum
• cortex_prometheus_rule_evaluation_failures_total
• cortex_prometheus_rule_evaluations_total
• cortex_prometheus_rule_group_duration_seconds
• cortex_prometheus_rule_group_duration_seconds_count
• cortex_prometheus_rule_group_duration_seconds_sum
• cortex_prometheus_rule_group_interval_seconds
• cortex_prometheus_rule_group_iterations_missed_total
• cortex_prometheus_rule_group_iterations_total
• cortex_prometheus_rule_group_last_duration_seconds
• cortex_prometheus_rule_group_last_evaluation_timestamp_seconds
• cortex_prometheus_rule_group_rules
• cortex_query_frontend_connected_schedulers
• cortex_query_frontend_queries_in_progress
• cortex_query_frontend_retries_bucket
• cortex_query_frontend_retries_count
• cortex_query_frontend_retries_sum
• cortex_query_scheduler_connected_frontend_clients
• cortex_query_scheduler_connected_querier_clients
• cortex_query_scheduler_inflight_requests
• cortex_query_scheduler_inflight_requests_count
• cortex_query_scheduler_inflight_requests_sum
• cortex_query_scheduler_queue_duration_seconds_bucket
• cortex_query_scheduler_queue_duration_seconds_count
• cortex_query_scheduler_queue_duration_seconds_sum
• cortex_query_scheduler_queue_length
• cortex_query_scheduler_running
• cortex_ring_member_heartbeats_total
• cortex_ring_member_tokens_owned
• cortex_ring_member_tokens_to_own
• cortex_ring_members
• cortex_ring_oldest_member_timestamp
• cortex_ring_tokens_total
• cortex_ruler_client_request_duration_seconds_bucket
• cortex_ruler_client_request_duration_seconds_count
• cortex_ruler_client_request_duration_seconds_sum
• cortex_ruler_clients
• cortex_ruler_config_last_reload_successful
• cortex_ruler_config_last_reload_successful_seconds
• cortex_ruler_config_updates_total
• cortex_ruler_managers_total
• cortex_ruler_ring_check_errors_total
• cortex_ruler_sync_rules_total

The metrics_namespace setting is deprecated already. It will be removed in the next minor release. The default prefix will be loki then.

LogCLI

Store for retrieving remote schema

Previously LogCLI used to fetch remote schema from the store configured in -boltdb.shipper.shared-store when -remote-schema is set to true. A new CLI flag -schema-store is introduced as a replacement to configure the store for retrieving remote schema.

Further reading

To upgrade from one of the versions of Loki 2.x, refer to Upgrade Loki 2.x versions.

To upgrade from one of the versions of Loki 1.x, refer to Upgrade Loki 1.x versions.