Run Grafana Mimir in production using the Helm chart on Grafana Labs

Configure high-availability deduplication with the Mimir distributed Helm chart

Fri, 31 Oct 2025 14:55:43 +0000

Configure high-availability deduplication with the Mimir distributed Helm chart

Grafana Mimir can deduplicate data from a high-availability (HA) Prometheus setup. Starting from Mimir 3.0, the HA tracker uses memberlist by default, which requires no external dependencies. For more information, see Configure high availability.

Note
Prior to Mimir 3.0, the HA tracker required an external key-value store such as Consul or etcd. These backends are now deprecated. If you’re currently using Consul or etcd for the HA tracker, refer to the migration guide.

Before you begin

You need to have a Grafana Mimir installed via the mimir-distributed Helm chart.

For conceptual information about how Mimir deduplicates incoming HA samples, refer to Configure high availability.

You also need to configure HA for Prometheus or Grafana Alloy.

Caution
Grafana Alloy is the new name for our distribution of the OTel collector. Grafana Agent has been deprecated and is in Long-Term Support (LTS) through October 31, 2025. Grafana Agent will reach an End-of-Life (EOL) on November 1, 2025. Read more about why we recommend migrating to Grafana Alloy.

Configure Prometheus or Grafana Alloy to send HA external labels

Configure the Prometheus or Grafana Alloy HA setup by setting the labels named cluster and __replica__, which are the default labels for a HA setup in Grafana Mimir. If you want to change the HA labels, make sure to change them in Mimir as well. This ensures that the configurations of Grafana Mimir, Prometheus, and Grafana Alloy all match each other. Otherwise, HA deduplication will not work.

The value of the cluster label must be same across replicas that belong to the same cluster.
The value of the __replica__ label must be unique across replicas within the same cluster.

global:
  external_labels:
    __replica__: replica-1
    cluster: my-prometheus

Reload or restart Prometheus or Grafana Alloy after you update the configuration.

Note: Configure high availability contains the same information on Prometheus setup for HA deduplication.

Configure Mimir high availability

You can configure Mimir HA deduplication globally or at the tenant level.

Globally configure HA deduplication

Before you begin, make sure that Mimir was installed using the mimir-distributed Helm chart.

Merge the following configuration to your custom.yaml file:

mimir:
  structuredConfig:
    limits:
      accept_ha_samples: true
      # The following two configurations must match those of external_labels in Prometheus
      # The config values below are the default and can be removed if you don't want to override to a new value
      ha_cluster_label: cluster
      ha_replica_label: __replica__
    distributor:
      ha_tracker:
        enable_ha_tracker: true
        kvstore:
          store: memberlist

Note
If memberlist is already configured for other Mimir components (such as the hash ring), the HA tracker will automatically use that configuration. In most Helm deployments, memberlist is already configured, so no additional configuration is needed.

Upgrade the Mimir Helm release using the following command:

 helm -n <mimir-namespace> upgrade mimir grafana/mimir-distributed -f custom.yaml

Configure HA deduplication per tenant

Before you begin, make sure that Mimir was installed using the mimir-distributed Helm chart.

Merge the following configuration to the custom.yaml file:

mimir:
  structuredConfig:
    limits:
      accept_ha_samples: true
      # The following two configurations must match those of external_labels in Prometheus
      # The config values below are the default and can be removed if you don't want to override to a new value
      ha_cluster_label: cluster
      ha_replica_label: __replica__
    distributor:
      ha_tracker:
        enable_ha_tracker: true
        kvstore:
          store: memberlist
runtimeConfig:
  overrides:
    <tenant-id>: # put real tenant ID here
      accept_ha_samples: true
      ha_cluster_label: cluster
      ha_replica_label: __replica__

The mimir configuration block is similar to the global HA deduplication configuration. The runtimeConfig block configures per-tenant HA deduplication.

Upgrade the Mimir’s helm release using the following command:

 helm -n <mimir-namespace> upgrade mimir grafana/mimir-distributed -f custom.yaml

Verifying deduplication

After Prometheus and Mimir are running, you can verify deduplication in several ways.

ha_tracker’s page

Port forward Mimir distributor service. The argument after port-forward must match your Mimir’s distributor name.

kubectl -n <mimir-namespace> port-forward service/mimir-distributor 8080:8080

You can use the following command to get the distributor name:

kubectl -n <mimir-namespace> get service | grep distributor

Open http://localhost:8080/distributor/ha_tracker in a browser. You should see the output similar like the following. If the table is empty, it means there is something wrong with the configuration.

Distributor metrics

If you have set up metamonitoring, Mimir distributor exposes some metrics related to HA deduplication. The relevant metrics are those with cortex_ha_tracker_ prefix.

Ensure HA metrics are deduplicated

Go to Grafana explore page and select Mimir datasource. Then execute the following query: up. In the Options drop down, select Format = Table. In the result you can see the several time series with different labels.

The most important thing is you will not find __replica__ label (or any label that you set in ha_replica_label config) anymore. This means you have configured the deduplication successfully.

Manage the configuration of Grafana Mimir with Helm

Fri, 31 Oct 2025 14:55:43 +0000

Manage the configuration of Grafana Mimir with Helm

The mimir-distributed Helm chart provides interfaces to set Grafana Mimir configuration parameters and to customize how to deploy Grafana Mimir on a Kubernetes cluster.

Overview

You can either manage the configuration of Grafana Mimir through the Helm chart or supply the configuration through a user-managed object.

If you want to manage the configuration through the Helm chart, refer to Manage the configuration with Helm.

If you want to manage the configuration externally, refer to Manage the configuration externally.

Handling sensitive information, such as credentials, is common between the two methods. Refer to Inject credentials.

Manage the configuration with Helm

There are three ways that you can modify configuration parameters:

Setting parameters via the mimir.structuredConfig value (recommended).
Copying the entire mimir.config value and modifying the configuration as text (discouraged, unless you want to prevent upgrades of the chart from automatically updating the configuration).
Setting extra CLI flags for components individually (discouraged, except for setting availability zones).

Refer to the Example for a practical application.

Limitation: It’s not possible to delete configuration parameters that were set in mimir.config through mimir.structuredConfig. Instead, set the configuration parameter to its default or to some other value.

How Grafana Mimir applies the configuration

Grafana Mimir components run with a configuration calculated by the following process:

The configuration YAML in mimir.config is evaluated as a Helm template. This step ensures that the configuration applies to the Kubernetes cluster where it will be installed. For example, setting up cluster-specific addresses.
The values from mimir.structuredConfig are recursively merged with mimir.config. The values from mimir.structuredConfig take precedence over the values in mimir.config. The result is again evaluated as a Helm template. This step applies user-specific customizations. For example, S3 storage details.
The resulting YAML configuration is then sorted alphabetically and stored in a ConfigMap (or Secret, depending on the value of configStorageType) and provided to all Grafana Mimir components.
The configuration file, as well as any extra CLI flags, are provided to the Mimir pods.
Each component evaluates the configuration, substituting environment variables as required. Note that extra CLI flags take precedence over the configuration file.

Note: CLI flags are component-specific, and therefore, don’t show up in the generated ConfigMap (or Secret). This makes it less obvious which configuration is running. Use these flags only when necessary.

Inspect changes to the configuration before upgrade

Follow these steps to inspect which change apply to the configuration.

Preparation:

Install the helm diff plugin.
Set configStorageType value to ConfigMap.
Inspect changes using the helm diff sub command:

helm -n mimir-test diff upgrade grafana/mimir-distributed -f custom.yaml

This command shows the differences between the running installation and the installation that would result from running the helm upgrade command. Search for name: mimir-config in the output to see the difference in configuration settings. Refer to Example output of helm diff command for a concrete example.

Note: You can find CLI flags and their differences in the Deployment and StatefulSet objects.

Manage runtime configuration

You can also use the Helm chart to manage runtime configuration files. Runtime configuration files contain configuration parameters and are periodically reloaded while Mimir is running. This process allows you to change a subset of Grafana Mimir’s configuration without having to restart the Grafana Mimir component or instance.

To manage runtime configuration with the Helm chart, add the following stanza to your Helm values file:

runtimeConfig:
  overrides:
    tenant-1: # limits for tenant-1 that the whole cluster enforces
      ingestion_tenant_shard_size: _`<SAMPLE_VALUE>`_
      max_global_series_per_user: _`<SAMPLE_VALUE>`_
      max_fetched_series_per_query: _`<SAMPLE_VALUE>`_

For more information about runtime configuration in Grafana Mimir, refer to About Grafana Mimir runtime configuration.

Manage the configuration externally

Prepare the configuration as text, without including Helm template functions or value evaluations. You can include references to environment variables, as explained in Injecting credentials.

Decide whether you want to use a ConfigMap or Secret to store the configuration. Handling ConfigMap is simpler, but beware of sensitive information.

Use external ConfigMap

Prepare a ConfigMap object that places the configuration under the mimir.yaml data key.

apiVersion: v1
kind: ConfigMap
metadata:
  name: my-mimir-config
data:
  mimir.yaml: |
    <configuration>

Replace <configuration> with the configuration as multi-line text, being mindful of indentation. The name my-mimir-config is just an example.

Set the following values in your custom values file, or on the Helm command line:

useExternalConfig: true
externalConfigSecretName: my-mimir-config
externalConfigVersion: "0"

Use external Secret

Prepare a Secret object, where the configuration is base64-encoded and placed under the mimir.yaml data key.

apiVersion: v1
kind: Secret
metadata:
  name: my-mimir-config
data:
  mimir.yaml: <configuration-base64>

Replace <configuration-base64> with the configuration encoded as base64 format string. The name my-mimir-config is just an example.

Set the following values in your custom values file, or on the Helm command line:

useExternalConfig: true
externalConfigSecretName: my-mimir-config
configStorageType: Secret
externalConfigVersion: "0"

Update the configuration

To make components aware of configuration changes, choose one of the following options:

Update the value in externalConfigVersion and run helm update.
Manually restart components affected by the configuration change.

Inject credentials

You can use the Helm chart value global.extraEnvFrom to inject credentials into the runtime environment variables of Grafana Mimir components. The data keys become environment variables and usable in the Grafana Mimir configuration. For example, you can reference AWS_SECRET_ACCESS_KEY as ${AWS_SECRET_ACCESS_KEY} in the configuration. Refer to the Example for a practical application.

To avoid complications, make sure that the key names in the secret contain only ASCII letters, numbers, and underscores. Prefer AWS_SECRET_ACCESS_KEY over secret-access-key.aws.

Grafana Mimir does not track changes to the credentials. If you update the credentials, restart Grafana Mimir pods to use the updated value. To trigger a restart, provide a global pod annotation in global.podAnnotation, which is applied to all Grafana Mimir pods. Changing the value of the global annotation makes Kubernetes recreate all pods. For example, changing global.podAnnotations.bucketSecretVersion from "0" to "1" triggers a restart. Note that pod annotations can only be strings.

Example configuration managed with Helm

This example shows how to set up the configuration to use an S3 bucket for blocks storage in a namespace called mimir-test.

Set up the external blocks storage, in this case S3 with buckets named, for example, my-blocks-bucket and my-ruler-bucket.
Create an external secret with the S3 credentials by writing the following to a mysecret.yaml file:
YAML
```
apiVersion: v1
kind: Secret
metadata:
  name: mimir-bucket-secret
data:
  AWS_ACCESS_KEY_ID: FAKEACCESSKEY
  AWS_SECRET_ACCESS_KEY: FAKESECRETKEY
```
Replace FAKEACCESSKEY and FAKESECRETKEY with the actual value encoded in base64.
Apply the secret to your cluster with the kubectl command:
Bash
```
kubectl -n mimir-test apply -f mysecret.yaml
```

Prepare your custom values file called custom.yaml:

global:
  extraEnvFrom:
    - secretRef:
        name: mimir-bucket-secret
  podAnnotations:
    bucketSecretVersion: "0"

# This turns of the built-in MinIO support
minio:
  enabled: false

mimir:
  structuredConfig:
    alertmanager_storage:
      s3:
        bucket_name: my-ruler-bucket
        access_key_id: ${AWS_ACCESS_KEY_ID}
        endpoint: s3.amazonaws.com
        secret_access_key: ${AWS_SECRET_ACCESS_KEY}
    blocks_storage:
      backend: s3
      s3:
        bucket_name: my-blocks-bucket
        access_key_id: ${AWS_ACCESS_KEY_ID}
        endpoint: s3.amazonaws.com
        secret_access_key: ${AWS_SECRET_ACCESS_KEY}
    ruler_storage:
      s3:
        bucket_name: my-ruler-bucket
        access_key_id: ${AWS_ACCESS_KEY_ID}
        endpoint: s3.amazonaws.com
        secret_access_key: ${AWS_SECRET_ACCESS_KEY}

Check the resulting configuration with the helm command before installing:

helm -n mimir-test template mimir grafana/mimir-distributed -f custom.yaml -s templates/mimir-config.yaml

You should see the following output:

---
# Source: mimir-distributed/templates/mimir-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: mimir-config
  labels:
    helm.sh/chart: mimir-distributed-3.0.0
    app.kubernetes.io/name: mimir
    app.kubernetes.io/instance: mimir
    app.kubernetes.io/version: "2.2.0"
    app.kubernetes.io/managed-by: Helm
  namespace: "mimir-test"
data:
  mimir.yaml: |
    activity_tracker:
      filepath: /active-query-tracker/activity.log
    alertmanager:
      data_dir: /data
      enable_api: true
      external_url: /alertmanager
    alertmanager_storage:
      s3:
        access_key_id: ${AWS_ACCESS_KEY_ID}
        bucket_name: my-ruler-bucket
        endpoint: s3.amazonaws.com
        secret_access_key: ${AWS_SECRET_ACCESS_KEY}
    blocks_storage:
      backend: s3
      bucket_store:
        sync_dir: /data/tsdb-sync
      s3:
        access_key_id: ${AWS_ACCESS_KEY_ID}
        bucket_name: my-blocks-bucket
        endpoint: s3.amazonaws.com
        secret_access_key: ${AWS_SECRET_ACCESS_KEY}
      tsdb:
        dir: /data/tsdb
    compactor:
      data_dir: /data
    frontend:
      align_queries_with_step: true
      log_queries_longer_than: 10s
    frontend_worker:
      frontend_address: mimir-query-frontend-headless.test.svc:9095
    ingester:
      ring:
        final_sleep: 0s
        num_tokens: 512
        unregister_on_shutdown: false
    ingester_client:
      grpc_client_config:
        max_recv_msg_size: 104857600
        max_send_msg_size: 104857600
    limits: {}
    memberlist:
      abort_if_cluster_join_fails: false
      compression_enabled: false
      join_members:
      - dns+mimir-gossip-ring.test.svc.cluster.local:7946
    ruler:
      alertmanager_url: dnssrvnoa+http://_http-metrics._tcp.mimir-alertmanager-headless.test.svc.cluster.local/alertmanager
      enable_api: true
      rule_path: /data
    ruler_storage:
      s3:
        access_key_id: ${AWS_ACCESS_KEY_ID}
        bucket_name: my-ruler-bucket
        endpoint: s3.amazonaws.com
        secret_access_key: ${AWS_SECRET_ACCESS_KEY}
    runtime_config:
      file: /var/mimir/runtime.yaml
    server:
      grpc_server_max_concurrent_streams: 1000

Install the chart with the helm command:

helm -n mimir-test install mimir grafana/mimir-distributed -f custom.yaml

Example output of helm diff command

The example is generated with the following steps:

Install Grafana Mimir with the helm command:

helm -n test install mimir grafana/mimir-distributed --version 3.0.0

Create a custom.yaml file with the following content:

mimir:
  structuredConfig:
    alertmanager:
      external_url: https://example.com/alerts
    server:
      log_level: debug

Produce the diff with the helm command:

helm -n test diff upgrade mimir grafana/mimir-distributed --version 3.0.0  -f custom.yaml

The output is an excerpt of the real output to reduce the size:

#... cut for size ...

test, mimir-config, ConfigMap (v1) has changed:
  # Source: mimir-distributed/templates/mimir-config.yaml
  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: mimir-config
    labels:
      helm.sh/chart: mimir-distributed-3.0.0
      app.kubernetes.io/name: mimir
      app.kubernetes.io/instance: mimir
      app.kubernetes.io/version: "2.2.0"
      app.kubernetes.io/managed-by: Helm
    namespace: "test"
  data:
    mimir.yaml: |
       activity_tracker:
        filepath: /active-query-tracker/activity.log
      alertmanager:
        data_dir: /data
        enable_api: true
-       external_url: /alertmanager
+       external_url: https://example.com/alerts
      alertmanager_storage:
        backend: s3
        s3:
          access_key_id: grafana-mimir
          bucket_name: mimir-ruler
          endpoint: mimir-minio.test.svc:9000
          insecure: true
          secret_access_key: supersecret
      blocks_storage:
        backend: s3
        bucket_store:
          sync_dir: /data/tsdb-sync
        s3:
          access_key_id: grafana-mimir
          bucket_name: mimir-tsdb
          endpoint: mimir-minio.test.svc:9000
          insecure: true
          secret_access_key: supersecret
        tsdb:
          dir: /data/tsdb
      compactor:
        data_dir: /data
      frontend:
        align_queries_with_step: true
        log_queries_longer_than: 10s
      frontend_worker:
        frontend_address: mimir-query-frontend-headless.test.svc:9095
      ingester:
        ring:
          final_sleep: 0s
          num_tokens: 512
          unregister_on_shutdown: false
      ingester_client:
        grpc_client_config:
          max_recv_msg_size: 104857600
          max_send_msg_size: 104857600
      limits: {}
      memberlist:
        abort_if_cluster_join_fails: false
        compression_enabled: false
        join_members:
        - mimir-gossip-ring
      ruler:
        alertmanager_url: dnssrvnoa+http://_http-metrics._tcp.mimir-alertmanager-headless.test.svc.cluster.local/alertmanager
        enable_api: true
        rule_path: /data
      ruler_storage:
        backend: s3
        s3:
          access_key_id: grafana-mimir
          bucket_name: mimir-ruler
          endpoint: mimir-minio.test.svc:9000
          insecure: true
          secret_access_key: supersecret
      runtime_config:
        file: /var/mimir/runtime.yaml
      server:
        grpc_server_max_concurrent_streams: 1000
+       log_level: debug

#... cut for size ...

test, mimir-distributor, Deployment (apps) has changed:
  # Source: mimir-distributed/templates/distributor/distributor-dep.yaml
  apiVersion: apps/v1
  kind: Deployment
  metadata:
    name: mimir-distributor
    labels:
      helm.sh/chart: mimir-distributed-3.0.0
      app.kubernetes.io/name: mimir
      app.kubernetes.io/instance: mimir
      app.kubernetes.io/component: distributor
      app.kubernetes.io/part-of: memberlist
      app.kubernetes.io/version: "2.2.0"
      app.kubernetes.io/managed-by: Helm
    annotations:
      {}
    namespace: "test"
  spec:
    replicas: 1
    selector:
      matchLabels:
        app.kubernetes.io/name: mimir
        app.kubernetes.io/instance: mimir
        app.kubernetes.io/component: distributor
    strategy:
      rollingUpdate:
        maxSurge: 0
        maxUnavailable: 1
      type: RollingUpdate
    template:
      metadata:
        labels:
          helm.sh/chart: mimir-distributed-3.0.0
          app.kubernetes.io/name: mimir
          app.kubernetes.io/instance: mimir
          app.kubernetes.io/version: "2.2.0"
          app.kubernetes.io/managed-by: Helm
          app.kubernetes.io/component: distributor
          app.kubernetes.io/part-of: memberlist
        annotations:
-         checksum/config: bad33a421a56693ebad68b64ecf407b5e897c3679b1a33b65672dbc4e98e918f
+         checksum/config: 02f080c347a1fcd6c9e49a38280330378d3afe12efc7151cd679935c96b35b83
        namespace: "test"
      spec:
        serviceAccountName: mimir
        securityContext:
          {}
        initContainers:
          []
        containers:
          - name: distributor
            image: "grafana/mimir:2.2.0"
            imagePullPolicy: IfNotPresent
            args:
              - "-target=distributor"
              - "-config.expand-env=true"
              - "-config.file=/etc/mimir/mimir.yaml"
            volumeMounts:
              - name: config
                mountPath: /etc/mimir
              - name: runtime-config
                mountPath: /var/mimir
              - name: storage
                mountPath: "/data"
                subPath:
            ports:
              - name: http-metrics
                containerPort: 8080
                protocol: TCP
              - name: grpc
                containerPort: 9095
                protocol: TCP
              - name: memberlist
                containerPort: 7946
                protocol: TCP
            livenessProbe:
              null
            readinessProbe:
              httpGet:
                path: /ready
                port: http-metrics
              initialDelaySeconds: 45
            resources:
              requests:
                cpu: 100m
                memory: 512Mi
            securityContext:
              readOnlyRootFilesystem: true
            env:
            envFrom:
        nodeSelector:
          {}
        affinity:
          podAntiAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                - key: target
                  operator: In
                  values:
                  - distributor
              topologyKey: kubernetes.io/hostname
        tolerations:
          []
        terminationGracePeriodSeconds: 60
        volumes:
          - name: config
            configMap:
              name: mimir-config
              items:
                - key: "mimir.yaml"
                  path: "mimir.yaml"
          - name: runtime-config
            configMap:
              name: mimir-runtime
          - name: storage
            emptyDir: {}
 #... cut for size ...

Lines starting with “-” are removed and lines starting with “+” are added. The change to the annotation checksum/config means the pods are restarted when this change is applied.