Deploy Grafana Mimir on Kubernetes on Grafana Labs

Getting started with Grafana Mimir using the Helm chart

Wed, 11 Mar 2026 16:56:47 -0400

Getting started with Grafana Mimir using the Helm chart

The Helm chart allows you to configure, install, and upgrade Grafana Mimir within a Kubernetes cluster.

Before you begin

The instructions that follow are common across any flavor of Kubernetes. They also assume that you know how to install a Kubernetes cluster, and configure and operate it.

It also assumes that you have an understanding of what the kubectl command does.

Caution: Do not use this getting-started procedure in a production environment.

Hardware requirements:

A single Kubernetes node with a minimum of 4 cores and 16GiB RAM

Software requirements:

Kubernetes 1.20 or higher
The kubectl command for your version of Kubernetes
Helm 3 or higher

Verify that you have:

Access to the Kubernetes cluster
Persistent storage is enabled in the Kubernetes cluster, which has a default storage class set up. You can change the default StorageClass.
DNS service works in the Kubernetes cluster
An ingress controller is set up in the Kubernetes cluster, for example ingress-nginx

Note: Although this is not strictly necessary, if you want to access Mimir from outside of the Kubernetes cluster, you will need an ingress. This procedure assumes you have an ingress controller set up.

Install the Helm chart in a custom namespace

Using a custom namespace solves problems later on because you do not have to overwrite the default namespace.

Create a unique Kubernetes namespace, for example mimir-test:
console
```
kubectl create namespace mimir-test
```
For more details, see the Kubernetes documentation about Creating a new namespace.
Set up a Helm repository using the following commands:
console
```
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
```
Note: The Helm chart at https://grafana.github.io/helm-charts is a publication of the source code at grafana/mimir.
Configure an ingress:

a. Create a YAML file of Helm values called custom.yaml.

b. Add the following configuration to the file:
YAML
```
nginx:
  ingress:
    enabled: true
    ingressClassName: nginx
    hosts:
      - host: <ingress-host>
        paths:
          - path: /
            pathType: Prefix
    tls:
      # empty, disabled.
```
An ingress enables you to externally access a Kubernetes cluster. Replace <ingress-host> with a suitable hostname that DNS can resolve to the external IP address of the Kubernetes cluster. For more information, see Ingress.

Note: On Linux systems, and if it is not possible for you set up local DNS resolution, you can use the --add-host=<ingress-host>:<kubernetes-cluster-external-address> command-line flag to define the <ingress-host> local address for the docker commands in the examples that follow.
Install Grafana Mimir using the Helm chart:
Bash
```
helm -n mimir-test install mimir grafana/mimir-distributed -f custom.yaml
```
Note: The output of the command contains the write and read URLs necessary for the following steps.

Check the statuses of the Mimir pods:

kubectl -n mimir-test get pods

The results look similar to this:

kubectl -n mimir-test get pods
NAME                                            READY   STATUS      RESTARTS   AGE
mimir-minio-78b59f5569-fhlhs                    1/1     Running     0          2m4s
mimir-nginx-74f8bff8dc-7kr7z                    1/1     Running     0          2m5s
mimir-distributed-make-bucket-job-z2hc8         0/1     Completed   0          2m4s
mimir-overrides-exporter-5fd94b745b-htrdr       1/1     Running     0          2m5s
mimir-query-frontend-68cbbfbfb5-pt2ng           1/1     Running     0          2m5s
mimir-ruler-56586c9774-28k7h                    1/1     Running     0          2m5s
mimir-querier-7894f6c5f9-pj9sp                  1/1     Running     0          2m5s
mimir-querier-7894f6c5f9-cwjf6                  1/1     Running     0          2m4s
mimir-alertmanager-0                            1/1     Running     0          2m4s
mimir-distributor-55745599b5-r26kr              1/1     Running     0          2m4s
mimir-compactor-0                               1/1     Running     0          2m4s
mimir-store-gateway-0                           1/1     Running     0          2m4s
mimir-ingester-1                                1/1     Running     0          2m4s
mimir-ingester-2                                1/1     Running     0          2m4s
mimir-ingester-0                                1/1     Running     0          2m4s

Wait until all of the pods have a status of Running or Completed, which might take a few minutes.

Configure Prometheus to write to Grafana Mimir

You can either configure Prometheus to write to Grafana Mimir or configure Grafana Agent to write to Mimir. Although you can configure both, you do not need to.

Make a choice based on whether or not you already have a Prometheus server set up:

For an existing Prometheus server:
1. Add the following YAML snippet to your Prometheus configuration file:
  YAML
```
remote_write:
  - url: http://<ingress-host>/api/v1/push
```
  In this case, your Prometheus server writes metrics to Grafana Mimir, based on what is defined in the existing scrape_configs configuration.
2. Restart the Prometheus server.
For a Prometheus server that does not exist yet:
1. Write the following configuration to a prometheus.yml file:
  YAML
```
remote_write:
  - url: http://<ingress-host>/api/v1/push

scrape_configs:
  - job_name: prometheus
    honor_labels: true
    static_configs:
      - targets: ["localhost:9090"]
```
  In this case, your Prometheus server writes metrics to Grafana Mimir that it scrapes from itself.
2. Start a Prometheus server by using Docker:
  Bash
```
docker run -p 9090:9090  -v <absolute-path-to>/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus
```
  Note: On Linux systems, if <ingress-host> cannot be resolved by the Prometheus server, use the additional command-line flag --add-host=<ingress-host>:<kubernetes-cluster-external-address> to set it up.

Configure Grafana Agent to write to Grafana Mimir

You can either configure Grafana Agent to write to Grafana Mimir or configure Prometheus to write to Mimir. Although you can configure both, you do not need to.

Make a choice based on whether or not you already have a Grafana Agent set up:

For an existing Grafana Agent:
1. Add the following YAML snippet to your Grafana Agent metrics configurations (metrics.configs):
  YAML
```
remote_write:
  - url: http://<ingress-host>/api/v1/push
```
  In this case, your Grafana Agent will write metrics to Grafana Mimir, based on what is defined in the existing metrics.configs.scrape_configs configuration.
2. Restart the Grafana Agent.
For a Grafana Agent that does not exist yet:
1. Write the following configuration to an agent.yaml file:
  YAML
```
metrics:
  wal_directory: /tmp/grafana-agent/wal

  configs:
    - name: agent
      scrape_configs:
        - job_name: agent
          static_configs:
            - targets: ["127.0.0.1:12345"]
      remote_write:
        - url: http://<ingress-host>/api/v1/push
```
  In this case, your Grafana Agent writes metrics to Grafana Mimir that it scrapes from itself.
2. Create an empty directory for the write ahead log (WAL) of the Grafana Agent
3. Start a Grafana Agent by using Docker:
  Bash
```
docker run -v <absolute-path-to-wal-directory>:/etc/agent/data -v <absolute-path-to>/agent.yaml:/etc/agent/agent.yaml -p 12345:12345 grafana/agent
```
  Note: On Linux systems, if <ingress-host> cannot be resolved by the Grafana Agent, use the additional command-line flag --add-host=<ingress-host>:<kubernetes-cluster-external-address> to set it up.

Query metrics in Grafana

First install Grafana, and then add Mimir as a Prometheus data source.

Start Grafana by using Docker:
Bash
```
docker run --rm --name=grafana -p 3000:3000 grafana/grafana
```
Note: On Linux systems, if <ingress-host> cannot be resolved by Grafana, use the additional command-line flag --add-host=<ingress-host>:<kubernetes-cluster-external-address> to set it up.
In a browser, go to the Grafana server at http://localhost:3000.
Sign in using the default username admin and password admin.
On the left-hand side, go to Configuration > Data sources.

Configure a new Prometheus data source to query the local Grafana Mimir cluster, by using the following settings:

Field	Value
Name	Mimir
URL	http://<ingress-host>/prometheus

To add a data source, see Add a data source.

Verify success:

You should be able to query metrics in Grafana Explore, as well as create dashboard panels by using your newly configured Mimir data source. For more information, see Monitor Grafana Mimir.

Set up metamonitoring

Grafana Mimir metamonitoring collects metrics or logs, or both, about Grafana Mimir itself. In the example that follows, metamonitoring scrapes metrics about Grafana Mimir itself, and then writes those metrics to the same Grafana Mimir instance.

To enable metamonitoring in Grafana Mimir, add the following YAML snippet to your Grafana Mimir custom.yaml file:

metaMonitoring:
  serviceMonitor:
    enabled: true
  grafanaAgent:
    enabled: true
    installOperator: true
    metrics:
      additionalRemoteWriteConfigs:
        - url: "http://mimir-nginx.mimir-test.svc:80/api/v1/push"

Upgrade Grafana Mimir by using the helm command:

helm -n mimir-test upgrade mimir grafana/mimir-distributed -f custom.yaml

From Grafana Explore, verify that your metrics are being written to Grafana Mimir, by querying sum(rate(cortex_ingester_ingested_samples_total[$__rate_interval])).

Query metrics in Grafana that is running within the same Kubernetes cluster

Install Grafana in the same Kubernetes cluster.

For details, see Deploy Grafana on Kubernetes.
Stop the Grafana instance that is running in the Docker container, to allow for port-forwarding.
Port-forward Grafana to localhost, by using the kubectl command:
Bash
```
kubectl port-forward service/grafana 3000:3000
```
In a browser, go to the Grafana server at http://localhost:3000.
Sign in using the default username admin and password admin.
On the left-hand side, go to Configuration > Data sources.

Configure a new Prometheus data source to query the local Grafana Mimir server, by using the following settings:

Field	Value
Name	Mimir
URL	http://mimir-nginx.mimir-test.svc:80/prometheus

To add a data source, see Add a data source.

Verify success:

You should be able to query metrics in Grafana Explore, as well as create dashboard panels by using your newly configured Mimir data source.

Deploying Grafana Mimir with Jsonnet and Tanka

Wed, 11 Mar 2026 16:56:47 -0400

Deploying Grafana Mimir with Jsonnet and Tanka

Grafana Labs publishes Jsonnet files that you can use to deploy Grafana Mimir in microservices mode. Jsonnet files are located in the Mimir repository.

Upgrade the Grafana Mimir Helm chart from version 2.1 to 3.0

Wed, 11 Mar 2026 16:56:47 -0400

Upgrade the Grafana Mimir Helm chart from version 2.1 to 3.0

There are breaking changes between the Grafana Mimir Helm chart versions 2.1 and 3.0. Several parameters that were available in version 2.1 of the mimir-distributed Helm chart have changed.

To upgrade from Helm chart 2.1 to 3.0:

Understand the improvements that we made to the Mimir configuration in the Helm chart:
- The Mimir configuration is now stored in a Kubernetes ConfigMap by default, instead of a Kubernetes Secret.
- You can override individual properties without copying the entire mimir.config block. Specify properties you want to override under the mimir.structuredConfig.
- You can move secrets outside the Mimir configuration via external secrets and environment variables. Environment variables can be used to externalize secrets from the configuration file.
Decide whether or not you need to update the Mimir configuration:
- If you are using external configuration (useExternalConfig: true), then you must set configStorageType: Secret.
  
  Note: It is now possible to use a ConfigMap to manage your external configuration instead. If your external configuration contains secrets, then you can externalize them and use a ConfigMap. See Externalize secrets.
- If you are not using external configuration (useExternalConfig: false), and your Mimir configuration contains secrets, chose one of two options:
  - Keep the previous location as-is by setting configStorageType: Secret.
  - Externalize secrets:
    1. Move secrets from the Mimir configuration to a Kubernetes Secret.
    2. Mount the Kubernetes Secret via global.extraEnvFrom:
      YAML
      
      global: extraEnvFrom: - secretRef: name: mysecret
      For more information, see Secrets - Use case: As container environment variables.
    3. Replace the values in the Mimir configuration with environment variables.
      
      For example:
      YAML
      
      mimir: structuredConfig: blocks_storage: s3: secret_access_key: ${AWS_SECRET_ACCESS_KEY}
- If you are not using an external configuration (useExternalConfig: false), and your Mimir configuration does not contain secrets, then the storage location is automatically changed by Helm and you do not need to do anything.
See Example migrated values file.
Update your memcached configuration via your customized Helm chart values, if needed:

The mimir-distributed Helm chart supports multiple cache types. If you have not enabled any memcached caches, and you are not overriding the values of memcached, memcached-queries, memcached-metadata, or memcached-results sections, then you do not need to update the memcached configuration.

Otherwise, check to see if you need to change any of the following configuration parameters:
- The memcached section was repurposed, and chunks-cache was added.
- The contents of the memcached section now contain the following common values that are shared across all memcached instances: image, podSecurityContext, and containerSecurityContext.
- The following sections were renamed:
  - memcached-queries is now index-cache
  - memcached-metadata is now metadata-cache
  - memcached-results is now results-cache
- The memcached*.replicaCount values were renamed:
  - memcached.replicaCount is now chunks-cache.replicas
  - memcached-queries.replicaCount is now index-cache.replicas
  - memcached-metadata.replicaCount is now metadata-cache.replicas
  - memcached-results.replicaCount is now results-cache.replicas
- The memcached*.architecture values were removed.
- The memcached*.arguments values were removed.
- The default arguments are now encoded in the Helm chart templates; the values *-cache.allocatedMemory, *-cache.maxItemMemory and *-cache.port control the arguments -m, -I and -u. To provide additional arguments, use *-cache.extraArgs.
- The memcached*.metrics values were consolidated under memcachedExporter.
See also an example of migration of customized memcached values between versions 2.1 and 3.0.
Update your memcached-related Mimir configuration via your customized Helm chart value that is named mimir.config, if needed:

The configuration parameters for memcached addresses and max_item_size have changed in the default mimir.config value. If you previously copied the value of mimir.config into your values file, then take the latest version of the memcached configuration in the mimir.config from the values.yaml file in the Helm chart.
(Conditional) If you have enabled serviceMonitor, or you are overriding the value of anything under the serviceMonitor section, or both, then move the serviceMonitor section under metaMonitoring.
Update the rbac section, based on the following changes:
- If you are not overriding the value of anything under the rbac section, then skip this step.
- The rbac.pspEnabled value was removed.
- To continue using Pod Security Policy (PSP), set rbac.create to true and rbac.type to psp.
- To start using Security Context Constraints (SCC) instead of PSP, set rbac.create to true and rbac.type to scc.
Update the mimir.config value, based on the following information:
- Compare your overridden value of mimir.config with the one in the values.yaml file in the chart. If you are not overriding the value of mimir.config, then skip this step.
Decide whether or not to update the nginx configuration:
- Unless you have overridden the value of nginx.nginxConfig.file, and you are using the default mimir.config, then skip this step.
- Otherwise, compare the overridden nginx.nginxConfig.file value to the one in the values.yaml file in the Helm chart, and incorporate the differences. Pay attention to the sections that contain x_scope_orgid. The value in the values.yaml file contains Nginx configuration that adds the X-Scope-OrgId header to incoming requests that do not already set it.
  
  Note: This change allows Mimir clients to keep sending requests without needing to specify a tenant ID, even though multi-tenancy is now enabled by default.

Example of migrated values

The example values file is compatible with version 2.1 of the mimir-distributed Helm chart, and demonstrates a few things:

All memcached caches are enabled.
The default pod security policy is disabled.
ServiceMonitors are enabled.
Object storage credentials for block storage are specified directly in the mimir.config value.

Note: The unmodified parts of the default mimir.config are omitted for brevity, even though in a valid 2.1 values file they need to be included.

rbac:
  pspEnabled: false

memcached:
  enabled: true
  replicaCount: 1

memcached-queries:
  enabled: true
  replicaCount: 1

memcached-metadata:
  enabled: true
  replicaCount: 1

memcached-results:
  enabled: true
  replicaCount: 1

serviceMonitor:
  enabled: true

mimir:
  config: |-
    #######
    # default contents omitted for brevity
    #######

    blocks_storage:
      backend: s3
      s3:
        endpoint: s3.amazonaws.com
        bucket_name: my-blocks-bucket
        access_key_id: FAKEACCESSKEY
        secret_access_key: FAKESECRETKEY

    #######
    # default contents omitted for brevity
    #######

After applying the migration steps listed in this guide, you now have a Kubernetes Secret that contains the S3 credentials, and a values file for version 3.0. The values file is does not have any omissions. The parts that were omitted in the 2.1 version are automatically included by the Helm chart in version 3.0.

Kubernetes Secret:

apiVersion: v1
kind: Secret
metadata:
  name: mimir-bucket-secret
data:
  AWS_ACCESS_KEY_ID: FAKEACCESSKEY
  AWS_SECRET_ACCESS_KEY: FAKESECRETKEY

Values file:

rbac:
  create: false

chunks-cache:
  enabled: true
  replicas: 1

index-cache:
  enabled: true
  replicas: 1

metadata-cache:
  enabled: true
  replicas: 1

results-cache:
  enabled: true
  replicas: 1

metaMonitoring:
  serviceMonitor:
    enabled: true

mimir:
  structuredConfig:
    blocks_storage:
      backend: s3
      s3:
        access_key_id: ${AWS_ACCESS_KEY_ID}
        bucket_name: my-blocks-bucket
        endpoint: s3.amazonaws.com
        secret_access_key: ${AWS_SECRET_ACCESS_KEY}

global:
  extraEnvFrom:
    - secretRef:
        name: mimir-bucket-secret

Example of migration of customized memcached values between versions 2.1 and 3.0

Version 2.1:

memcached:
  replicaCount: 12
  arguments:
    - -m 2048
    - -I 128m
    - -u 12345
  image:
    repository: memcached
    tag: 1.6.9-alpine

memcached-queries:
  replicaCount: 3
  architecture: modern
  image:
    repository: memcached
    tag: 1.6.9-alpine

Version 3.0:

memcached:
  image:
    repository: memcached
    tag: 1.6.9-alpine

chunks-cache:
  allocatedMemory: 2048
  maxItemMemory: 128
  port: 12345
  replicas: 12

index-cache:
  replicas: 3