Deploy Grafana Mimir with Jsonnet and Tanka on Grafana Labs

Deploy Grafana Mimir with Jsonnet and Tanka

Thu, 14 Aug 2025 18:42:07 +0000

Deploy Grafana Mimir with Jsonnet and Tanka

You can use Tanka and jsonnet-bundler to generate Kubernetes YAML manifests from the jsonnet files.

Install tanka and jb:

Follow the steps at https://tanka.dev/install. If you have go installed locally you can also use:

# make sure to be outside of GOPATH or a go.mod project
go install github.com/grafana/tanka/cmd/tk@latest
go install github.com/jsonnet-bundler/jsonnet-bundler/cmd/jb@latest

Set up a Jsonnet project, based on the example that follows:

Initialize Tanka
Install Grafana Mimir and Kubernetes Jsonnet libraries
Set up an environment

#!/usr/bin/env bash
# SPDX-License-Identifier: AGPL-3.0-only

set -e

# Initialise the Tanka.
mkdir jsonnet-example && cd jsonnet-example
tk init --k8s=1.29

# Install Mimir jsonnet.
jb install github.com/grafana/mimir/operations/mimir@main

# Use the provided example.
cp vendor/mimir/mimir-manifests.jsonnet.example environments/default/main.jsonnet

# Generate the YAML manifests.
export PAGER=cat
tk show environments/default

Generate the Kubernetes YAML manifests and store them in the ./manifests directory:

# Generate the YAML manifests:
export PAGER=cat
tk show environments/default
tk export manifests environments/default

Configure the environment specification file at environments/default/spec.json.

To learn about how to use Tanka and to configure the spec.json file, see Using Jsonnet: Creating a new project.
Deploy the manifests to a Kubernetes cluster, in one of two ways:
- Use the tk apply command.
  
  Tanka supports commands to show the diff and apply changes to a Kubernetes cluster:
  sh
```
# Show the difference between your Jsonnet definition and your Kubernetes cluster:
tk diff environments/default

# Apply changes to your Kubernetes cluster:
tk apply environments/default
```
- Use the kubectl apply command.
  
  You generated the Kubernetes manifests and stored them in the ./manifests directory in the previous step.
  
  You can run the following command to directly apply these manifests to your Kubernetes cluster:
  sh
```
# Review the changes that will apply to your Kubernetes cluster:
kubectl apply --dry-run=client -k manifests/

# Apply the changes to your Kubernetes cluster:
kubectl apply -k manifests/
```
Note
The generated Kubernetes manifests create resources in the default namespace. To use a different namespace, change the namespace configuration option in the environments/default/main.jsonnet file, and re-generate the Kubernetes manifests.

Warning
A Jsonnet-based installation uses etcd for the HA tracker by default. The Jsonnet-based installation creates the etcd using the etcd-operator. Before applying the tk apply or kubectl apply command, make sure that you have the etcd-operator running in your Kubernetes cluster.

Configure Grafana Mimir to use low resources with Jsonnet

Thu, 14 Aug 2025 18:42:07 +0000

Configure Grafana Mimir to use low resources with Jsonnet

This page describes how to configure Jsonnet to deploy Grafana Mimir in a Kubernetes cluster with low CPU and memory resources available.

Anti-affinity

Given the distributed nature of Mimir, both performance and reliability are improved when pods are spread across different nodes. For example, losing multiple ingesters can cause data loss, so it’s better to distribute them across different nodes.

For this reason, by default, anti-affinity rules are applied to some Kubernetes Deployments and StatefulSets. These anti-affinity rules can become an issue when playing with Mimir in a single-node Kubernetes cluster. You can disable anti-affinity by setting the configuration values _config.<component>_allow_multiple_replicas_on_same_node.

Example: disable anti-affinity

local mimir = import 'mimir/mimir.libsonnet';

mimir {
  _config+:: {
    ingester_allow_multiple_replicas_on_same_node: true,
    store_gateway_allow_multiple_replicas_on_same_node: true,
  },
}

Resources

Default scaling of Mimir components in the provided Jsonnet is opinionated and based on engineers’ years of experience running it at Grafana Labs. The default resource requests and limits are also fine-tuned for the provided alerting rules. For more information, see Monitor Grafana Mimir.

However, there are use cases where you might want to change the default resource requests, their limits, or both. For example, if you are just testing Mimir and you want to run it on a small (possibly one-node) Kubernetes cluster, and you do not have tens of gigabytes of memory or multiple cores to schedule the components, consider overriding the scaling requirements as follows:

local k = import 'github.com/grafana/jsonnet-libs/ksonnet-util/kausal.libsonnet',
      deployment = k.apps.v1.deployment,
      statefulSet = k.apps.v1.statefulSet;
local mimir = import 'mimir/mimir.libsonnet';

mimir {
  _config+:: {
    // ... configuration values
  },

  compactor_container+: k.util.resourcesRequests('100m', '128Mi'),
  compactor_statefulset+: statefulSet.mixin.spec.withReplicas(1),

  distributor_container+: k.util.resourcesRequests('100m', '128Mi'),
  distributor_deployment+: deployment.mixin.spec.withReplicas(2),

  ingester_container+: k.util.resourcesRequests('100m', '128Mi'),
  ingester_statefulset+: statefulSet.mixin.spec.withReplicas(3),

  querier_container+: k.util.resourcesRequests('100m', '128Mi'),
  querier_deployment+: deployment.mixin.spec.withReplicas(2),

  query_frontend_container+: k.util.resourcesRequests('100m', '128Mi'),
  query_frontend_deployment+: deployment.mixin.spec.withReplicas(2),

  store_gateway_container+: k.util.resourcesRequests('100m', '128Mi'),
  store_gateway_statefulset+: statefulSet.mixin.spec.withReplicas(1),

  local smallMemcached = {
    cpu_requests:: '100m',
    memory_limit_mb:: 64,
    memory_request_overhead_mb:: 8,
    statefulSet+: statefulSet.mixin.spec.withReplicas(1),
  },

  memcached_chunks+: smallMemcached,
  memcached_frontend+: smallMemcached,
  memcached_index_queries+: smallMemcached,
  memcached_metadata+: smallMemcached,
}

Configure the Grafana Mimir object storage backend with Jsonnet

Thu, 14 Aug 2025 18:42:07 +0000

Configure the Grafana Mimir object storage backend with Jsonnet

You can configure the object storage backend for all Mimir components from a single place. The minimum Jsonnet code required for this is:

{
  _config+:: {
    storage_backend: 'gcs',
    blocks_storage_bucket_name: 'blocks-bucket',
  }
}

The storage_backend option must be one of either azure, gcs, or s3. Additional configuration options are available for each one of these providers.

Amazon S3 (`s3`) storage configuration options

Amazon S3 storage can be accessed without credentials when using Amazon VPC. In this case, storage_s3_secret_access_key and storage_s3_access_key_id are optional and can be left null, as in the following example:

{
  _config+:: {
    storage_backend: 's3',
    blocks_storage_bucket_name: 'blocks-bucket',
    aws_region: 'af-south-1',
  }
}

If credentials are required, it is a good practice to keep them in secrets. In that case environment variable interpolation can be used:

{
  _config+:: {
    storage_backend: 's3',
    storage_s3_access_key_id: '$(BLOCKS_STORAGE_S3_ACCESS_KEY_ID)',
    storage_s3_secret_access_key: '$(BLOCKS_STORAGE_S3_SECRET_ACCESS_KEY)',
    aws_region: 'af-south-1',
    blocks_storage_bucket_name: 'blocks-bucket',
  }
}

Azure (`azure`) storage configuration options

Hierarchical namespace must be disabled in Azure Blob Storage. Otherwise, Grafana Mimir will leave empty directories behind after deleting blocks.

The Azure storage client requires the storage_azure_account_name and storage_azure_account_key to be configured. It is a good practice to keep them in secrets. In that case environment variable interpolation can be used:

{
  _config+:: {
    storage_backend: 'azure',
    storage_azure_account_name: '$(STORAGE_AZURE_ACCOUNT_NAME)',
    storage_azure_account_key: '$(STORAGE_AZURE_ACCOUNT_KEY)',
    blocks_storage_bucket_name: 'blocks-bucket',
  }
}

Google Cloud Storage (`gcs`) storage configuration options

There are multiple ways to configure Google Cloud Storage client. If you run Mimir on Google Cloud Platform it is possible that the environment already has the credentials configured, in that case the minimum jsonnet configuration is valid:

{
  _config+:: {
    storage_backend: 'gcs',
    blocks_storage_bucket_name: 'blocks-bucket',
  }
}

You can use the storage_gcs_service_account configuration key to provide the service account when authentication is needed. It is a good practice to keep credentials in secrets, so environment variable interpolation can be used:

{
  _config+:: {
    storage_backend: 'gcs',
    storage_gcs_service_account: '$(STORAGE_GCS_SERVICE_ACCOUNT)',
    blocks_storage_bucket_name: 'blocks-bucket',
  }
}

Alternatively, you can set the GOOGLE_APPLICATION_CREDENTIALS environment variable to point to the service account file mounted from a secret.

Configure the Grafana Mimir ruler with Jsonnet

Thu, 14 Aug 2025 18:42:07 +0000

Configure the Grafana Mimir ruler with Jsonnet

The ruler is an optional component and is therefore not deployed by default when using Jsonnet. For more information about the ruler, see Grafana Mimir ruler.

To enable it, add the following Jsonnet code to the _config section:

{
  _config+:: {
    ruler_enabled: true,
    ruler_storage_bucket_name: 'ruler-bucket-name',
  },
}

By default, the object storage backend used for the ruler will be the one set by the $._config.storage_backend option. If desired, you can override it by setting the $._config.ruler_storage_backend option. The ruler_storage_backend option must be one of either local, azure, gcs, or s3. For more information about the options available for storing ruler state, see Grafana Mimir ruler: State.

To get started, use the local client type for initial testing:

{
  _config+:: {
    ruler_enabled: true,
    ruler_storage_backend: 'local',
    ruler_local_directory: '/path/to/local/directory',
  },
}

If you are using object storage, you must set ruler_storage_bucket_name to the name of the bucket that you want to use.

Note
If ruler object storage credentials differ from the ones defined in the common section, you need to manually provide them by using additional command line arguments. For more information, refer to Grafana Mimir configuration parameters: ruler_storage.

Operational modes

The ruler has two operational modes: internal and remote. By default, the Jsonnet deploys the ruler by using the internal operational mode. For more information about these modes, see Operational modes.

To enable the remote operational mode, add the following code to the Jsonnet:

{
  _config+:: {n
    ruler_remote_evaluation_enabled: true,
  },
}

Note
To support the remote operational mode, the configuration includes three additional Kubernetes Deployments as a separate query path.

These are:

ruler-query-frontend

ruler-query-scheduler

ruler-querier

Migrate to remote evaluation

To perform a zero downtime migration from internal to remote rule evaluation, follow these steps:

Deploy the following changes to enable remote evaluation in migration mode. Doing so causes the three new and previously listed Kubernetes deployments to start. However, they will not reconfigure the ruler to use them just yet.
jsonnet
```
{
 _config+:: {
   ruler_remote_evaluation_enabled: true,
   ruler_remote_evaluation_migration_enabled: true,
 },
}
```
Check that all of pods for the following deployments have successfully started before moving to the next step:
- ruler-query-frontend
- ruler-query-scheduler
- ruler-querier
Reconfigure the ruler pods to perform remote evaluation, by deploying the following changes:
jsonnet
```
{
  _config+:: {
    ruler_remote_evaluation_enabled: true,
  },
}
```

Configure Grafana Mimir autoscaling with Jsonnet

Thu, 14 Aug 2025 18:42:07 +0000

Configure Grafana Mimir autoscaling with Jsonnet

Mimir Jsonnet supports autoscaling for the following components:

Autoscaling, which is based on Prometheus metrics and Kubernetes-based Event Driven Autoscaler (KEDA), uses Kubernetes’ Horizontal Pod Autoscaler (HPA).

HPA is not configured directly in Jsonnet but it’s created and updated by KEDA. KEDA is an operator, running in the Kubernetes cluster, which is responsible to simplify the setup of HPA with custom metrics (Prometheus in our case).

Warning
Do not use the same Mimir or Grafana Enterprise Metrics cluster for storing and querying autoscaling metrics. Using the same cluster can create a dangerous feedback loop:

Don’t use the same Mimir or Grafana Enterprise Metrics cluster for storing and querying autoscaling metrics. Using the same cluster can create a dangerous feedback loop.

For instance, if the Mimir or GEM cluster becomes unavailable, autoscaling stops working, because it cannot query the metrics. This prevents the cluster from automatically scaling up during high load or recovery. This inability to scale further exacerbates the cluster’s unavailability, which might, in turn, prevent the cluster from recovering.

Instead, use a separate Prometheus instance or a different metrics backend for autoscaling metrics.

How KEDA works

KEDA is a Kubernetes operator aiming to simplify the wiring between HPA and Prometheus.

Kubernetes HPA, out of the box, is not capable of autoscaling based on metrics scraped by Prometheus, but it allows to configure a custom metrics API server which proxies metrics from a data source (e.g. Prometheus) to Kubernetes. Setting up the custom metrics API server for Prometheus in a Kubernetes can be a tedious operation, so KEDA offers an operator to set it up automatically. KEDA supports proxying metrics for a variety of sources, including Prometheus.

KEDA in a nutshell

Runs an operator and an external metrics server.
The metrics server supports proxying for many metric sources, including Prometheus.
The operator watches for ScaledObject custom resource definition (CRD), defining the minimum and maximum replicas, and scaling trigger metrics of a Deployment or StatefulSet, and then configures the related HPA resource. You don’t create the HPA resource in Kubernetes, but the operator creates it for you whenever a ScaledObject CRD is created (and keeps it updated for its whole lifecycle).

Refers to KEDA documentation for more information.

What happens if KEDA is unhealthy

The autoscaling of deployments is always managed by HPA, which is a native Kubernetes feature. KEDA, as we use it, never changes the number of replicas of Mimir Deployments or StatefulSets.

However, if KEDA is not running successfully, there are consequences for Mimir autoscaling too:

keda-operator is down (critical): as the operator is the single point of truth, it will not request the metrics when keda-operator-metrics-apiserver requests them. Changes to ScaledObject CRD will not be reflected to the HPA until the operator is back online. The deployment (e.g. queriers) will keep working but, if there is a surge of traffic, HPA will not be able to detect it due to a lack of metrics and so will not scale up.
keda-operator-metrics-apiserver is down (critical): HPA is not able to fetch updated metrics and it will stop scaling the deployment until metrics will be back. The deployment (e.g. queriers) will keep working but, if there is a surge of traffic, HPA will not be able to detect it due to a lack of metrics and so will not scale up.
keda-admission-webhooks is down (not critical): CRD validation will not be executed. Based on KEDA’s configuration, this can block changes on CRDs until the service is restored. HPA functionality is not affected.

Note
Use a high availability KEDA configuration if autoscaling is critical for your use case.

The alert MimirAutoscalerNotActive fires if HPA is unable to scale the deployment for any reason (e.g. unable to scrape metrics from KEDA metrics API server).

How Kubernetes HPA works

Refer to Kubernetes Horizontal Pod Autoscaling documentation to have a full understanding of how HPA works.

How to enable autoscaling

The following Jsonnet configuration snippet shows an example of how to enable Mimir autoscaling with Jsonnet:

local mimir = import 'mimir/mimir.libsonnet';

mimir {
    _config+:: {
        autoscaling_prometheus_url: 'http://prometheus.default:9090/prometheus',
        // Enable queriers autoscaling.
        autoscaling_querier_enabled: true,
        autoscaling_querier_min_replicas: 10,
        autoscaling_querier_max_replicas: 40,
        // Enable ruler queriers autoscaling.
        autoscaling_ruler_querier_enabled: true,
        autoscaling_ruler_querier_min_replicas: 10,
        autoscaling_ruler_querier_max_replicas: 40,
        // Enable distributor autoscaling.
        autoscaling_distributor_enabled: true,
        autoscaling_distributor_min_replicas: 10,
        autoscaling_distributor_max_replicas: 40,
    }
}

Note
The Mimir Jsonet doesn’t install KEDA.

To install KEDA, follow the instructions in Deploying KEDA.

How to disable autoscaling

There are two options to disable autoscaling in a Mimir cluster:

Set minimum replicas = maximum replicas.
Decommission HPA.

Set minimum replicas = maximum replicas

If KEDA and Kubernetes HPA work correctly but the HPA configuration (metric and threshold) are not giving the expected results (e.g. not scaling up when required), a simple solution to bypass the autoscaling algorithm is to set the minimum and maximum replicas to the same value. (e.g. autoscaling_querier_min_replicas: 40 and autoscaling_querier_max_replicas: 40).

Decommission HPA

To fully decommission HPA in a Mimir cluster you have to:

Disable the relevant autoscaler (e.g. autoscaling_querier_enabled: false).
Manually set the expected number of replicas for the given Mimir component.

The following example shows how to disable querier autoscaler and configure querier Deployment with 10 replicas:

local k = import 'github.com/grafana/jsonnet-libs/ksonnet-util/kausal.libsonnet';
local deployment = k.apps.v1.deployment;

mimir {
    _config+:: {
        autoscaling_querier_enabled: false,
    },

    querier_deployment+: deployment.mixin.spec.withReplicas(10),
}

Configure deployment mode

Thu, 14 Aug 2025 18:42:07 +0000

Configure deployment mode

Grafana Mimir supports multiple deployment modes. By default, the provided Jsonnet deploys in microservices mode. Monolithic mode is not supported in Jsonnet.

Use Read-Write deployment mode

Warning
Read-Write deployment mode is experimental.

Requirements

Read-write deployment mode requires that you use multi-zone ingesters and multi-zone store gateways. Additionally, rule evaluation is performed within mimir-backend, so you must disable ruler remote evaluation.

You can set the deployment mode via the deployment_mode configuration variable:

{
  _config+:: {
    deployment_mode: 'read-write',

    mimir_write_replicas: 15,
    mimir_read_replicas: 6,
    mimir_backend_replicas: 9,

    // Requirements.
    multi_zone_ingester_enabled: true,
    multi_zone_store_gateway_enabled: true,
    ruler_remote_evaluation_enabled: false,

    // Disable microservices autoscaling.
    autoscaling_querier_enabled: false,
    autoscaling_ruler_querier_enabled: false,
  }

}

You can configure autoscaling for the read path:

{
  _config+:: {
    autoscaling_mimir_read_enabled: true,
    autoscaling_mimir_read_min_replicas: 2,
    autoscaling_mimir_read_max_replicas: 20,
  }
}

CLI flags for read-write components are inherited from the microservices. For example:

{
  _config+:: {
    // This change also applies to mimir-backend.
    store_gateway_args+:: {
      'blocks-storage.bucket-store.sync-interval': '5m',
    },
  }
}

⚠️ Pitfall: Kubernetes resources’ overrides are not inherited. Remember to apply overrides to both microservices and read-write components, if you make changes to any of the following items:

Container specification, for example environment variables
Deployment
StatefulSet
Service
PodDisruptionBudget

For example:

{
  _config+:: {
    // This change will NOT be applied to mimir-write too.
    ingester_container+::
      container.withEnvMixin(
        [envVar.new('GOGC', '50')]
      ),

    mimir_write_container+::
      container.withEnvMixin(
        [envVar.new('GOGC', '50')]
      ),
  }
}

Migrate from Consul to memberlist KV store for hash rings without downtime

Thu, 14 Aug 2025 18:42:07 +0000

Migrate from Consul to memberlist KV store for hash rings without downtime

Mimir Jsonnet uses memberlist as KV store for hash rings since Mimir 2.2.0.

Memberlist can be disabled by using the following configuration:

{
  _config+:: {
    memberlist_ring_enabled: false
  }
}

If you are running Mimir hash rings with Consul and would like to migrate to memberlist without any downtime, you can follow instructions in this document.

Step 1: Enable memberlist and multi KV store.

{
  _config+:: {
    memberlist_ring_enabled: true,
    multikv_migration_enabled: true,
  }
}

Step 1 configures components to use multi KV store, with consul as primary and memberlist as secondary stores. This step requires rollout of all Mimir components. After applying this step all Mimir components will expose /memberlist page on HTTP admin interface, which can be used to check health of memberlist cluster.

Step 2: Enable KV store mirroring

{
  _config+:: {
    memberlist_ring_enabled: true,
    multikv_migration_enabled: true,
    multikv_mirror_enabled: true,  // Changed in this step.
  }
}

In this step we enable writes to primary KV store (Consul) to be mirrored into secondary store (memberlist). Applying this change will not cause restart of Mimir components.

You can monitor following metrics to check if mirroring was enabled on all components and if it works correctly:

cortex_multikv_mirror_enabled – shows which components have KV store mirroring enabled. All Mimir components should start mirroring to secondary KV store reloading runtime configuration.
rate(cortex_multikv_mirror_writes_total[1m]) – shows rate of writes to secondary KV store in writes per second.
rate(cortex_multikv_mirror_write_errors_total[1m]) – shows rate of write errors to secondary KV store, in errors per second.

After mirroring is enabled, you should see a key for each Mimir hash ring in the Memberlist cluster information admin page. See list of components that use hash ring.

Step 3: Switch Primary and Secondary store

{
  _config+:: {
    memberlist_ring_enabled: true,
    multikv_migration_enabled: true,
    multikv_mirror_enabled: true,
    multikv_switch_primary_secondary: true,  // Changed in this step.
  }
}

This change will switch primary and secondary stores as used by multi KV. From this point on Mimir components will use memberlist as primary KV store, and they will mirror updates to Consul. This step does not require restart of Mimir components.

To see if all components started to use memberlist as primary store, please watch cortex_multikv_primary_store metric.

Step 4: Disable mirroring to Consul

{
  _config+:: {
    memberlist_ring_enabled: true,
    multikv_migration_enabled: true,
    multikv_mirror_enabled: false,  // Changed in this step.
    multikv_switch_primary_secondary: true,
  }
}

This step does not require restart of any Mimir component. After applying the change components will stop writing ring updates to Consul, and will only use memberlist. You can watch cortex_multikv_mirror_enabled metric to see if all components have picked up updated configuration.

Step 5: Disable `multi` KV Store

{
  _config+:: {
    memberlist_ring_enabled: true,
    multikv_migration_enabled: false,  // Changed in this step.
    multikv_mirror_enabled: false,
    multikv_switch_primary_secondary: true,
    multikv_migration_teardown: true,  // Added in this step.
  }
}

This configuration change will cause a new rollout of all components. After the restart components will no longer use multi KV store and will be configured to use memberlist only. We use multikv_migration_teardown to preserve runtime configuration for multi KV store for components that haven’t restarted yet.

All cortex_multikv_* metrics are only exposed by components that use multi KV store. As components restart, these metrics will disappear.

Note
Setting multikv_migration_enabled: false while keeping memberlist_ring_enabled: true removes the Consul workload.

That’s expected, since Consul isn’t used anymore—you disabled mirroring to it in step 4.

If you need to keep consul running, you can explicitly set consul_enabled: true in _config.

Step 6: Cleanup

We have successfully migrated Mimir cluster from using Consul to memberlist without any downtime! As a final step, we can remove all migration-related config options:

multikv_migration_enabled
multikv_mirror_enabled
multikv_switch_primary_secondary
multikv_migration_teardown

Our final memberlist configuration will be:

{
  _config+:: {
    memberlist_ring_enabled: true,
  }
}

This will not trigger new restart of Mimir components. After applying this change, you are finished.

Migrate from microservices to read-write mode without downtime

Thu, 14 Aug 2025 18:42:07 +0000

Migrate from microservices to read-write mode without downtime

Warning
Read-Write mode, and migrating between modes, is experimental.

At a high level, the steps involved are as follows:

Deploy read-write components alongside microservices; they both join the same ring.
Switch over end points in your ingress.
Decommission microservices.

Steps can be applied vice-versa for migrating from read-write mode to microservices.

Step 1: Configure prerequisite zone-awareness

Read-write mode requires that you enable multi-zone ingesters and store-gateways.

{
  _config+:: {
    multi_zone_ingester_enabled: true,
    multi_zone_store_gateway_enabled: true,
  },
}

Ruler remote evaluation is also required to be disabled, however this is done later in the migration process if you are presently using it in microservices.

Step 2: Deploy read-write components with 0 replicas

By setting the deployment_mode to migration, jsonnet will configure the components for both read-write and microservices. To begin with we want to set their replicas to 0.

{
  _config+:: {
    deployment_mode: 'migration',

    mimir_write_replicas: 0,
    mimir_read_replicas: 0,
    mimir_backend_replicas: 0,
    autoscaling_mimir_read_enabled: false,
  },
}

Optionally double check configuration

Optionally at this point you may like to compare the component configuration between microservice and read-write modes.

For example:

# Export all of the Kubernetes objects to yaml:

kubectl get -o yaml deployment distributor > distributor.yaml; yq eval -i '.spec' distributor.yaml
kubectl get -o yaml deployment overrides-exporter > overrides-exporter.yaml; yq eval -i '.spec' overrides-exporter.yaml
kubectl get -o yaml deployment querier > querier.yaml; yq eval -i '.spec' querier.yaml
kubectl get -o yaml deployment query-frontend > query-frontend.yaml; yq eval -i '.spec' query-frontend.yaml
kubectl get -o yaml deployment query-scheduler > query-scheduler.yaml; yq eval -i '.spec' query-scheduler.yaml
kubectl get -o yaml deployment ruler > ruler.yaml; yq eval -i '.spec' ruler.yaml
kubectl get -o yaml deployment ruler-querier > ruler-querier.yaml; yq eval -i '.spec' ruler-querier.yaml
kubectl get -o yaml deployment ruler-query-frontend > ruler-query-frontend.yaml; yq eval -i '.spec' ruler-query-frontend.yaml
kubectl get -o yaml deployment ruler-query-scheduler > ruler-query-scheduler.yaml; yq eval -i '.spec' ruler-query-scheduler.yaml
kubectl get -o yaml deployment mimir-read > mimir-read.yaml; yq eval -i '.spec' mimir-read.yaml
kubectl get -o yaml statefulset compactor > compactor.yaml; yq eval -i '.spec' compactor.yaml
kubectl get -o yaml statefulset ingester-zone-a > ingester-zone-a.yaml; yq eval -i '.spec' ingester-zone-a.yaml
kubectl get -o yaml statefulset ingester-zone-b > ingester-zone-b.yaml; yq eval -i '.spec' ingester-zone-b.yaml
kubectl get -o yaml statefulset ingester-zone-c > ingester-zone-c.yaml; yq eval -i '.spec' ingester-zone-c.yaml
kubectl get -o yaml statefulset store-gateway-zone-a > store-gateway-zone-a.yaml; yq eval -i '.spec' store-gateway-zone-a.yaml
kubectl get -o yaml statefulset store-gateway-zone-b > store-gateway-zone-b.yaml; yq eval -i '.spec' store-gateway-zone-b.yaml
kubectl get -o yaml statefulset store-gateway-zone-c > store-gateway-zone-c.yaml; yq eval -i '.spec' store-gateway-zone-c.yaml
kubectl get -o yaml statefulset mimir-write-zone-a > mimir-write-zone-a.yaml; yq eval -i '.spec' mimir-write-zone-a.yaml
kubectl get -o yaml statefulset mimir-write-zone-b > mimir-write-zone-b.yaml; yq eval -i '.spec' mimir-write-zone-b.yaml
kubectl get -o yaml statefulset mimir-write-zone-c > mimir-write-zone-c.yaml; yq eval -i '.spec' mimir-write-zone-c.yaml
kubectl get -o yaml statefulset mimir-backend-zone-a > mimir-backend-zone-a.yaml; yq eval -i '.spec' mimir-backend-zone-a.yaml
kubectl get -o yaml statefulset mimir-backend-zone-b > mimir-backend-zone-b.yaml; yq eval -i '.spec' mimir-backend-zone-b.yaml
kubectl get -o yaml statefulset mimir-backend-zone-c > mimir-backend-zone-c.yaml; yq eval -i '.spec' mimir-backend-zone-c.yaml

# Diff deployments and statefulsets:

## Write
diff --color=always distributor.yaml mimir-write-zone-a.yaml
diff --color=always ingester-zone-a.yaml mimir-write-zone-a.yaml
diff --color=always ingester-zone-b.yaml mimir-write-zone-b.yaml
diff --color=always ingester-zone-c.yaml mimir-write-zone-c.yaml

## Read
diff --color=always query-frontend.yaml mimir-read.yaml
diff --color=always querier.yaml mimir-read.yaml
diff --color=always ruler-query-frontend.yaml mimir-read.yaml
diff --color=always ruler-querier.yaml mimir-read.yaml

## Backend
diff --color=always overrides-exporter.yaml mimir-backend-zone-a.yaml
diff --color=always query-scheduler.yaml mimir-backend-zone-a.yaml
diff --color=always ruler-query-scheduler.yaml mimir-backend-zone-a.yaml
diff --color=always ruler.yaml mimir-backend-zone-a.yaml
diff --color=always compactor.yaml mimir-backend-zone-a.yaml
diff --color=always store-gateway-zone-a.yaml mimir-backend-zone-a.yaml
diff --color=always store-gateway-zone-b.yaml mimir-backend-zone-b.yaml
diff --color=always store-gateway-zone-c.yaml mimir-backend-zone-c.yaml

Step 3: Migrate read-path to read-write service

Step 3.1: Scale up read component

Scale up the Mimir read component using either the autoscaler or explicitly setting the number of replicas. (Keep the current microservices at their present level of replicas or autoscaling).

{
  _config+:: {
    deployment_mode: 'migration',

    mimir_write_replicas: 0,
    mimir_read_replicas: 3,
    mimir_backend_replicas: 0,
    autoscaling_mimir_read_enabled: false,
  },
}

{
  _config+:: {
    deployment_mode: 'migration',

    mimir_write_replicas: 0,
    mimir_backend_replicas: 0,
    autoscaling_mimir_read_enabled: true,
    autoscaling_mimir_read_min_replicas: 3,
    autoscaling_mimir_read_max_replicas: 30,
  },
}

The Read-write querier at this point will start running queries from the query-scheduler (as they share the same ring).

Step 3.2: Check mimir-read is working

Perform a test query by port-forwarding to mimir-read.

Ensure mimir-read is running queries:

sum by (pod) (rate(cortex_querier_request_duration_seconds_count{job=~".*mimir-read.*", route=~"(prometheus|api_prom)_api_v1_.+"}[1m]))

Step 3.3: Route traffic to mimir-read

Configure your load-balancer to route read requests to mimir-read.

Ensure the query-frontend microservice is no longer getting requests:

sum by(pod) (rate(cortex_query_frontend_queries_total{pod!~"ruler-query-frontend.*"}[1m]))

Step 4: Migrate backend components to backend service

Step 4.1: Scale up backend component

Scale up the Mimir backend component.

{
  _config+:: {
    mimir_backend_replicas: 3,
  },
}

Step 4.2: Check mimir-backend is working

Check the following rings:

Query-scheduler ring should include both microservices and read-write components.
Store-gateway ring should include both microservices and read-write components.
Compactor ring should include both microservices and read-write components.
Ruler ring should include both microservices and read-write components.

Run a few test queries on long-term data.

Step 4.3: Route traffic to mimir-backend

Configure your load-balancer to route compactor and ruler endpoints to mimir-backend.

Step 5: Scale down microservice read and backend components

Now that we have mimir-read and mimir-backend scaled up and receiving traffic, we can safely decommission the microservices on those paths.

First though, configure the microservice store-gateway to leave the ring:

{
  // Configure microservices store-gateway to leave the ring.
  store_gateway_args+:: {
    'store-gateway.sharding-ring.unregister-on-shutdown': true,
  },

  mimir_backend_args+:: {
    'store-gateway.sharding-ring.unregister-on-shutdown': false,
  },
}

Then scale down the replicas for all components:

{
  _config+:: {
    multi_zone_store_gateway_replicas: 0,
    autoscaling_querier_enabled: false,
  },

  query_frontend_deployment+:
    deployment.mixin.spec.withReplicas(0),

  query_scheduler_deployment+:
    deployment.mixin.spec.withReplicas(0),

  querier_deployment+:
    deployment.mixin.spec.withReplicas(0),

  ruler_deployment+:
    deployment.mixin.spec.withReplicas(0),

  overrides_exporter_deployment+:
    deployment.mixin.spec.withReplicas(0),

  compactor_statefulset+:
    statefulSet.mixin.spec.withReplicas(0),
}

Ensure backend components (query-scheduler, compactor, ruler, store-gateway) correctly left their respective rings (Query-scheduler, Compactor, Ruler, Store-gateway).

It is now safe to disable ruler remote evaluation. (This needs to be done after the microservices ruler has been scaled down, otherwise rule evaluations may fail).

{
  _config+:: {
    ruler_remote_evaluation_enabled: false,
    autoscaling_ruler_querier_enabled: false,
  },
}

Step 6: Migrate write path to read-write deployment

Step 6.1: Scale up write component

Scale up mimir-write.

{
  _config+:: {
    mimir_write_replicas: 3,
  },
}

Step 6.2: Route traffic to mimir-write

Configure your load-balancer to route write requests to mimir-write.

Ensure the microservice distributor is no longer receiving write requests:

sum by (job) (rate(cortex_request_duration_seconds_count{job=~".*distributor.*", route=~"/distributor.Distributor/Push|/httpgrpc.*|api_(v1|prom)_push|otlp_v1_metrics"}[1m]))

Step 7: Scale down write microservices

Step 7.1: Scale down distributors

Set distributor replicas to 0:

{
  distributor_deployment+:
    deployment.mixin.spec.withReplicas(0),
}

Wait the next TSDB head compaction for ingesters (2 hours).

Step 7.2: Scale down ingesters

Warning
You must follow the shutdown ingester procedure to avoid data loss.

Follow the procedure for shutting down ingesters in ingester-zone-a.

Scale down zone-a replicas (this can be done before waiting for step 4 in the shutdown procedure):

{
  ingester_zone_a_statefulset+:
    statefulSet.mixin.spec.withReplicas(0),
}

Wait the required amount of time as per step 4 in the shutdown procedure.

Repeat shutting down and scaling down ingesters for zone-b and zone-c waiting the required amount of time between each zone.

Step 8: Final cleanup

Now that migration is complete you can clean up your deployment and jsonnet.

Changing deployment_mode to the final state of read-write will remove all of the microservice Kubernetes objects.

{
  _config+:: {
    deployment_mode: 'read-write',
  },
}

As the objects such as query_frontend_deployment are no longer defined, you’ll also need to remove the scaling that we did for those components. This is a good time to remove any other left over scaling or microservice configuration you may have set.

Lastly you can remove and release any now unused volumes from microservices. For example, to get a list of unused PVC’s:

kubectl get pvc --no-headers | grep -E '(ingester|store-gateway|compactor)' | awk '{print $1}'

Migrate query-scheduler from DNS-based to ring-based service discovery

Thu, 14 Aug 2025 18:42:07 +0000

Migrate query-scheduler from DNS-based to ring-based service discovery

The query-scheduler supports two service discovery mechanisms:

To migrate the query-scheduler from DNS-based to ring-based service discovery when your Mimir cluster is deployed using Jsonnet:

Configure the query-scheduler instances to join a ring, but keep the querier and query-frontend instances discovering query-schedulers via DNS:
jsonnet
```
{
  _config+:: {
    query_scheduler_service_discovery_mode: 'ring',
    query_scheduler_service_discovery_ring_read_path_enabled: false,
  }
}
```
Wait until query-scheduler changes have been applied.
Open the query-scheduler ring status page and ensure all query-scheduler instances are registered to the ring.

Configure query-frontend and querier instances to discover query-schedulers via the ring:

{
  _config+:: {
    query_scheduler_service_discovery_mode: 'ring',
    query_scheduler_service_discovery_ring_read_path_enabled: true,
  }
}

Deploy Grafana Mimir with Jsonnet and Tanka on Grafana Labs

Deploy Grafana Mimir with Jsonnet and Tanka

Deploy Grafana Mimir with Jsonnet and Tanka

Configure Grafana Mimir to use low resources with Jsonnet

Configure Grafana Mimir to use low resources with Jsonnet

Anti-affinity

Example: disable anti-affinity

Resources

Configure the Grafana Mimir object storage backend with Jsonnet

Configure the Grafana Mimir object storage backend with Jsonnet

Amazon S3 (s3) storage configuration options

Azure (azure) storage configuration options

Google Cloud Storage (gcs) storage configuration options

Configure the Grafana Mimir ruler with Jsonnet

Configure the Grafana Mimir ruler with Jsonnet

Operational modes

Migrate to remote evaluation

Configure Grafana Mimir autoscaling with Jsonnet

Configure Grafana Mimir autoscaling with Jsonnet

How KEDA works

KEDA in a nutshell

What happens if KEDA is unhealthy

How Kubernetes HPA works

How to enable autoscaling

How to disable autoscaling

Set minimum replicas = maximum replicas

Decommission HPA

Configure deployment mode

Configure deployment mode

Use Read-Write deployment mode

Requirements

Migrate from Consul to memberlist KV store for hash rings without downtime

Migrate from Consul to memberlist KV store for hash rings without downtime

Step 1: Enable memberlist and multi KV store.

Step 2: Enable KV store mirroring

Step 3: Switch Primary and Secondary store

Step 4: Disable mirroring to Consul

Step 5: Disable multi KV Store

Step 6: Cleanup

Migrate from microservices to read-write mode without downtime

Migrate from microservices to read-write mode without downtime

Step 1: Configure prerequisite zone-awareness

Step 2: Deploy read-write components with 0 replicas

Optionally double check configuration

Step 3: Migrate read-path to read-write service

Step 3.1: Scale up read component

Step 3.2: Check mimir-read is working

Step 3.3: Route traffic to mimir-read

Step 4: Migrate backend components to backend service

Step 4.1: Scale up backend component

Step 4.2: Check mimir-backend is working

Step 4.3: Route traffic to mimir-backend

Step 5: Scale down microservice read and backend components

Step 6: Migrate write path to read-write deployment

Step 6.1: Scale up write component

Step 6.2: Route traffic to mimir-write

Step 7: Scale down write microservices

Step 7.1: Scale down distributors

Step 7.2: Scale down ingesters

Step 8: Final cleanup

Migrate query-scheduler from DNS-based to ring-based service discovery

Migrate query-scheduler from DNS-based to ring-based service discovery

Amazon S3 (`s3`) storage configuration options

Azure (`azure`) storage configuration options

Google Cloud Storage (`gcs`) storage configuration options

Step 5: Disable `multi` KV Store