Configure Grafana Mimir Helm chart on Grafana Labs

Configure Grafana Mimir autoscaling with Helm

Fri, 31 Oct 2025 14:55:43 +0000

Configure Grafana Mimir autoscaling with Helm

Warning
Autoscaling support in the Helm chart is currently experimental. Use with caution in production environments and thoroughly test in a non-production environment first.

You can configure autoscaling for Mimir components using (Kubernetes Event-driven Autoscaling) KEDA and Kubernetes Horizontal Pod Autoscaler (HPA).

Before you begin

Ensure you have a running Mimir cluster deployed with Helm.
Verify you have the required permissions to modify Helm deployments.
Familiarize yourself with Kubernetes HPA concepts.

Prerequisites

To use autoscaling, you need:

KEDA installed in your Kubernetes cluster
Prometheus metrics available for scaling decisions

Warning
Don’t use the same Mimir cluster for storing and querying autoscaling metrics. Using the same cluster can create a dangerous feedback loop.

For instance, if the Mimir cluster becomes unavailable, autoscaling stops working, because it cannot query the metrics. This prevents the cluster from automatically scaling up during high load or recovery. This inability to scale further exacerbates the cluster’s unavailability, which might, in turn, prevent the cluster from recovering.

Instead, use a separate Prometheus instance or a different metrics backend for autoscaling metrics.

Supported components

The Mimir Helm chart supports autoscaling for the following components:

About KEDA

KEDA is a Kubernetes operator that simplifies the setup of HPA with custom metrics from Prometheus. It consists of:

An operator and external metrics server
Support for multiple metric sources, including Prometheus
Custom resources (ScaledObject) that define scaling parameters
Automatic HPA resource management

For more information, refer to the KEDA documentation.

Configure autoscaling for a new installation

Follow these steps to enable autoscaling when deploying Mimir for the first time.

Steps

Configure the Prometheus metrics source in your values file:

kedaAutoscaling:
  prometheusAddress: "http://prometheus.monitoring:9090"
  pollingInterval: 10

Enable and configure autoscaling for desired components:

querier:
  kedaAutoscaling:
    enabled: true
    minReplicaCount: 2
    maxReplicaCount: 10

Deploy Mimir using Helm:

helm upgrade --install mimir grafana/mimir-distributed -f values.yaml

Expected outcome

After deployment:

KEDA creates ScaledObject resources for configured components.
HPA resources are automatically created and begin monitoring metrics.
Components scale based on configured thresholds and behaviors.

Migrate existing deployments to autoscaling

Follow these steps to enable autoscaling for an existing Mimir deployment.

Warning
Autoscaling support in the Helm chart is currently experimental. Migrating to autoscaling carries risks for cluster availability.

Enabling autoscaling removes the replicas field from deployments. If KEDA/HPA hasn’t started autoscaling a deployment yet, Kubernetes interprets no replicas as meaning 1 replica. This can cause an outage if the transition is not handled carefully. If you’re using GitOps tools like FluxCD or ArgoCD, you might need to take additional steps to manage the transition.

Consider testing the migration in a non-production environment first.

Before you begin

Back up your current Helm values.
Plan for potential service disruption.
Consider testing in a non-production environment first.
Ensure you have a rollback plan ready.
Consider migrating one component at a time to minimize risk.

Steps

Add the autoscaling configuration with preserveReplicas enabled:

querier:
  kedaAutoscaling:
    enabled: true
    preserveReplicas: true # Maintains stability during migration
    # ... autoscaling configuration ...

Apply the changes and verify the KEDA setup:

# Apply changes
helm upgrade mimir grafana/mimir-distributed -f values.yaml

# Verify setup
kubectl get hpa
kubectl get scaledobject
kubectl describe hpa

Wait 2-3 polling intervals to confirm that KEDA is managing scaling.

Remove preserveReplicas.

querier:
  kedaAutoscaling:
    enabled: true
    # Remove preserveReplicas

Apply the updated configuration:

helm upgrade mimir grafana/mimir-distributed -f values.yaml

Troubleshooting

If pods scale down to 1 replica after removing preserveReplicas:

Revert changes:

querier:
  kedaAutoscaling:
    enabled: true
    preserveReplicas: true

Verify KEDA setup:
- Check HPA status
- Verify metrics are being received
- Check for conflicts with other tools
- Ensure enough time was given for KEDA to take control (at least 2-3 polling intervals)
Try migrating again after resolving issues.

Note
If you’re using GitOps tools like FluxCD or ArgoCD, they might try to reconcile the state and conflict with HPA’s scaling decisions. Consult your GitOps tool’s documentation for handling HPA transitions.

Monitor autoscaling health

The following conditions indicate unhealthy autoscaling:

KEDA operator is down: ScaledObject changes don’t propagate to HPA.
KEDA metrics server is down: HPA can’t receive updated metrics.
HPA is unable to scale: MimirAutoscalerNotActive alert fires.

For production deployments, configure high availability for KEDA.

For more information about monitoring autoscaling, refer to Monitor Grafana Mimir.

Configure Grafana Mimir to allow Vault Agent to inject certificates and keys into Pods

Fri, 31 Oct 2025 14:55:43 +0000

Configure Grafana Mimir to allow Vault Agent to inject certificates and keys into Pods

When you enable this feature, Helm updates the annotations on all Pods that have TLS (transport layer security) configurable components. When you deploy Mimir, Vault Agent fetches the relevant secrets from Vault according to the annotations and mounts them to the Pod.

Note: Vault and Vault Agent must to be running already.

Example values.yaml file:

vaultAgent:
  enabled: true
  roleName: "test-role"
  clientCertPath: "client/cert/path"
  clientKeyPath: "client/key/path"
  serverCertPath: "server/cert/path"
  serverKeyPath: "server/key/path"
  caCertPath: "ca/cert/path"

Generated Kubernetes Deployment YAML file based on the preceding example:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: release-name
  labels:
    helm.sh/chart: mimir-release-1.2.3
    app.kubernetes.io/name: mimir
    app.kubernetes.io/instance: release-name
    app.kubernetes.io/component: comonent-name
    app.kubernetes.io/version: "123"
    app.kubernetes.io/managed-by: Helm
  annotations: {}
  namespace: "default"
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: mimir
      app.kubernetes.io/instance: release-name
      app.kubernetes.io/component: component-name
  strategy:
    type: RollingUpdate
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/role: "test-role"
        vault.hashicorp.com/agent-inject-secret-client.crt: "client/cert/path"
        vault.hashicorp.com/agent-inject-secret-client.key: "client/key/path"
        vault.hashicorp.com/agent-inject-secret-server.crt: "server/cert/path"
        vault.hashicorp.com/agent-inject-secret-server.key: "server/key/path"
        vault.hashicorp.com/agent-inject-secret-root.crt: "ca/cert/path"

vault.hashicorp.com/agent-inject-secret-<FILENAME>: '<PATH>' tells Vault Agent where to find the secret, and the name of the file to write the secret to. For example: vault.hashicorp.com/agent-inject-secret-client.crt: 'client/cert/path' will look for the secret at the path client/cert/path within Vault, and mount this secret to the pod as client.crt in the /vault/secrets/ directory.

For more information about Vault and Vault Agent, see Injecting Vault Secrets Into Kubernetes Pods via a Sidecar.

To configure TLS in Mimir, refer to Securing Grafana Mimir communications with TLS.

Configure native histograms

Fri, 31 Oct 2025 14:55:43 +0000

Configure native histograms

To enable support for ingesting Prometheus native histograms over the remote write API endpoint, set the configuration parameter native_histograms_ingestion_enabled to true.

To enable support for querying native histograms together with Grafana Mimir query sharding, set the configuration parameter query_result_response_format to protobuf.

Example values file:

mimir:
  structuredConfig:
    frontend:
      query_result_response_format: protobuf
    limits:
      native_histograms_ingestion_enabled: true

Note
Native histograms are an experimental feature of Grafana Mimir.

To configure bucket limits for native histograms, refer to Configure native histograms.

To configure Grafana Agent or Prometheus to write native histograms to Grafana Mimir, refer to Send native histograms to Mimir.

To visualize native histograms in Mimir, refer to Visualize native histograms.