Monitor Grafana Mimir on Grafana Labs

About Grafana Mimir dashboards and alerts requirements

Fri, 06 Mar 2026 22:21:21 +0000

About Grafana Mimir dashboards and alerts requirements

Grafana Mimir dashboards and alerts require certain labels to exist on metrics scraped from Grafana Mimir.

The mimir-distributed Helm chart provides metamonitoring support, which takes care of these labels. For more information about Helm chart metamonitoring, refer to Collect metrics and logs via the Helm chart. If you are using Helm chart metamonitoring, go to Installing Grafana Mimir dashboards and alerts.

If you are not, then continue reading. Your Prometheus or Grafana Agent must be configured to add these labels in order for the dashboards and alerts to function. The following table shows the required label names and whether they can be customized when compiling dashboards or alerts from sources.

Label name	Configurable?	Description
`cluster`	Yes	The Kubernetes cluster or datacenter where the Mimir cluster is running. You can configure the cluster label via the `per_cluster_label` field in the mixin configuration.
`namespace`	Yes	The Kubernetes namespace where the Mimir cluster is running. You can configure the namespace label via the `per_namespace_label` field in the mixin configuration.
`job`	Yes	A prefix (by default `<namespace>/`) followed by the Mimir component. When running in monolithic mode, make sure that the `<component>` is `mimir`. When running in microservices mode, make sure that the `<component>` is the name of the specific Mimir component (singular), such as `distributor`, `ingester`, or `store-gateway`. Similarly, in read-write mode, make sure the `<component>` is either `mimir-read`, `mimir-write`, or `mimir-backend`. In the mixin configuration, you can configure the prefix via the `job_prefix` field, the label name via the `per_job_label` field, and the regular expressions that are used to match components via the `job_names` field.
`pod`	Yes	The unique identifier of a Mimir replica, for example the Pod ID when running on Kubernetes. You can configure the instance label via the `per_instance_label` field in the mixin configuration.
`instance`	Yes	The unique identifier of the node or machine where the Mimir replica is running, for example the node when running on Kubernetes. You can configure the node label via the `per_node_label` field in the mixin configuration.

For rules and alerts to function properly, you must configure your Prometheus or Grafana Agent to scrape metrics from Grafana Mimir at an interval of 15s or shorter.

Deployment type

By default, Grafana Mimir dashboards assume Mimir is deployed in containers orchestrated by Kubernetes. If you’re running Mimir on baremetal, set the configuration field deployment_type: 'baremetal' and re-compile the dashboards.

Job selection

A metric could be exposed by multiple Grafana Mimir components, or even different applications running in the same namespace. To provide accurate dashboards and alerts, the job label (by default job) selects a metric from specific components. A job is a combination of a prefix and component. The default prefix is the Kubernetes namespace followed by a slash, for example <namespace>/ in <namespace>/ingester.

Pre-compiled dashboards and alerts are shipped with a default configuration. If you compile dashboards and alerts from source, you have the option to customize a) the label used for the job selection via the per_job_label field, b) the prefix expected in the label value, which you can omit by setting it to '' via the job_prefix field, and c) the regular expression used to select each Mimir component via the job_names field in the mixin configuration.

Default `job` selection in monolithic mode

When running Grafana Mimir in monolithic mode and using the pre-compiled dashboards and alerts, the job label should be set to <namespace>/mimir.

Default `job` selection in microservices mode

When running Grafana Mimir in microservices mode and using the pre-compiled dashboards and alerts, the job label should be set according to the following table.

Mimir service	Expected `job` label
Distributor	`<namespace>/distributor`
Ingester	`<namespace>/ingester`
Querier	`<namespace>/querier`
Ruler	`<namespace>/ruler`
Query-frontend	`<namespace>/query-frontend`
Query-scheduler	`<namespace>/query-scheduler`
Store-gateway	`<namespace>/store-gateway`
Compactor	`<namespace>/compactor`

Default `job` selection in read-write mode

When running Grafana Mimir in read-write mode and using the pre-compiled dashboards and alerts, set the job label accordingly:

Mimir service	Expected `job` label
Mimir Read	`<namespace>/mimir-read`
Mimir Write	`<namespace>/mimir-write`
Mimir Backend	`<namespace>/mimir-backend`

Additional resources metrics

The Grafana Mimir dashboards displaying CPU, memory, disk, and network resources utilization require Prometheus metrics scraped from the following endpoints:

For more information about the kubelet metrics and cAdvisor metrics exported by the kubelet, refer to Metrics For Kubernetes System Components.

Metrics from kubelet, kube-state-metrics, and cAdvisor must all have a cluster label with the same value as in the Mimir metrics.

Metrics from node_exporter must all have an instance label on them that has the same value as the instance label on Mimir metrics.

Log labels

The Slow queries dashboard uses a Loki data source with the logs from Grafana Mimir to visualize slow queries. The query-frontend component logs slow queries based on how you configured the -query-frontend.log-queries-longer-than parameter. These logs need to have specific labels in order for the dashboard to work.

Label name	Configurable?	Description
`cluster`	Yes	The Kubernetes cluster or datacenter where the Mimir cluster is running. You can configure the cluster label via the `per_cluster_label` field in the mixin configuration.
`namespace`	No	The Kubernetes namespace where the Mimir cluster is running.
`name`	No	Name of the component. For example, `query-frontend`.

Deploying the Grafana Mimir monitoring mixin

Fri, 06 Mar 2026 22:21:21 +0000

Deploying the Grafana Mimir monitoring mixin

Grafana Mimir exposes a /metrics endpoint returning Prometheus metrics. You can configure your Prometheus server to scrape Grafana Mimir or you can use the built-in functionality of the Helm chart to automatically send these metrics to a remote. The endpoint is exposed on the Mimir HTTP server address / port which can be customized through -server.http-listen-address and -server.http-listen-port CLI flags or their respective YAML config options.

Dashboards and alerts

Grafana Mimir is shipped with a comprehensive set of production-ready Grafana dashboards and alerts to monitor the state and health of a Mimir cluster.

Dashboards provide both a high-level and in-depth view of every aspect of a Grafana Mimir cluster. You can take a look at all the available dashboards in this overview.

Alerts allow you to monitor the health of a Mimir cluster. For each alert, we provide detailed runbooks to further investigate and fix the issue.

The requirements documentation lists prerequisites for using the Grafana Mimir dashboards and alerts.

The installation instructions show available options to install Grafana Mimir dashboards and alerts.

Installing Grafana Mimir dashboards and alerts

Fri, 06 Mar 2026 22:21:21 +0000

Installing Grafana Mimir dashboards and alerts

Grafana Mimir is shipped with a comprehensive set of production-ready Grafana dashboards and alerts to monitor the state and health of a Mimir cluster.

Requirements

Grafana Mimir dashboards and alerts require specific labels to be set by Prometheus or Grafana Agent when scraping your Mimir cluster metrics
Some dashboards require recording rules that you should install in your Prometheus

Install from package

Grafana Mimir provides ready to use Grafana dashboards in the .json format and Prometheus alerts in the .yaml format, that you can directly import into your Grafana installation and Prometheus config.

The packaged dashboards and alerts have been compiled from the sources using a default configuration and don’t allow you to customize the required metrics label names. If you need to customize the required metrics label names please choose one of the other installation options.

Download dashboards, recording rules and alerts from Grafana Mimir repository
Import dashboards in Grafana
Install recording rules and alerts in your Prometheus

Install from sources

Grafana Mimir dashboards and alerts are built using Jsonnet language and you can compile them from sources. If you choose this option, you can change the configuration to match your deployment, like customizing the required label names.

Checkout Mimir source code

git clone https://github.com/grafana/mimir.git

Review the mixin configuration at operations/mimir-mixin/config.libsonnet, and apply your changes if necessary.
Compile the mixin
Bash
```
make build-mixin
```
Import the dashboards saved at operations/mimir-mixin-compiled/dashboards/ in Grafana
Install the recording rules saved at operations/mimir-mixin-compiled/rules.yaml in your Prometheus
Install the alerts saved at operations/mimir-mixin-compiled/alerts.yaml in your Prometheus

Install dashboards from Jsonnet mixin

In case you’re already using Jsonnet to define your infrastructure as a code, you can vendor the Grafana Mimir mixin directly into your infrastructure repository and configure it overriding the _config fields. Given the exact setup really depends on a case-by-case basis, the following instructions are not meant to be prescriptive but just show the main steps required to vendor the mixin.

Initialize Jsonnet
Bash
```
jb init
```

Install Grafana Mimir mixin

jb install github.com/grafana/mimir/operations/mimir-mixin@main

Import and configure it

(import 'github.com/grafana/mimir/operations/mimir-mixin/mixin.libsonnet') + {
  _config+:: {
    // Override the Grafana Mimir mixin config here.
  },
}

Deploy mixin with Terraform

Based on Jsonnet configuration file, you can use Terraform to deploy the mixin in both Grafana and Grafana Mimir. To do so, you can use grafana/grafana, ovh/mixtool, and ovh/mimirtool providers.

Create a Terraform main.tf file:

# Specify which providers to use
terraform {
  required_version = ">= 1.3.5"
  required_providers {
    mixtool = {
      source  = "ovh/mixtool"
      version = "~> 0.1.1"
    }
    mimirtool = {
      source  = "ovh/mimirtool"
      version = "~> 0.1.1"
    }
    grafana = {
      source  = "grafana/grafana"
      version = "~> 1.32.0"
    }
  }
}

# Configure providers if needed
provider "grafana" {
  url  = "http://localhost:9000"
  auth = "admin:admin"
}

provider "mimirtool" {
  address   = "http://localhost:9009"
  tenant_id = "anonymous"
}

locals {
  mixin_source   = "custom.libsonnet"
  jsonnet_path   = "vendor"
}

# Build alerts
data "mixtool_alerts" "mimir" {
  source       = local.mixin_source
  jsonnet_path = [local.jsonnet_path]
}

# Build rules
data "mixtool_rules" "mimir" {
  source       = local.mixin_source
  jsonnet_path = [local.jsonnet_path]
}

# Build dashboards
data "mixtool_dashboards" "mimir" {
  source       = local.mixin_source
  jsonnet_path = [local.jsonnet_path]
}

# Deploy rules
resource "mimirtool_ruler_namespace" "rules" {
  namespace   = "rules_community"
  config_yaml = data.mixtool_rules.mimir.rules
}

# Deploy alerts
resource "mimirtool_ruler_namespace" "alerts" {
  namespace   = "alerts_community"
  config_yaml = data.mixtool_alerts.mimir.alerts
}

# Deploy dashboards
resource "grafana_dashboard" "mimir" {
  for_each    = data.mixtool_dashboards.mimir.dashboards
  config_json = each.value
}

Initialize Terraform:
Bash
```
terraform init
```
Review the changes that Terraform would apply to your infrastructure:
Bash
```
terraform plan
```
Deploy the changes:
Bash
```
terraform apply
```

Viewing Grafana Mimir dashboards

Fri, 06 Mar 2026 22:21:21 +0000

Viewing Grafana Mimir dashboards

Grafana Mimir provides the following production-ready dashboards.

Collecting metrics and logs from Grafana Mimir

Fri, 06 Mar 2026 22:21:21 +0000

Collecting metrics and logs from Grafana Mimir

You can collect logs and metrics from a Mimir or GEM cluster. To set up dashboards and alerts, see Installing Grafana Mimir dashboards and alerts or Grafana Cloud: Self-hosted Grafana Mimir integration.

It is easier and best to monitor a cluster if it was installed via the Grafana Mimir Helm chart. For more information, see the documentation for the Grafana Mimir Helm chart.

It is also possible to use this integration if Mimir was deployed another way. For more information, see Collect metrics and logs without the Helm chart.

Collect metrics and logs without the Helm chart

You can still use the dashboards and rules in the monitoring-mixin, even if Mimir or GEM is not deployed via the Helm chart. If you are not using the Helm chart, start by using the Agent configuration from Collect metrics and logs via Grafana Agent. You might need to modify it. For more information, see dashboards and alerts requirements.

Service discovery

The Agent configuration relies on Kubernetes service discovery and Pod labels to constrain the collected metrics and logs to ones that are strictly related to the Grafana Mimir deployment. If you are deploying Grafana Mimir on something other than Kubernetes, then replace the kubernetes_sd_configs block with a block from the Agent configuration that can discover the Mimir processes.

Collect metrics and logs via Grafana Agent

Set up a Grafana Agent that collects logs and metrics from Mimir or GEM. To set up Grafana Agent, see Set up Grafana Agent. After your Agent is deployed, use the example Agent configuration to configure the Agent to scrape Mimir or GEM.

Caveats

Managing your own Agent comes with some caveats:

You will have to keep the Agent configuration up to date manually as you update the Mimir Helm chart. While we will try to keep this article up to date, we cannot guarantee that the example Agent configuration will always work.
The static configuration makes some assumptions about the naming of the chart, such as that you have not overridden the fullnameOverride in the Helm chart.
The static configuration cannot be selective in the PersistentVolumes metrics it collects from Kubelet, so it will scrape metrics for all PersistentVolumes.
The static configuration hardcodes the value of the cluster label on all metrics and logs. This means that the configuration cannot account for multiple installations of the Helm chart.

If possible, upgrade the Mimir Helm chart to version 3.0 or higher and use the built-in Grafana Agent operator. Using the Agent operator allows the chart to automatically configure the Agent, eliminating the aforementioned caveats.

Example Agent configuration

In the following example Grafana Agent configuration file for collecting logs and metrics, replace url, password, and username in the logs and metrics blocks with the details of your Prometheus and Loki clusters.

logs:
  configs:
    - clients:
        - basic_auth:
            password: xxx
            username: xxx
          url: https://example.com/loki/api/v1/push
      name: integrations
      positions:
        filename: /tmp/positions.yaml
      scrape_configs:
        - job_name: integrations/grafana-mimir-logs
          kubernetes_sd_configs:
            - role: pod
          pipeline_stages:
            - cri: {}
          relabel_configs:
            - action: keep
              regex: mimir-distributed-.*
              source_labels:
                - __meta_kubernetes_pod_label_helm_sh_chart
            - source_labels:
                - __meta_kubernetes_pod_node_name
              target_label: __host__
            - action: replace
              replacement: $1
              separator: /
              source_labels:
                - __meta_kubernetes_namespace
                - __meta_kubernetes_pod_container_name
              target_label: job
            - action: replace
              regex: ""
              replacement: k8s-cluster
              separator: ""
              source_labels:
                - cluster
              target_label: cluster
            - action: replace
              source_labels:
                - __meta_kubernetes_namespace
              target_label: namespace
            - action: replace
              source_labels:
                - __meta_kubernetes_pod_name
              target_label: pod
            - action: replace
              source_labels:
                - __meta_kubernetes_pod_container_name
              target_label: name
            - action: replace
              source_labels:
                - __meta_kubernetes_pod_container_name
              target_label: container
            - replacement: /var/log/pods/*$1/*.log
              separator: /
              source_labels:
                - __meta_kubernetes_pod_uid
                - __meta_kubernetes_pod_container_name
              target_label: __path__
      target_config:
        sync_period: 10s
metrics:
  configs:
    - name: integrations
      remote_write:
        - basic_auth:
            password: xxx
            username: xxx
          url: https://example.com/api/prom/push
      scrape_configs:
        - job_name: integrations/grafana-mimir/kube-state-metrics
          kubernetes_sd_configs:
            - role: pod
          metric_relabel_configs:
            - action: keep
              regex: (.*-mimir-)?alertmanager.*|(.*-mimir-)?compactor.*|(.*-mimir-)?distributor.*|(.*-mimir-)?(gateway|cortex-gw|cortex-gw).*|(.*-mimir-)?ingester.*|(.*-mimir-)?querier.*|(.*-mimir-)?query-frontend.*|(.*-mimir-)?query-scheduler.*|(.*-mimir-)?ruler.*|(.*-mimir-)?store-gateway.*
              separator: ""
              source_labels:
                - deployment
                - statefulset
                - pod
          relabel_configs:
            - action: keep
              regex: kube-state-metrics
              source_labels:
                - __meta_kubernetes_pod_label_app_kubernetes_io_name
            - action: replace
              regex: ""
              replacement: k8s-cluster
              separator: ""
              source_labels:
                - cluster
              target_label: cluster
        - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
          job_name: integrations/grafana-mimir/kubelet
          kubernetes_sd_configs:
            - role: node
          metric_relabel_configs:
            - action: keep
              regex: kubelet_volume_stats.*
              source_labels:
                - __name__
          relabel_configs:
            - replacement: kubernetes.default.svc.cluster.local:443
              target_label: __address__
            - regex: (.+)
              replacement: /api/v1/nodes/${1}/proxy/metrics
              source_labels:
                - __meta_kubernetes_node_name
              target_label: __metrics_path__
            - action: replace
              regex: ""
              replacement: k8s-cluster
              separator: ""
              source_labels:
                - cluster
              target_label: cluster
          scheme: https
          tls_config:
            ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
            insecure_skip_verify: false
            server_name: kubernetes
        - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
          job_name: integrations/grafana-mimir/cadvisor
          kubernetes_sd_configs:
            - role: node
          metric_relabel_configs:
            - action: keep
              regex: (.*-mimir-)?alertmanager.*|(.*-mimir-)?compactor.*|(.*-mimir-)?distributor.*|(.*-mimir-)?(gateway|cortex-gw|cortex-gw).*|(.*-mimir-)?ingester.*|(.*-mimir-)?querier.*|(.*-mimir-)?query-frontend.*|(.*-mimir-)?query-scheduler.*|(.*-mimir-)?ruler.*|(.*-mimir-)?store-gateway.*
              source_labels:
                - pod
          relabel_configs:
            - replacement: kubernetes.default.svc.cluster.local:443
              target_label: __address__
            - regex: (.+)
              replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
              source_labels:
                - __meta_kubernetes_node_name
              target_label: __metrics_path__
            - action: replace
              regex: ""
              replacement: k8s-cluster
              separator: ""
              source_labels:
                - cluster
              target_label: cluster
          scheme: https
          tls_config:
            ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
            insecure_skip_verify: false
            server_name: kubernetes
        - job_name: integrations/grafana-mimir/metrics
          kubernetes_sd_configs:
            - role: pod
          relabel_configs:
            - action: keep
              regex: .*metrics
              source_labels:
                - __meta_kubernetes_pod_container_port_name
            - action: keep
              regex: mimir-distributed-.*
              source_labels:
                - __meta_kubernetes_pod_label_helm_sh_chart
            - action: replace
              regex: ""
              replacement: k8s-cluster
              separator: ""
              source_labels:
                - cluster
              target_label: cluster
            - action: replace
              source_labels:
                - __meta_kubernetes_namespace
              target_label: namespace
            - action: replace
              source_labels:
                - __meta_kubernetes_pod_name
              target_label: pod
            - action: replace
              source_labels:
                - __meta_kubernetes_pod_container_name
              target_label: container
            - action: replace
              separator: ""
              source_labels:
                - __meta_kubernetes_pod_label_name
                - __meta_kubernetes_pod_label_app_kubernetes_io_component
              target_label: __tmp_component_name
            - action: replace
              separator: /
              source_labels:
                - __meta_kubernetes_namespace
                - __tmp_component_name
              target_label: job
            - action: replace
              source_labels:
                - __meta_kubernetes_pod_node_name
              target_label: instance
  global:
    scrape_interval: 15s
  wal_directory: /tmp/grafana-agent-wal

Monitor Grafana Mimir on Grafana Labs

About Grafana Mimir dashboards and alerts requirements

About Grafana Mimir dashboards and alerts requirements

Deployment type

Job selection

Default job selection in monolithic mode

Default job selection in microservices mode

Default job selection in read-write mode

Additional resources metrics

Log labels

Deploying the Grafana Mimir monitoring mixin

Deploying the Grafana Mimir monitoring mixin

Dashboards and alerts

Installing Grafana Mimir dashboards and alerts

Installing Grafana Mimir dashboards and alerts

Requirements

Install from package

Install from sources

Install dashboards from Jsonnet mixin

Deploy mixin with Terraform

Viewing Grafana Mimir dashboards

Viewing Grafana Mimir dashboards

Collecting metrics and logs from Grafana Mimir

Collecting metrics and logs from Grafana Mimir

Collect metrics and logs without the Helm chart

Service discovery

Collect metrics and logs via Grafana Agent

Caveats

Example Agent configuration

Default `job` selection in monolithic mode

Default `job` selection in microservices mode

Default `job` selection in read-write mode