Enabling multi-cluster support (optional)

The Kube Prometheus Helm chart offers some support for multi-cluster use cases. You may need to adjust some dashboard panels, recording rules, and alerting rules for your specific architecture. These adjustments go beyond the scope of this guide.

Add a cluster label to rule aggregations

If you want to modify recording and alerting rules to aggregate metrics over a cluster label, you can use cortex-tools’s rules prepare command to add a cluster label to a set of recording and alerting rules. You can modify the aggregation label using the -l flag. The default label is cluster. If you use this command, ensure that you’ve set the corresponding aggregation label using external_labels in your Prometheus configuration (in this guide, we’ve already set this to cluster: test).

Extract the rules from the Prometheus container as demonstrated in Step 1 of Importing Recording and Alerting rules and run the following command in the directory containing the rules files:

mimirtool rules prepare -i *.yaml

The -i flag will modify the rules files in place. You can omit this flag to test the command. This command modifies many rules to keep or add a cluster label to aggregations, which enables multi-cluster support.

You will then need to load the rules into Grafana Cloud Alerting. To learn more, see Step 2 of Importing Recording and Alerting rules.

Enable dashboard multi-cluster support

Modify your values.yaml Helm chart configuration file to enable multi-cluster support for Grafana dashboards:

. . .
grafana:
  sidecar:
    dashboards:
      multicluster: true

Roll out the changes using helm upgrade:

helm upgrade -f values.yaml your_release_name prometheus-community/kube-prometheus-stack

To learn more, please see the Kube Prometheus Helm chart’s values.yaml file.

You will need to repeat Step 3 of Migrating a Kube-Prometheus Helm stack to Grafana Cloud to import these modified dashboards to Grafana Cloud.