Enable multi-cluster support (optional)
The Kube Prometheus Helm chart offers some support for multi-cluster use cases. You may need to adjust some dashboard panels, recording rules, and alerting rules for your specific architecture. These adjustments go beyond the scope of this guide.
Add a cluster
label to rule aggregations
If you want to modify recording and alerting rules to aggregate metrics over a cluster
label, you can use cortex-tools’s rules prepare
command to add a cluster
label to a set of recording and alerting rules. You can modify the aggregation label using the -l
flag. The default label is cluster
. If you use this command, ensure that you’ve set the corresponding aggregation label using external_labels
in your Prometheus configuration (in this guide, we’ve already set this to cluster: test
).
Extract the rules from the Prometheus container as demonstrated in Step 1 of Importing Recording and Alerting rules and run the following command in the directory containing the rules files:
mimirtool rules prepare -i *.yaml
The -i
flag will modify the rules files in place. You can omit this flag to test the command. This command modifies many rules to keep or add a cluster
label to aggregations, which enables multi-cluster support.
You will then need to load the rules into Grafana Cloud Alerting. To learn more, see Step 2 of Importing Recording and Alerting rules.
Enable dashboard multi-cluster support
Modify your values.yaml
Helm chart configuration file to enable multi-cluster support for Grafana dashboards:
. . .
grafana:
sidecar:
dashboards:
multicluster: true
Roll out the changes using helm upgrade
:
helm upgrade -f values.yaml your_release_name prometheus-community/kube-prometheus-stack
To learn more, see the Kube Prometheus Helm chart’s values.yaml
file.
You will need to repeat Step 3 of Migrating a Kube-Prometheus Helm stack to Grafana Cloud to import these modified dashboards to Grafana Cloud.