Monitor the health of your system
You can monitor Grafana Mimir or Grafana Enterprise Metrics itself, by collecting metrics and logs from Mimir or GEM that is running on a Kubernetes cluster. This is called metamonitoring.
Note: In Grafana, you can create dashboards and receive alerts about those metrics and logs. To set up dashboards and alerts, see Installing Grafana Mimir dashboards and alerts or Grafana Cloud: Self-hosted Grafana Mimir integration.
Alternatively, to monitor the health of your system without using the Helm chart, see Collect metrics and logs without the Helm chart.
Configure the Grafana Agent operator via the Helm chart
In the Helm chart, you can configure where to send metrics and logs. You can send metrics to a Prometheus-compatible server and logs to a Loki cluster. The Helm chart can also scrape additional metrics from kube-state-metrics, kubelet, and cAdvisor.
The Helm chart does not collect Prometheus node_exporter metrics; metrics from node_exporter must all have an instance label on them that has the same value as the instance label on Mimir metrics.
You can configure your collection of metrics and logs by using the Grafana Agent operator. The Helm chart can install and use the Grafana Agent operator.
Note: Before the Helm chart can use the operator, you need to manually install all of the Kubernetes Custom Resource Definitions (CRDs) from the Grafana Agent operator YAML files.
It’s best to use the Grafana Agent operator for metrics and logs collection. However, if you prefer not to use it or you already have an existing Grafana Agent that you want to use, see Collect metrics and logs via Grafana Agent documentation in Grafana Mimir version 2.5.0.
Store credentials in a Secret:
If Prometheus and Loki are running without authentication, then you scan skip this section. Metamonitoring supports multiple ways of authentication for metrics and logs. If you are using a secret such as an API key to authenticate with Prometheus or Loki, then you need to create a Kubernetes Secret with that secret.
This is an example Kubernetes Secret:
apiVersion: v1 kind: Secret metadata: name: metamonitoring-credentials data: prometheus-api-key: FAKEACCESSKEY loki-api-key: FAKESECRETKEY
For information about how to create a Kubernetes Secret, see Creating a Secret.
Configure Helm chart values:
Merge the following YAML configuration into your Helm values file, and replace the values for
passwordSecretKeywith the details of the Prometheus and Loki clusters, and the Secret that you created. If your Prometheus and Loki servers are running without authentication, then remove the
authblocks from the YAML below.
If you already have the Agent operator installed in your Kubernetes cluster, then set
metaMonitoring: serviceMonitor: enabled: true grafanaAgent: enabled: true installOperator: true logs: remote: url: "https://example.com/loki/api/v1/push" auth: username: "12345" passwordSecretName: "metamonitoring-credentials" passwordSecretKey: "prometheus-api-key" metrics: remote: url: "https://example.com/api/v1/push" auth: username: "54321" passwordSecretName: "metamonitoring-credentials" passwordSecretKey: "loki-api-key" scrapeK8s: enabled: true kubeStateMetrics: namespace: kube-system labelSelectors: app.kubernetes.io/name: kube-state-metrics
Send metrics back into Mimir or GEM
You can also send the metrics that you collected (metamonitoring metrics) back into Mimir or GEM itself rather than sending them elsewhere.
When you leave the
metamonitoring.grafanaAgent.metrics.remote.url field empty,
then the chart automatically fills in the address of the GEM gateway Service
or the Mimir NGINX Service.
If you have deployed Mimir, and
metamonitoring.grafanaAgent.metrics.remote.url is not set,
then the metamonitoring metrics are be sent to the Mimir cluster.
You can query these metrics using the HTTP header X-Scope-OrgID: metamonitoring
If you have deployed GEM, then there are two alternatives:
If are using the
trustauthentication type (
mimir.structuredConfig.auth.type=trust), then the same instructions apply as for Mimir.
If you are using the enterprise authentication type (
mimir.structuredConfig.auth.type=enterprise, which is also the default when
enterprise.enabled=true), then you also need to provide a Secret with the authentication token for the tenant.The token should be to an access policy with
metrics:writescope. Assuming you are using the GEM authentication model, the Helm chart values should look like the following example.
metaMonitoring: serviceMonitor: enabled: true grafanaAgent: enabled: true installOperator: true metrics: remote: auth: username: metamonitoring passwordSecretName: gem-tokens passwordSecretKey: metamonitoring