Monitoring on Grafana Labs

Configure monitoring and alerting

Wed, 06 Sep 2023 12:47:18 +0000

Configure monitoring and alerting

By default this Helm Chart configures meta-monitoring of metrics (service monitoring) and logs (self monitoring). This topic will walk you through configuring monitoring using a monitoring solution local to the same cluster where Loki is installed.

The ServiceMonitor resource works with either the Prometheus Operator or the Grafana Agent Operator, and defines how Loki’s metrics should be scraped. Scraping this Loki cluster using the scrape config defined in the SerivceMonitor resource is required for the included dashboards to work. A MetricsInstance can be configured to write the metrics to a remote Prometheus instance such as Grafana Cloud Metrics.

Self monitoring is enabled by default. This will deploy a GrafanaAgent, LogsInstance, and PodLogs resource which will instruct the Grafana Agent Operator (installed seperately) on how to scrape this Loki cluster’s logs and send them back to itself. Scraping this Loki cluster using the scrape config defined in the PodLogs resource is required for the included dashboards to work.

Rules and alerts are automatically deployed.

Before you begin:

Helm 3 or above. See Installing Helm.
A running Kubernetes cluster with a running Loki deployment.
A running Grafana instance.
A running Prometheus Operator installed using the kube-prometheus-stack Helm chart.

Prometheus Operator Prequisites

The dashboards require certain metric labels to display Kubernetes metrics. The best way to accomplish this is to install the kube-prometheus-stack Helm chart with the following values file, replacing CLUSTER_NAME with the name of your cluster. The cluster name is what you specify during the helm installation, so a cluster installed with the command helm install loki-cluster grafana/loki would be called loki-cluster.

kubelet:
  serviceMonitor:
    cAdvisorRelabelings:
      - action: replace
        replacement: <CLUSTER_NAME>
        targetLabel: cluster
      - targetLabel: metrics_path
        sourceLabels:
          - "__metrics_path__"
      - targetLabel: "instance"
        sourceLabels:
          - "node"

defaultRules:
  additionalRuleLabels:
    cluster: <CLUSTER_NAME>

"kube-state-metrics":
  prometheus:
    monitor:
      relabelings:
        - action: replace
          replacement: <CLUSTER_NAME>
          targetLabel: cluster
        - targetLabel: "instance"
          sourceLabels:
            - "__meta_kubernetes_pod_node_name"

"prometheus-node-exporter":
  prometheus:
    monitor:
      relabelings:
        - action: replace
          replacement: <CLUSTER_NAME>
          targetLabel: cluster
        - targetLabel: "instance"
          sourceLabels:
            - "__meta_kubernetes_pod_node_name"

prometheus:
  monitor:
    relabelings:
      - action: replace
        replacement: <CLUSTER_NAME>
        targetLabel: cluster

The kube-prometheus-stack installs ServiceMonitor and PrometheusRule resources for monitoring Kubernetes, and it depends on the kube-state-metrics and prometheus-node-exporter helm charts which also install ServiceMonitor resources for collecting kubelet and node-exporter metrics. The above values file adds the necessary additional labels required for these metrics to work with the included dashboards.

If you are using this helm chart in an environment which does not allow for the installation of kube-prometheus-stack or custom CRDs, you should run helm template on the kube-prometheus-stack helm chart with the above values file, and review all generated ServiceMonitor and PrometheusRule resources. These resources may have to be modified with the correct ports and selectors to find the various services such as kubelet and node-exporter in your environment.

To install the dashboards:

Dashboards are enabled by default. Set monitoring.dashboards.namespace to the namespace of the Grafana instance if it is in a different namespace than this Loki cluster.
Dashbards must be mounted to your Grafana container. The dashboards are in ConfigMaps named loki-dashboards-1 and loki-dashboards-2 for Loki, and enterprise-logs-dashboards-1 and enterprise-logs-dashboards-2 for GEL. Mount them to /var/lib/grafana/dashboards/loki-1 and /var/lib/grafana/dashboards/loki-2 in your Grafana container.

Create a dashboard provisioning file called dashboards.yaml in /etc/grafana/provisioning/dashboards of your Grafana container with the following contents (note: you may need to edit the orgId):

---
apiVersion: 1
providers:
  - disableDeletion: true
    editable: false
    folder: Loki
    name: loki-1
    options:
      path: /var/lib/grafana/dashboards/loki-1
    orgId: 1
    type: file
  - disableDeletion: true
    editable: false
    folder: Loki
    name: loki-2
    options:
      path: /var/lib/grafana/dashboards/loki-2
    orgId: 1
    type: file

To add add additional Prometheus rules:

Modify the configuration file values.yaml:

monitoring:
  rules:
    additionalGroups:
      - name: loki-rules
        rules:
          - record: job:loki_request_duration_seconds_bucket:sum_rate
            expr: sum(rate(loki_request_duration_seconds_bucket[1m])) by (le, job)
          - record: job_route:loki_request_duration_seconds_bucket:sum_rate
            expr: sum(rate(loki_request_duration_seconds_bucket[1m])) by (le, job, route)
          - record: node_namespace_pod_container:container_cpu_usage_seconds_total:sum_rate
            expr: sum(rate(container_cpu_usage_seconds_total[1m])) by (node, namespace, pod, container)

To disable monitoring:

Modify the configuration file values.yaml:

selfMonitoring:
  enabled: false

serviceMonitor:
  enabled: false

To use a remote Prometheus and Loki instance such as Grafana Cloud

Create a secrets.yaml file with credentials to access the Grafana Cloud services:

---
apiVersion: v1
kind: Secret
metadata:
  name: primary-credentials-metrics
  namespace: default
stringData:
  username: "<instance ID>"
  password: "<API key>"
---
apiVersion: v1
kind: Secret
metadata:
  name: primary-credentials-logs
  namespace: default
stringData:
  username: "<instance ID>"
  password: "<API key>"

Add the secret to Kubernetes with kubectl create -f secret.yaml.

Add a remoteWrite section to serviceMonitor in values.yaml:

monitoring:
...
  serviceMonitor:
    enabled: true
    ...
    metricsInstance:
      remoteWrite:
      - url: <metrics remote write endpoint>
        basicAuth:
          username:
            name: primary-credentials-metrics
            key: username
          password:
            name: primary-credentials-metrics
            key: password

Add a client to monitoring.selfMonitoring.logsInstance.clients:

monitoring:
---
selfMonitoring:
  enabled: true
  logsInstance:
    clients:
      - url: <logs remote write endpoint>
        basicAuth:
          username:
            name: primary-credentials-logs
            key: username
          password:
            name: primary-credentials-logs
            key: password
lokiCanary:
  enabled: false

Install the Loki meta-motoring connection on Grafana Cloud.

Configure monitoring and alerting of Loki using Grafana Cloud

Mon, 18 Sep 2023 12:10:25 +0000

Configure monitoring and alerting of Loki using Grafana Cloud

This topic will walk you through using Grafana Cloud to monitor a Loki installation that is installed with the Helm chart. This approach leverages many of the chart’s self monitoring features, but instead of sending logs back to Loki itself, it sends them to a Grafana Cloud Logs instance. This approach also does not require the installation of the Prometheus Operator and instead sends metrics to a Grafana Cloud Metrics instance. Using Grafana Cloud to monitor Loki has the added benefit of being able to troubleshoot problems with Loki when the Helm installed Loki is down, as the logs will still be available in the Grafana Cloud Logs instance.

Before you begin:

Helm 3 or above. See Installing Helm.
A Grafana Cloud account and stack (including Cloud Grafana, Cloud Metrics, and Cloud Logs)
Grafana Kubernetes Monitoring using Agent Flow configured for the Kubernetes cluster.
A running Loki deployment installed in that Kubernetes cluster via the Helm chart

Prequisites for Monitoring Loki:

You must setup the Grafana Kubernetes Integration following the instructions in Grafana Kubernetes Monitoring using Agent Flow as this will install necessary components for collecting metrics about your Kubernetes cluster and sending them to Grafana Cloud. Many of the dashboards installed as a part of the Loki integration rely on these metrics.

Walking through this installation will create two Grafana Agent configurations, one for metrics and one for logs, that will add the external label cluster: cloud. In order for the Dashboards in the self-hosted Grafana Loki integration to work, the cluster name needs to match your Helm installation name. If you installed Loki using the command helm install best-loki-cluster grafana/loki, you would need to change the cluster value in both Grafana Agent configurations from cloud to best-loki-cluster when setting up the Grafana Kubernetes integration.

To set up the Loki integration in Grafana Cloud:

Get valid Push credentials for your Cloud Metrics and Cloud Logs instances.

Create a secret in the same namespace as Loki to store your Cloud Logs credentials.

cat <<'EOF' | NAMESPACE=loki /bin/sh -c 'kubectl apply -n $NAMESPACE -f -'
apiVersion: v1
data:
  password: <BASE64_ENCODED_CLOUD_LOGS_PASSWORD>
  username: <BASE64_ENCODED_CLOUD_LOGS_USERNAME>
kind: Secret
metadata:
  name: grafana-cloud-logs-credentials
type: Opaque
EOF

Create a secret to store your Cloud Metrics credentials.

cat <<'EOF' | NAMESPACE=loki /bin/sh -c 'kubectl apply -n $NAMESPACE -f -'
apiVersion: v1
data:
  password: <BASE64_ENCODED_CLOUD_METRICS_PASSWORD>
  username: <BASE64_ENCODED_CLOUD_METRICS_USERNAME>
kind: Secret
metadata:
  name: grafana-cloud-metrics-credentials
type: Opaque
EOF

Enable monitoring metrics and logs for the Loki installation to be sent your cloud database instances by adding the following to your Helm values.yaml file:

---
monitoring:
  dashboards:
    enabled: false
  rules:
    enabled: false
  selfMonitoring:
    logsInstance:
      clients:
        - url: <CLOUD_LOGS_URL>
          basicAuth:
            username:
              name: grafana-cloud-logs-credentials
              key: username
            password:
              name: grafana-cloud-logs-credentials
              key: password
  serviceMonitor:
    metricsInstance:
      remoteWrite:
        - url: <CLOUD_METRICS_URL>
          basicAuth:
            username:
              name: grafana-cloud-metrics-credentials
              key: username
            password:
              name: grafana-cloud-metrics-credentials
              key: password

Install the self-hosted Grafana Loki integration by going to your hosted Grafana instance, selecting Connections from the Home menu, then search for and install the Self-hosted Grafana Loki integration.
Once the self-hosted Grafana Loki integration is installed, click the View Dashboards button to see the installed dashboards.