Menu

Important: This documentation is about an older version. It's relevant only to the release noted, many of the features and functions have been updated or replaced. Please view the current version.

Open source

Configure Monitoring and Alerting of Loki Using Grafana Cloud

This topic will walk you through using Grafana Cloud to monitor a Loki installation that is installed with the Helm chart. This approach leverages many of the chart’s self monitoring features, but instead of sending logs back to Loki itself, it sends them to a Grafana Cloud Logs instance. This approach also does not require the installation of the Prometheus Operator and instead sends metrics to a Grafana Cloud Metrics instance. Using Grafana Cloud to monitor Loki has the added benefit of being able to troubleshoot problems with Loki when the Helm installed Loki is down, as the logs will still be available in the Grafana Cloud Logs instance.

Before you begin:

Prequisites for Monitoring Loki:

You must setup the Grafana Kubernetes Integration following the instructions in Grafana Kubernetes Monitoring using Agent Flow as this will install necessary components for collecting metrics about your Kubernetes cluster and sending them to Grafana Cloud. Many of the dashboards installed as a part of the Loki integration rely on these metrics.

Walking through this installation will create two Grafana Agent configurations, one for metrics and one for logs, that will add the external label cluster: cloud. In order for the Dashboards in the self-hosted Grafana Loki integration to work, the cluster name needs to match your Helm installation name. If you installed Loki using the command helm install best-loki-cluster grafana/loki, you would need to change the cluster value in both Grafana Agent configurations from cloud to best-loki-cluster when setting up the Grafana Kubernetes integration.

To set up the Loki integration in Grafana Cloud:

  1. Get valid Push credentials for your Cloud Metrics and Cloud Logs instances.

  2. Create a secret in the same namespace as Loki to store your Cloud Logs credentials.

    bash
    cat <<'EOF' | NAMESPACE=loki /bin/sh -c 'kubectl apply -n $NAMESPACE -f -'
    apiVersion: v1
    data:
      password: <BASE64_ENCODED_CLOUD_LOGS_PASSWORD>
      username: <BASE64_ENCODED_CLOUD_LOGS_USERNAME>
    kind: Secret
    metadata:
      name: grafana-cloud-logs-credentials
    type: Opaque
    EOF
  3. Create a secret to store your Cloud Metrics credentials.

    bash
    cat <<'EOF' | NAMESPACE=loki /bin/sh -c 'kubectl apply -n $NAMESPACE -f -'
    apiVersion: v1
    data:
      password: <BASE64_ENCODED_CLOUD_METRICS_PASSWORD>
      username: <BASE64_ENCODED_CLOUD_METRICS_USERNAME>
    kind: Secret
    metadata:
      name: grafana-cloud-metrics-credentials
    type: Opaque
    EOF
  4. Enable monitoring metrics and logs for the Loki installation to be sent your cloud database instances by adding the following to your Helm values.yaml file:

    yaml
    ---
    monitoring:
      dashboards:
        enabled: false
      rules:
        enabled: false
      selfMonitoring:
        logsInstance:
          clients:
            - url: <CLOUD_METRICS_URL>
              basicAuth:
                username:
                  name: grafana-cloud-logs-credentials
                  key: username
                password:
                  name: grafana-cloud-logs-credentials
                  key: password
      serviceMonitor:
        metricsInstance:
          remoteWrite:
            - url: <CLOUD_LOGS_URL>
              basicAuth:
                username:
                  name: grafana-cloud-metrics-credentials
                  key: username
                password:
                  name: grafana-cloud-metrics-credentials
                  key: password
  5. Install the self-hosted Grafana Loki integration by going to your hosted Grafana instance, clicking the lightning bolt icon labeled Integrations and Connections, then search for and install the Self-hosted Grafana Loki integration.

  6. Once the self-hosted Grafana Loki integration is installed, click the View Dashboards button to see the installed dashboards.