DocumentationGrafana LokiManageMonitor LokiInstall mixins

Install Loki mixins

Loki is instrumented to expose metrics about itself via the /metrics endpoint, designed to be scraped by Prometheus. Each Loki release includes a mixin. The Loki mixin provides a set of Grafana dashboards, Prometheus recording rules and alerts for monitoring Loki.

To set up monitoring using the mixin, you need to:

• Deploy an instance of Prometheus (or a Prometheus-compatible time series database, like Mimir) which can store Loki metrics.
• Deploy an agent, such as Grafana Alloy, or Grafana Agent, to scrape Loki metrics.
• Set up Grafana to visualize Loki metrics, by installing the dashboards.
• Install the recording rules and alerts into Prometheus using mimirtool.

This procedure assumes that you have set up Loki using the Helm chart.

Before you begin

To make full use of the Loki mixin, you’ll need the following running in your environment:

• Loki instance - A Loki instance which you want to monitor.
• Grafana - For visualizing logs and metrics (install on Kubernetes).
• Prometheus or Mimir - An instance of Prometheus or Mimir which will store metrics from Loki.

To scrape metrics from Loki, you can use Grafana Alloy or the OpenTelemetry Collector. This procedure provides examples only for Grafana Alloy.

If you have installed Loki using a Helm Chart, this documentation assumes that the Loki and Grafana instances are located on the same Kubernetes cluster.

Configure Alloy to scrape Loki metrics

Loki exposes Prometheus metrics from all of its components to allow meta-monitoring. To retrieve these metrics, you need to configure a suitable scraper. Grafana Alloy can collect metrics and act as a Prometheus scraper. To use this capability, you need to configure Alloy to scrape from all of the components.

To scrape metrics from Loki, follow these steps:

Install Grafana Alloy using the provided instructions for your platform.

• Standalone
• Kubernetes
• Docker

Add a configuration block to scrape metrics from your Loki component instances and forward to a Prometheus or Mimir instance.

• On Kubernetes, you can use the Alloy discovery.kubernetes component to discover Loki Pods to scrape metrics from.
• On non-Kubernetes deployments, you may use prometheus.scrape and an explicit list of targets to discover Loki instances to scrape.

For an example, see Collect and forward Prometheus metrics.

Configure Grafana

In your Grafana instance, you’ll need to create a Prometheus data source to visualize the metrics scraped from your Loki cluster.

Install Loki dashboards in Grafana

After Loki metrics are scraped by Grafana Alloy and stored in a Prometheus compatible time-series database, you can monitor Loki’s operation using the Loki mixin.

Each Loki release includes a mixin that includes:

• Relevant dashboards for overseeing the health of Loki as a whole, as well as its individual Loki components
• Recording rules that compute metrics that are used in the dashboards
• Alerts that trigger when Loki generates metrics that are outside of normal parameters

To install the mixins in Grafana and Mimir, the general steps are as follows:

1. Download the mixin dashboards from the Loki repository.

2. Import the dashboards in your Grafana instance.

3. Upload alerts.yaml and rules.yaml files to Prometheus or Mimir with mimirtool.

Download the loki-mixin dashboards

1. First, clone the Loki repository from Github:

   bash Copy

   git clone https://github.com/grafana/loki
   cd loki

2. Once you have a local copy of the repository, navigate to the production/loki-mixin-compiled-ssd directory.

   bash Copy

   cd production/loki-mixin-compiled-ssd

   OR, if you’re deploying Loki in microservices mode:

   bash Copy

   cd production/loki-mixin-compiled

This directory contains a compiled version of the alert and recording rules, as well as the dashboards.

Import the dashboards to Grafana

The dashboards directory includes the monitoring dashboards that can be installed into your Grafana instance. Refer to Import a dashboard in the Grafana documentation.

To create a folder:

1. Open your Grafana instance and select Dashboards.
2. Click the New button.
3. Select New folder from the New menu.
4. Name your folder, for example, “Loki Monitoring”.
5. Click Create.

To import a dashboard:

1. Open your Grafana instance and select Dashboards.
2. Click the New button.
3. Select Import from the New menu.
4. On the Import dashboard screen, select Upload dashboard JSON file.
5. Browse to production/loki-mixin-compiled-ssd/dashboards and select the dashboard to import. Or, drag the dashboard file, for example, loki-operational.json, onto the Upload area of the Import dashboard screen.
6. Select a folder in the Folder menu where you want to save the imported dashboard. For example, select “Loki Monitoring” created in the earlier steps.
7. Click Import.

The imported files are listed in the Loki Monitoring dashboard folder.

To view the dashboards in Grafana:

1. Select Dashboards in your Grafana instance.
2. Select Loki Monitoring, or the folder where you uploaded the imported dashboards.
3. Select any file in the folder to view the dashboard.

Add alerts and recording rules to Prometheus or Mimir

The rules and alerts need to be installed into a Prometheus instance, Mimir or a Grafana Enterprise Metrics cluster.

You can find the YAML files for alerts and rules in the following directories in the Loki repo:

For SSD mode: production/loki-mixin-compiled-ssd/alerts.yaml production/loki-mixin-compiled-ssd/rules.yaml

For microservices mode: production/loki-mixin-compiled/alerts.yaml production/loki-mixin-compiled/rules.yaml

You use mimirtool to load the mixin alerts and rules definitions into a Prometheus instance, Mimir or a Grafana Enterprise Metrics cluster.

1. Download mimirtool.

2. Using the details of a Prometheus instance or Mimir cluster, run the following command to load the recording rules:

   bash Copy

   mimirtool rules load --address=http://prometheus:9090 rules.yaml

   Or, for example if your Mimir cluster requires an API key, as is the case with Grafana Enterprise Metrics:

   bash Copy

   mimirtool rules load --id=<tenant-id> --address=http://<mimir-hostname>:<port> --key="<mimir-api key>" rules.yaml

3. To load alerts:

   bash Copy

   mimirtool alertmanager load --address=http://prometheus:9090 alerts.yaml

   or

   bash Copy

   mimirtool alertmanager load --id=<tenant-id> --address=http://<mimir-hostname>:<port> --key="<mimir-api key>" alerts.yaml

Refer to the mimirtool documentation for more information.