Loki meta-monitoring on Grafana Labs

Deploy Loki meta-monitoring

Thu, 09 Apr 2026 02:28:18 +0000

Deploy Loki meta-monitoring

The primary method for collecting and monitoring a Loki cluster is to use the Kubernetes Monitoring Helm chart. This chart provides a comprehensive monitoring solution for Kubernetes clusters and includes direct integrations for monitoring the full LGTM (Loki, Grafana, Tempo, and Mimir) stack. This procedure will walk you through deploying the Kubernetes Monitoring Helm chart to monitor your Loki cluster.

Note
We recommend running a production cluster of Loki in distributed mode using Kubernetes. This procedure assumes you have a running Kubernetes cluster and a running Loki deployment. There are other methods for deploying Loki, such as using Docker or VM installations. meta-monitoring is still possible when using these deployment methods but not covered in this procedure.

Prerequisites

kubectl
Helm 3 or above. See Installing Helm.
A running Kubernetes cluster with a running Loki deployment.
A Grafana Cloud account or a separate LGTM stack for monitoring.

Preparing your environment

Before deploying the Kubernetes Monitoring Helm chart, you need to set up several components in your environment.

Add the Grafana Helm repository:

helm repo add grafana https://grafana.github.io/helm-charts

Update your Helm repositories:
Bash
```
helm repo update
```
Create a namespace for the monitoring stack:
Bash
```
kubectl create namespace meta
```

Authentication

The Kubernetes Monitoring Helm chart requires a Grafana Cloud account or a separate LGTM stack for monitoring. You will need to provide the necessary credentials to the Helm chart to authenticate with your Grafana Cloud account or LGTM stack. In this procedure, we will use Grafana Cloud as an example.

Create a new Cloud Access Policy in Grafana Cloud.
1. Sign into Grafana Cloud.
2. In the main menu, select Security > Access Policies.
3. Click Create access policy.
4. Give the policy a Name and select the following permissions:
  - Metrics: Write
  - Logs: Write
5. Click Create.
6. Click Add Token. Give the token a name and click Create. Save the token for later use.
Collect url and username for Prometheus and Loki.
1. Navigate to the Grafana Cloud Portal Overview page.
2. Click the Details button for your Prometheus instance.
  1. From the Sending metrics using Grafana Alloy section, collect the instance username and url.
  2. Navigate back to the Overview page.
3. Click the Details button for your Loki instance.
  1. From the Sending Logs to Grafana Cloud using Grafana Alloy section, collect the instance username and url.
  2. Navigate back to the Overview page.

Create the Kubernetes Secrets with the collected credentials from Grafana Cloud.

kubectl create secret generic metrics -n meta \
 --from-literal=username=<PROMETHEUS-USER> \
 --from-literal=password=<CLOUD-TOKEN>

kubectl create secret generic logs -n meta \
 --from-literal=username=<LOKI-USER> \
 --from-literal=password=<CLOUD-TOKEN>

Note that the Kubernetes Monitoring Helm supports many different authentication methods based upon your requirements including:

Bearer Tokens
OAuth2
SigV4
External Secrets

For further information on how to configure these methods, see the Kubernetes Monitoring Helm examples.

Deploy the Kubernetes Monitoring Helm chart

Now that you have prepared your environment and collected the necessary credentials, you can deploy the Kubernetes Monitoring Helm chart to monitor your Loki cluster. To do this we need to copy a values.yaml file to our local machine and modify it to include the necessary configuration.

Download the values.yaml file from the Kubernetes Monitoring Helm chart repository:

curl -O https://raw.githubusercontent.com/grafana/loki/main/production/helm/meta-monitoring/values.yaml

Open the values.yaml file in a text editor of your choosing and add the Prometheus and Loki endpoints.

destinations:
  - name: prometheus
    type: prometheus
    url: https://<PROMETHEUS-ENDPOINT>/api/prom/push
    auth:
      type: basic
      usernameKey: username
      passwordKey: password
    secret:
      create: false
      name: metrics
      namespace: meta

  - name: loki
    type: loki
    url: https://<LOKI-ENDPOINT>/loki/api/v1/push
    auth:
      type: basic
      usernameKey: username
      passwordKey: password
    secret:
      create: false
      name: logs
      namespace: meta

(Optional) Update the cluster name to a human-readable name to identify your cluster in Grafana Cloud.

# Global Label to be added to all telemetry data. Should reflect a recognizable name for the cluster.
cluster:
  name: loki-meta-monitoring-cluster

The default values file assumes that you have deployed Loki in the loki namespace and will deploy the Kubernetes monitoring stack in the meta namespace. If you have deployed Loki in a different namespace, you will need to update namespaces in the values.yaml file to match the namespace where Loki is deployed. Here is an example:
YAML
```
namespaces:
    - loki
```
Note if you would like to collect from all available namespaces, you can remove the namespaces key from the values.yaml file.
Deploy the Kubernetes Monitoring Helm chart using the modified values.yaml file:
Bash
```
helm install meta-loki grafana/k8s-monitoring \
 --namespace meta \
 -f values.yaml
```

Verify that the Kubernetes Monitoring Helm chart has been deployed successfully:

 kubectl get pods -n meta

You should see a list of pods running in the meta namespace.

NAME                                           READY   STATUS    RESTARTS ...        
meta-loki-alloy-singleton-6d7f8d8b86-sg4wx     2/2     Running   0        ...       
meta-loki-kube-state-metrics-64bdcfcbd-5snqz   1/1     Running   0        ...       
meta-loki-node-exporter-855l5                  1/1     Running   0        ...       
meta-loki-node-exporter-b976b                  1/1     Running   0        ...       
meta-loki-node-exporter-vsm4s                  1/1     Running   0        ...

Next Steps

You have successfully deployed the Kubernetes Monitoring Helm chart to monitor your Loki cluster. You can now move onto the next step of deploying the Loki mixin to visualize the metrics and logs from your Loki cluster. For more information, see Install the Loki Mixin.

Install dashboards, alerts, and recording rules

Thu, 09 Apr 2026 02:28:18 +0000

Install dashboards, alerts, and recording rules

Loki is instrumented to expose metrics about itself via the /metrics endpoint, designed to be scraped by Prometheus. Each Loki release includes a mixin. The Loki mixin provides a set of Grafana dashboards, Prometheus recording rules and alerts for monitoring Loki.

To set up monitoring using the mixin, you need to:

Deploy the Kubernetes Monitoring Helm chart. Follow the instructions in the Deploy Loki Meta-monitoring documentation.
Be actively storing metrics from your Loki cluster in Grafana Cloud or a separate LGTM stack.

This procedure assumes that you have set up Loki using the Helm chart.

Note
Be sure to update the commands and configuration to match your own deployment.

Install Loki dashboards in Grafana

After Loki metrics are scraped by the Kubernetes Monitoring Helm chart and stored in a Prometheus compatible time-series database, you can monitor Loki’s operation using the Loki mixin.

Each Loki release includes a mixin that includes:

Relevant dashboards for overseeing the health of Loki as a whole, as well as its individual Loki components
Recording rules that compute metrics that are used in the dashboards
Alerts that trigger when Loki generates metrics that are outside of normal parameters

To install the mixins in Grafana and Mimir, the general steps are as follows:

Download the mixin dashboards from the Loki repository.
Import the dashboards in your Grafana instance.
Upload alerts.yaml and rules.yaml files to Prometheus or Mimir with mimirtool.

Download the `loki-mixin` dashboards

First, clone the Loki repository from Github:
Bash
```
git clone https://github.com/grafana/loki
cd loki
```
Once you have a local copy of the repository, navigate to the production/loki-mixin-compiled directory.
Bash
```
cd production/loki-mixin-compiled
```

This directory contains a compiled version of the alert and recording rules, as well as the dashboards.

Note
If you want to change any of the mixins, make your updates in the production/loki-mixin directory. Use the instructions in the README in that directory to regenerate the files.

Import the dashboards to Grafana

The dashboards directory includes the monitoring dashboards that can be installed into your Grafana instance. Refer to Import a dashboard in the Grafana documentation.

Tip
Install all dashboards. You can only import one dashboard at a time. Create a new folder in the Dashboards area, for example “Loki Monitoring”, as an easy location to save the imported dashboards.

To create a folder:

Open your Grafana instance and select Dashboards.
Click the New button.
Select New folder from the New menu.
Name your folder, for example, “Loki Monitoring”.
Click Create.

To import a dashboard:

Open your Grafana instance and select Dashboards.
Click the New button.
Select Import from the New menu.
On the Import dashboard screen, select Upload dashboard JSON file.
Browse to production/loki-mixin-compiled/dashboards and select the dashboard to import. Or, drag the dashboard file, for example, loki-operational.json, onto the Upload area of the Import dashboard screen.
Select a folder in the Folder menu where you want to save the imported dashboard. For example, select “Loki Monitoring” created in the earlier steps.
Click Import.

The imported files are listed in the Loki Monitoring dashboard folder.

To view the dashboards in Grafana:

Select Dashboards in your Grafana instance.
Select Loki Monitoring, or the folder where you uploaded the imported dashboards.
Select any file in the folder to view the dashboard.

Add alerts and recording rules to Prometheus or Mimir

The rules and alerts need to be installed into a Prometheus instance, Mimir or a Grafana Enterprise Metrics cluster.

You can find the YAML files for alerts and rules in the following directories in the Loki repo:

For microservices mode:

production/loki-mixin-compiled/alerts.yaml (Optional)
production/loki-mixin-compiled/rules.yaml (Required)

You use mimirtool to load the mixin alerts and rules definitions into a Prometheus instance, Mimir, or a Grafana Enterprise Metrics cluster. The following examples show how to load the mixin alerts and rules into a Grafana Cloud instance.

Download mimirtool.
Export the authentication credentials for connecting to your Grafana Cloud Mimir instance.
Bash
```
export MIMIR_ADDRESS=<CLOUD-MIMIR-URL>
export MIMIR_API_USER=<CLOUD-MIMIR-USER>
export MIMIR_API_KEY=<CLOUD-MIMIR-API-KEY>
export MIMIR_TENANT_ID=<CLOUD-MIMIR-USER> # Same as MIMIR_API_USER when using Grafana Cloud
```
The best place to locate these credentials is to:
1. Sign into Grafana Cloud and create a new access policy.
  1. In the main menu, select Security > Access Policies.
  2. Click Create access policy.
  3. Give the policy a Name and select the following permissions:
    - Alerts: Write & Read
    - Rules: Write & Read
  4. Click Create.
  5. Click Add Token. Give the token a name and click Create.
2. Collect URL and user for Prometheus
  1. Navigate to the Grafana Cloud Portal Overview page.
  2. Click the Details button for your Alerts instance.
  3. From the Configuring your Alerting Stacks section, collect the instance User and URL for Metrics Authentication Settings.
Using the same terminal we exported the Mimir environment variables into earlier, run the following command to load the recording rules:
Bash
```
mimirtool rules load rules.yaml
```
(Optional) Load the alert rules:
Bash
```
mimirtool rules load alerts.yaml
```

Refer to the mimirtool documentation for more information.

Next steps

After you have installed the Loki mixin dashboards, alerts, and recording rules, you can now monitor your production Loki cluster using Grafana. Make sure you review the mixins when you upgrade Loki to make sure you are using the latest version of the mixin.

You can now move onto:

Send Logs: Ready to start sending your own logs to Loki, there a several methods you can use. For more information, refer to send data.
Query Logs: LogQL is an extensive query language for logs and contains many tools to improve log retrival and generate insights. For more information see the Query section.
Alert: Lastly you can use the ruler component of Loki to create alerts based on log queries. For more information refer to Alerting.

Single binary meta-monitoring

Thu, 09 Apr 2026 02:28:18 +0000

Single binary meta-monitoring

Meta monitoring for monolithic mode deployments involves some additional configuration. This approach does not use the Kubernetes Monitoring Helm chart.

Metrics

Configure Alloy or Prometheus to scrape the Loki metrics endpoint, adding the additional labels that are expected by the mixin dashboards, alerts, and recording rules:

Alloy

prometheus.scrape "loki" {
  targets = [{
    __address__ = "localhost:3100",
    cluster     = "prod",
    namespace   = "default",
    job         = "default/loki-single-binary",
    pod         = "loki-single-binary",
    container   = "loki",
  }]
  forward_to = [prometheus.remote_write.default.receiver]
}

prometheus.remote_write "default" {
  endpoint {  
    url = "<PROMETHEUS_REMOTE_WRITE_URL>"
  }  
}

Prometheus

scrape_configs:
  - job_name: loki
    static_configs:
      - targets: ['localhost:3100']
        labels:
          cluster: prod
          namespace: default
          job: default/loki-single-binary
          pod: loki-single-binary
          container: loki

Mixins

To generate the mixins:

Install jb.
Install mixtool.

Clone the Loki repository from Github.

git clone https://github.com/grafana/loki
cd loki

Create the output directory.

mkdir loki/production/loki-mixin-compiled-single-binary

Navigate to the mixin directory.
Bash
```
cd loki/production/loki-mixin
```
Use jb to install mixin dependencies and generate the vender directory.
Bash
```
jb install
```

Create the mixin configuration file single-binary.libsonnet.

cat <<EOF > single-binary.libsonnet
local loki = import 'mixin.libsonnet';

loki + {
  _config+:: {
    meta_monitoring: {
      enabled: true,
    },
    canary+: {
      // Whether or not to include the loki-canary dashboard
      enabled: true
    },
    promtail: {
      // Whether or not to include promtail specific dashboards
      enabled: false,
    },
    // Tunes histogram recording rules to aggregate over this interval.
    // Set to at least twice the scrape interval; otherwise, recording rules will output no data.
    // Set to four times the scrape interval to account for edge cases: https://www.robustperception.io/what-range-should-i-use-with-rate/        
    recording_rules_range_interval: '5m'
  }  
}
EOF

Generate dashboards, alerts, and recording rules.

mixtool generate all \
  --output-alerts ../loki-mixin-compiled-single/alerts.yaml \
  --output-rules ../loki-mixin-compiled-single/rules.yaml \
  --directory ../loki-mixin-compiled-single/dashboards \
  single-binary.libsonnet

See the generated dashboards, alerts, and recording rules in the loki/production/loki-mixin-compiled-single-binary directory.
Follow the instructions in the Install Mixins documentation to set up the dashboards, alerts, and recordings.

Key metrics for monitoring Loki

Thu, 09 Apr 2026 02:28:18 +0000

Key metrics for monitoring Loki

Loki exposes many metrics, and each component behaves differently under load. This page focuses on the highest-signal metrics for detecting negative trends early.

Note
The example queries on this page are PromQL. Run them against the Prometheus-compatible data source where your Loki metrics are stored (for example, Prometheus, Mimir, or Grafana Cloud Metrics).

For setup and prebuilt dashboards and alerts, refer to:

Request error rate

Watch request failures first. A sustained increase in 5xx responses is usually the earliest sign of user-visible impact.

Key metric:

loki_request_duration_seconds_count (counter with labels including status_code, job, and route)

Example query:

100 * sum(rate(loki_request_duration_seconds_count{status_code=~"5.."}[2m])) by (cluster, namespace, job, route)
/
sum(rate(loki_request_duration_seconds_count[2m])) by (cluster, namespace, job, route)

Abnormal behavior:

Any sustained increase in 5xx ratio.
The Loki mixin alert LokiRequestErrors fires when this ratio is greater than 10% for 15 minutes.

Request latency (p99)

Latency degradation can appear before hard failures. Track p99 for read and write routes.

Key metric:

loki_request_duration_seconds_bucket (histogram buckets)

Example query:

histogram_quantile(0.99, sum(rate(loki_request_duration_seconds_bucket[1m])) by (le, cluster, namespace, job, route))

Abnormal behavior:

Rising p99 over time, especially in query-frontend and distributor paths.
The Loki mixin alert LokiRequestLatency fires when p99 exceeds 1 second for 15 minutes.

Panics

Panics are high-severity faults and should stay at zero.

Key metric:

loki_panic_total

Example query:

sum(increase(loki_panic_total[10m])) by (cluster, namespace, job)

Abnormal behavior:

Any value above zero. The Loki mixin alert LokiRequestPanics treats this as critical.

Discarded samples

Discarded samples indicate data that Loki rejected or dropped. This is one of the most important ingestion-quality signals.

Key metric:

loki_discarded_samples_total

Example query:

topk(10, sum by (tenant, reason) (rate(loki_discarded_samples_total{cluster="$cluster", namespace="$namespace"}[$__rate_interval])))

Abnormal behavior:

Increasing discard rate.
New or growing reason values (for example, tenant limits or stream limits).

Compaction health

Compaction issues can silently degrade read performance and retention behavior over time.

Note
The compaction and retention metrics use a loki_boltdb_shipper_ prefix for historical reasons. The compactor emits these metrics regardless of which index type you use, including TSDB.

Key metrics:

loki_boltdb_shipper_compactor_running
loki_boltdb_shipper_compact_tables_operation_last_successful_run_timestamp_seconds
loki_boltdb_shipper_compact_tables_operation_total
loki_boltdb_shipper_compact_tables_operation_duration_seconds

Example queries:

sum(loki_boltdb_shipper_compactor_running) by (cluster, namespace)

time() - (loki_boltdb_shipper_compact_tables_operation_last_successful_run_timestamp_seconds > 0)

Abnormal behavior:

More than one compactor running.
No successful compaction for multiple hours (the mixin alert threshold is 3 hours).

Ingester health and flush behavior

Ingester pressure often appears as memory growth, poor chunk utilization, or flush backlog.

Key metrics:

loki_ingester_memory_streams
loki_ingester_memory_chunks
loki_ingester_flush_queue_length
loki_ingester_chunk_utilization
loki_ingester_chunks_flushed_total

Example queries:

sum(loki_ingester_memory_streams{cluster="$cluster", namespace="$namespace"})

sum(loki_ingester_flush_queue_length{cluster="$cluster", namespace="$namespace"})

Abnormal behavior:

Persistent growth in in-memory streams or chunks.
Increasing flush queue length.
Low chunk utilization for long periods.

Distributor throughput

Throughput changes help identify upstream sender issues, sudden traffic shifts, or ingestion bottlenecks.

Key metrics:

loki_distributor_bytes_received_total
loki_distributor_lines_received_total

Example queries:

sum(rate(loki_distributor_bytes_received_total{cluster="$cluster", namespace="$namespace"}[$__rate_interval]))

sum(rate(loki_distributor_lines_received_total{cluster="$cluster", namespace="$namespace"}[$__rate_interval]))

Abnormal behavior:

Sharp drops (possible data path interruption).
Unexpected spikes (possible overload or noisy tenants).

Object store operations

Object store latency and failures directly impact query and retention workflows.

Key metrics:

loki_objstore_bucket_operations_total
loki_objstore_bucket_operation_failures_total
loki_objstore_bucket_operation_duration_seconds

Example queries:

sum by (operation) (rate(loki_objstore_bucket_operation_failures_total{cluster="$cluster", namespace="$namespace"}[$__rate_interval]))

histogram_quantile(0.99, sum(rate(loki_objstore_bucket_operation_duration_seconds_bucket{cluster="$cluster", namespace="$namespace"}[$__rate_interval])) by (le, operation))

Abnormal behavior:

Growing failure rates by operation.
Increasing p99 latency for get, get_range, or upload.

Resource and runtime health

Resource pressure can explain or predict service degradation before alert thresholds are crossed.

Common signals to track:

Container CPU usage
Container memory working set
Go heap in use
Disk read and write rates
Container restarts

Abnormal behavior:

Repeated restart spikes.
Sustained CPU saturation.
Memory growth without recovery.

Loki Canary (end-to-end data verification)

If you run Loki Canary, use it as an end-to-end correctness signal, not only a performance signal.

Key metrics:

loki_canary_missing_entries_total
loki_canary_spot_check_missing_entries_total
loki_canary_response_latency_seconds_bucket

Example query:

sum(increase(loki_canary_missing_entries_total{cluster=~"$cluster", namespace=~"$namespace"}[$__range]))
/
sum(increase(loki_canary_entries_total{cluster=~"$cluster", namespace=~"$namespace"}[$__range]))
* 100

Abnormal behavior:

Any non-zero missing rate sustained over time.

Internal error log rate

Internal logs provide fast context when metrics indicate degradation.

Key metric:

loki_internal_log_messages_total

Use this metric with component logs to correlate where failures begin.

Retention and sweeper progress

Retention and sweeper lag can cause storage growth and delayed data lifecycle actions.

Key metrics:

loki_compactor_apply_retention_last_successful_run_timestamp_seconds
loki_boltdb_shipper_retention_sweeper_marker_file_processing_current_time
loki_boltdb_shipper_retention_sweeper_chunk_deleted_duration_seconds_count

Example query:

time() - (loki_boltdb_shipper_retention_sweeper_marker_file_processing_current_time{cluster="$cluster", namespace="$namespace"} > 0)

Abnormal behavior:

Increasing sweeper lag.
Falling delete throughput or sustained delete failures.

Next steps

Install and keep the latest Loki mixin dashboards and alerts.
Build component-specific alerts on top of these baseline signals.
Pair metrics with Loki component logs, including metrics.go, during incident response.

Loki meta-monitoring on Grafana Labs

Deploy Loki meta-monitoring

Deploy Loki meta-monitoring

Prerequisites

Preparing your environment

Authentication

Deploy the Kubernetes Monitoring Helm chart

Next Steps

Install dashboards, alerts, and recording rules

Install dashboards, alerts, and recording rules

Install Loki dashboards in Grafana

Download the loki-mixin dashboards

Import the dashboards to Grafana

Add alerts and recording rules to Prometheus or Mimir

Next steps

Single binary meta-monitoring

Single binary meta-monitoring

Metrics

Alloy

Prometheus

Mixins

Key metrics for monitoring Loki

Key metrics for monitoring Loki

Request error rate

Request latency (p99)

Panics

Discarded samples

Compaction health

Ingester health and flush behavior

Distributor throughput

Object store operations

Resource and runtime health

Loki Canary (end-to-end data verification)

Internal error log rate

Retention and sweeper progress

Next steps

Download the `loki-mixin` dashboards