--- title: "Monitor Loki using a local LGTM (Loki, Grafana, Tempo and Mimir) stack | Grafana Enterprise Logs documentation" description: "Monitor Loki using a local LGTM (Loki, Grafana, Tempo and Mimir) stack" --- > For a curated documentation index, see [llms.txt](/llms.txt). For the complete documentation index, see [llms-full.txt](/llms-full.txt). # Monitor Loki using a local LGTM (Loki, Grafana, Tempo and Mimir) stack > Warning > > We no longer recommend using the meta-monitoring Helm chart to monitor Loki. To consolidate monitoring efforts into one Helm chart, Grafana Labs recommends using the Kubernetes monitoring Helm chart. Instructions for setting up the Kubernetes monitoring Helm chart can be found under [Manage](/docs/loki/v3.6.x/operations/meta-monitoring/). This topic will walk you through using the meta-monitoring Helm chart to deploy a local stack to monitor your production Loki installation. This approach leverages many of the chart’s *self monitoring* features, but instead of sending logs back to Loki itself, it sends them to a small Loki, Grafana, Tempo, Mimir (LGTM) stack running within the `meta` namespace. ## Before you begin - Helm 3 or above. See [Installing Helm](https://helm.sh/docs/intro/install/). - A running Kubernetes cluster with a running Loki deployment. ## Configure the meta namespace The meta-monitoring stack will be installed in a separate namespace called `meta`. To create this namespace, run the following command: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash kubectl create namespace meta ``` ## Configuration and Installation The meta-monitoring stack is installed using the `meta-monitoring` Helm chart. The local mode deploys a small LGTM stack that includes Alloy, Grafana, Mimir, Loki, and Tempo. To configure the meta-monitoring stack, create a `values.yaml` file with the following content: YAML ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```yaml namespacesToMonitor: - default cloud: logs: enabled: false metrics: enabled: false traces: enabled: false local: grafana: enabled: true logs: enabled: true metrics: enabled: true traces: enabled: true minio: enabled: true ``` For further configuration options, refer to the [sample values.yaml file](https://github.com/grafana/meta-monitoring-chart/blob/main/charts/meta-monitoring/values.yaml). Local mode by default will also enable Minio, which will act as the object storage for the LGTM stack. To provide access to Minio, you need to create a generic secret. To create the generic secret, run the following command: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash kubectl create secret generic minio -n meta \ --from-literal= \ --from-literal= ``` > Note > > Username and password must have a minimum of 8 characters. To install the meta-monitoring stack, run the following commands: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash helm repo add grafana https://grafana.github.io/helm-charts helm repo update helm install meta-monitoring grafana/meta-monitoring -n meta -f values.yaml ``` or when upgrading the configuration: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash helm upgrade meta-monitoring grafana/meta-monitoring -n meta -f values.yaml ``` To verify the installation, run the following command: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash kubectl get pods -n meta ``` It should return the following pods: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash grafana-59d664f55f-dtfqr 1/1 Running 2 (2m7s ago) 137m loki-backend-0 2/2 Running 2 (2m7s ago) 137m loki-backend-1 2/2 Running 4 (2m7s ago) 137m loki-backend-2 2/2 Running 3 (2m7s ago) 137m loki-read-6f775d8c5-6t749 1/1 Running 1 (2m7s ago) 137m loki-read-6f775d8c5-kdd8m 1/1 Running 1 (2m7s ago) 137m loki-read-6f775d8c5-tsw2r 1/1 Running 1 (2m7s ago) 137m loki-write-0 1/1 Running 1 (2m7s ago) 137m loki-write-1 1/1 Running 1 (2m7s ago) 137m loki-write-2 1/1 Running 1 (2m7s ago) 137m meta-alloy-0 2/2 Running 2 (2m7s ago) 137m meta-alloy-1 2/2 Running 2 (2m7s ago) 137m ... ``` ## Enable Loki Tracing By default, Loki does not have tracing enabled. To enable tracing, modify the Loki configuration by editing the `values.yaml` file and adding the following configuration: Set the `tracing.enabled` configuration to `true`: YAML ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```yaml loki: tracing: enabled: true ``` Next, instrument each of the Loki components to send traces to the meta-monitoring stack. Add the `extraEnv` configuration to each of the Loki components: YAML ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```yaml ingester: replicas: 3 extraEnv: - name: JAEGER_ENDPOINT value: "http://mmc-alloy-external.default.svc.cluster.local:14268/api/traces" # This sets the Jaeger endpoint where traces will be sent. # The endpoint points to the mmc-alloy service in the default namespace at port 14268. - name: JAEGER_AGENT_TAGS value: 'cluster="prod",namespace="default"' # This specifies additional tags to attach to each span. # Here, the cluster is labeled as "prod" and the namespace as "default". - name: JAEGER_SAMPLER_TYPE value: "ratelimiting" # This sets the sampling strategy for traces. # "ratelimiting" means that traces will be sampled at a fixed rate. - name: JAEGER_SAMPLER_PARAM value: "1.0" # This sets the parameter for the sampler. # For ratelimiting, "1.0" typically means one trace per second. ``` ## Install kube-state-metrics Metrics about Kubernetes objects are scraped from [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics). This needs to be installed in the cluster. The `kubeStateMetrics.endpoint` entry in the meta-monitoring `values.yaml` should be set to its address (without the `/metrics` part in the URL): YAML ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```yaml kubeStateMetrics: # Scrape https://github.com/kubernetes/kube-state-metrics by default enabled: true # This endpoint is created when the helm chart from # https://artifacthub.io/packages/helm/prometheus-community/kube-state-metrics/ # is used. Change this if kube-state-metrics is installed somewhere else. endpoint: kube-state-metrics.kube-state-metrics.svc.cluster.local:8080 ``` ## Accessing the meta-monitoring stack To access the meta-monitoring stack, you can use port-forwarding to access the Grafana dashboard. To do this, run the following command: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash kubectl port-forward -n meta svc/grafana 3000:3000 ``` ## Dashboards and Rules The local meta-monitoring stack comes with a set of pre-configured dashboards and alerting rules. These can be accessed via [http://localhost:3000](http://localhost:3000) using the default credentials `admin` and `admin`.