Set up monitoring for Tempo

Metamonitoring for Tempo is handled by the Grafana Kubernetes Helm chart (>=v2.1). Metamonitoring can be used with both microservices and single binary deployments of Tempo.

The Helm chart configures Grafana Alloy to collect metrics and logs.

Steps

This procudure uses the Grafana Kubernetes Helm chart and the values.yml file sets parameters in the Helm chart.

1. Add the Grafana Helm Chart repository, or update, if already added.

   Copy

   helm repo add grafana https://grafana.github.io/helm-charts  
   helm repo update

2. Create a new file named values.yml. Add the following example into your values.yml file and save it. Where indicated, add the values specific to your instance.

   YAML Copy

   cluster:
       name: traces # Name of the cluster. This populates the cluster label.
   
   integrations:
       tempo:
           instances:
             - name: "traces" # This is the name for the instance label that reports.
               namespaces:
                   - traces # This is the namespace that is searched for Tempo instances. Change this accordingly.
               metrics:
                   enabled: true
                   portName: prom-metrics
               logs:
                   enabled: true
               labelSelectors:
                   app.kubernetes.io/name: tempo
   
   alloy:
       name: "traces-monitoring"
   
   destinations:
   - name: "metrics"
     type: prometheus
     url: "<url>" # URL for Prometheus. Should look similar to "https://<prometheus host>/api/prom/push".
     auth:
       type: basic
       username: "<username>"
       password: "<password>"
   
   - name: "logs"
     type: loki
     url: "<url>" # URL for Loki. Should look similar to "https://<loki host>/loki/api/v1/push".
     auth:
       type: basic
       username: "<username>" 
       password: "<password>"
   
   alloy-metrics:
       enabled: true
   
   podLogs:
       enabled: true
       gatherMethod: kubernetesApi
       namespaces: [traces] # Set to namespace from above under instances.
       collector: alloy-singleton
   
   alloy-singleton:
       enabled: true
   
   alloy-metrics:
       enabled: true # Sends Grafana Alloy metrics to ensure the monitoring is working properly.

3. Install the Helm chart using the following command to create Grafana Alloy instances to scrape metrics and logs:

   Copy

   helm install k8s-monitoring grafana/k8s-monitoring \
   --namespace monitoring \
   --create-namespace \
   -f values.yml

4. Verify that data is being sent to Grafana.

   • Log into Grafana.
   • Select Metrics Drilldown and select cluster=<cluster.name> where cluster.name is the name specified in the values.yml file.
   • Do the same for Logs Drilldown.

   This example doesn’t include ingestion for any other data such as traces for sending to Tempo, but can be included with some configuration updates. Refer to Configure Alloy to remote-write to Tempo for more information.

Install Tempo dashboards in Grafana

Alloy scrapes metrics from Tempo and sends them to Mimir or another Prometheus compatible time-series database. You can then monitor Tempo using the mixins.

Tempo ships with mixins that includes:

• Relevant dashboards for overseeing the health of Tempo as a whole, as well as its individual components
• Recording rules that simplify the generation of metrics for dashboards and free-form queries
• Alerts that trigger when Tempo falls out of operational parameters

To install the mixins in Grafana, you need to:

1. Download the mixin dashboards from the Tempo repository.

2. Import the dashboards in your Grafana instance.

3. Upload alerts.yaml and rules.yaml files for Mimir or Prometheus

Download the tempo-mixin dashboards

1. First, clone the Tempo repository from Github:

   Bash Copy

   git clone git+ssh://github.com/grafana/tempo

2. Once you have a local copy of the repository, navigate to the operations/tempo-mixin-compiled directory.

   Bash Copy

   cd operations/tempo-mixin-compiled

This contains a compiled version of the alert and recording rules, as well as the dashboards.

Import the dashboards to Grafana

The dashboards directory includes the six monitoring dashboards that can be installed into your Grafana instance. Refer to Import a dashboard in the Grafana documentation.

To create a folder:

1. Open your Grafana instance and select Dashboards.
2. Select New in the right corner.
3. Select New folder from the New drop-down.
4. Name your folder, for example, “Tempo Monitoring”.
5. Select Create.

To import a dashboard:

1. Open your Grafana instance and select Dashboards.
2. Select New in the right corner.
3. Select Import.
4. On the Import dashboard screen, select Upload.
5. Browse to operations/tempo-mixin-compiled/dashboards and select the dashboard to import.
6. Drag the dashboard file, for example, tempo-operational.json, onto the Upload area of the Import dashboard screen. Alternatively, you can browse to and select a file.
7. Select a folder in the Folder drop-down where you want to save the imported dashboard. For example, select Tempo Monitoring created in the earlier steps.
8. Select Import.

The imported files are listed in the Tempo Monitoring dashboard folder.

To view the dashboards in Grafana:

1. Select Dashboards in your Grafana instance.
2. Select Tempo Monitoring, or the folder where you uploaded the imported dashboards.
3. Select any files in the folder to view it.

The ‘Tempo Operational’ dashboard shows read (query) information:

Add alerts and rules to Prometheus or Mimir

The rules and alerts need to be installed into your Mimir or Prometheus instance. To do this in Prometheus, refer to the recording rules and alerting rules documentation.

For Mimir, you can use [mimirtool](https://grafana.com/docs/mimir/latest/manage/tools/mimirtool/) to upload rule and alert configuration. Using a default installation of Mimir used as the metrics store for the Alloy configuration, you might run the following:

Bash Copy

mimirtool rules load operations/tempo-mixin-compiles/rules.yml --address=https://mimir-cluster.distributor.mimir.svc.cluster.local:9001

mimirtool alertmanager load operations/tempo-mixin-compiles/alerts.yml --address=https://mimir-cluster.distributor.mimir.svc.cluster.local:9001

For Grafana Cloud, you need to add the username and API key as well. Refer to the mimirtool documentation for more information.