Set up and use tracing

Along with metrics and logs, tracing is one of the three pillars of modern observability. Tracing is a way to track a single request and log that single request as it crosses through all of the services in your infrastructure.

Introduction to Grafana Cloud Traces

Grafana Cloud Traces is based on Tempo, an open source, easy-to-use and high-scale distributed tracing backend. Tempo is cost-efficient, requiring only object storage to operate, and is deeply integrated with Grafana, Prometheus, and Loki. Tempo can be used with any of the open source tracing protocols, including Jaeger, Zipkin, and OpenTelemetry.

How to use Grafana Cloud Traces

To send traces to Grafana Cloud Traces you will need a running Grafana Agent instance or an OpenTelemetry Collector instance in your environment as in the following diagram -

In the configuration section of the agent, set the following fields

tempo:
  configs:
  - name: default
    receivers:  # enable the receivers that you need
      jaeger:
        protocols:
          thrift_http:
          thrift_compact:
    push_config:
      endpoint: <Your Traces instance remote_write endpoint>
      basic_auth:
        username: <Your Traces instance instance ID>
        password: <Your Grafana.com API Key>

See the Tempo documentation to learn more about using the Grafana Agent for traces.

If you would like to use the OpenTelemetry Collector to send traces to Grafana Cloud, check out this blog post that explains how to do so in detail.

A deeper introduction to Tempo

Grafana Tempo is a high volume distributed tracing backend that can retrieve a trace when queried for the trace-id. It builds an index on the high cardinality trace-id field and uses an object store as backend which allows for high parallelization of queries. Read more about this in the architecture section of the docs.

Tempo has strong integrations with a number of existing open source tools including -

• Grafana! Grafana ships with native support for Tempo using the built-in Tempo data source.
• Grafana Loki - Loki, with its powerful query language LogQL v2 allows us to filter down on requests that we care about, and jump to traces using the Derived fields support in Grafana.
• Prometheus exemplars - which allow to jump from Prometheus metrics to Tempo traces by clicking on recorded exemplars. Read more about this integration in this blog post.

Documentation

• Getting started docs that explain the different components of a distributed tracing pipeline and how to use Tempo.
• API documentation for the different ingestion formats supported and the query API.
• Deployment and Operations on deployment options including Helm & Jsonnet, as well as operations & monitoring of Tempo.
• Troubleshooting for help around frequent error messages.