Getting started with Tempo
Distributed tracing visualizes the lifecycle of a request as it passes through a set of applications. There are a few components that must be configured in order to get a working distributed tracing visualization.
This document discusses the four major pieces necessary to build out a tracing system: Instrumentation, Pipeline, Backend and Visualization. If one were to build a diagram laying out these pieces it may look something like this:
The first building block to a functioning distributed tracing visualization pipeline is client instrumentation, which is the process of adding instrumentation points in the application that creates and offloads spans.
Below is a list of the most popular frameworks used for client instrumentation. Each of these have SDKs in most commonly used programming languages and you should pick one according to your application needs.
OpenTelemetry Auto Instrumentation
Some languages have support for auto-instrumentation. These libraries capture telemetry information from a client application with minimal manual instrumentation of the codebase.
- OpenTelemetry Java Autoinstrumentation
- OpenTelemetry .NET Autoinstrumentation
- OpenTelemetry Python Autoinstrumentation
Note: Check out our instrumentation examples to learn how to instrument your favourite language for distributed tracing.
2. Pipeline (Grafana Agent)
Once your application is instrumented for tracing, the next step is to send these traces to a backend for storage and visualization. It is common to build a tracing pipeline that offloads spans from your application, buffers them and eventually forwards them to a backend. Tracing pipelines are optional (most clients can send directly to Tempo), but you will find that they become more critical the larger and more robust your tracing system is.
The Grafana Agent is a service that is deployed close to the application, either on the same node or within the same cluster (in kubernetes) to quickly offload traces from the application and forward them to a storage backend. It also abstracts features like trace batching and backend routing away from the client.
To learn more about the Grafana Agent and how to set it up for tracing with Tempo, refer to this blog post.
3. Backend (Tempo)
Grafana Tempo is an easy-to-use and high-scale distributed tracing backend used to store and query traces. The purpose of the tracing backend is to store and retrieve traces on demand.
Getting started with Tempo is easy.
- If you’re looking for examples of how to get started with Tempo, check out the examples topic.
- For production workloads, refer to the deployment section.
4. Visualization (Grafana)
Grafana has a built in Tempo datasource that can be used to query Tempo and visualize traces. For more information refer to the Tempo data source topic.
See here for details about Grafana configuration.
Related Tempo resources
Be the first to learn about exciting next-generation features in Grafana 8.0, be inspired by what community members are building, and attend expert-led sessions and workshops on Grafana, Prometheus, Loki logs, and more.
Getting started with tracing and Grafana Tempo
In this session, we will use an instrumented application to demonstrate how to use logs and Prometheus exemplars to find traces effectively in Tempo. The demo will hit on the basics of operating Tempo and reveal how Tempo allows us to scale tracing as far as possible with less operational cost and complexity than ever before.