Introduction on Grafana Labs

Traces and telemetry

Thu, 28 May 2026 17:50:33 +0100

Traces and telemetry

Metrics, logs, traces, and profiles form the pillars of observability. Correlating between the four pillars of observability helps create a holistic view of your application and infrastructure.

Metrics

Metrics provide a high level picture of the state of a system. Metrics are the foundation of alerts because metrics are numeric values and can be compared against known thresholds. Alerts constantly run in the background and trigger when a value is outside of an expected range. This is typically the first sign that something is going on and are where discovery first starts. Metrics indicate that something is happening.

Logs

Logs provide an audit trail of activity from a single process that create informational context. Logs act as atomic events, detailing what’s occurring in the services in your application. Whereas metrics are quantitative (numeric) and structured, logs are qualitative (textual) and unstructured or semi-structured. They offer a higher degree of detail, but also at the expense of creating significantly higher data volumes. Logs let you know what’s happening to your application.

Traces

Traces add further to the observability picture by telling you what happens at each step or action in a data pathway. Traces provide the map–the where–something is going wrong. A trace provides a graphic representation of how long each step in the data flow pathway takes to complete. For example, how long a HTTP request, a database lookup, or a call to a third party service takes. It can show where requests initiate and finish, as well as how your system responds. This data helps you locate problem areas and assess their impact, often in places you never would have anticipated or found without this ability to trace the request flow.

Profiles

Profiles help you understand how your applications utilize compute resources such as CPU time and memory. This helps identify specific lines of code or functions to optimize and improve performance and efficiency.

Why traces?

Metrics in themselves aren’t sufficient to find the root cause and solve complex issues. The same can be said for logs, which can contain a significant amount of information but lack the context of the interactions and dependencies between the different components of your complex environment. Each pillar of observability—metrics, logs, traces, profiles—has its own unique strength when it comes to root causing issues. To get the most value of your observability strategy, you need to be able to correlate them.

Traces have the unique ability to show relationships between services. They help identify which services are upstream from your service, which is helpful when you want to understand which services might be negatively impacted by problems in your service. Traces also help identify which services are downstream from your service. This is valuable since your application relies on their downstream services, and problems with those services may be the cause of elevated errors or latency reported by your service. For example, you can directly see the failing database and all impacted failing edge endpoints.

Using traces and exemplars, you can go from a metric data point and get to an associated trace.

Or from traces to logs:

And vice versa, from logs to traces.

Trace structure

Thu, 28 May 2026 17:50:33 +0100

Trace structure

Traces are telemetry data structured as trees. Traces are made of spans (for example, a span tree); there is a root span that can have zero to multiple branches that are called child spans. Each child span can itself be a parent span of one or multiple child spans, and so on so forth.

In the specific context of Tempo and TraceQL query language, a span has the following associated fields:

name: the span name
duration: difference between the end time and start time of the span
status: enum: {ok, error, unset}. For details, refer to OTel span status documentation.
kind: enum: {server, client, producer, consumer, internal, unspecified}. For more details, refer to OTel span kind documentation.
Attributes

The first four properties are intrinsics. Intrinsics are the core, built-in fields that are fundamental to the identity and lifecycle of spans and traces. These fields are defined by the OpenTelemetry specification and are always present. They describe the essential structure of distributed tracing data.

Attributes are custom span metadata in the form of key-value pairs. There are four types of attributes: span attributes, resource attributes, event attributes, and link attributes.

Span attributes are key-value pairs that contain metadata that you can use to annotate a span to carry information about the operation it’s tracking. For example, in an eCommerce application, if a span tracks an operation that adds an item to a user’s shopping cart, the user ID, added item ID and cart ID can be captured and attached to the span as span attributes.

A resource attribute represents information about an entity producing the span. For example, a span created by a process running in a container deployed by Kubernetes could link a resource that specifies the cluster name, namespace, pod, and container name. Resource attributes are resource-related metadata (key-value pairs) that are describing the Resource.

An event attribute represents a unique point in the time during the span’s duration. For more information, refer to All about span events and Span events in the OTEL documentation.

A link attribute lets you query link data in a span. A span link associates one span with one or more other spans that are a causal relationship. For more information on span links, refer to the Span Links documentation in the Open Telemetry project.

The OpenTelemetry specification defines Semantic Attributes for Spans and for Resources. Semantic Span Attributes are a set of naming schemes for attributes shared across languages, frameworks, and runtimes. For more details, refer to Trace Semantic Conventions and Resource Semantic Conventions.

Tempo architecture

Thu, 28 May 2026 17:50:33 +0100

Tempo architecture

The journey of a trace is usually from an instrumented application, to a trace receiver/collector, to a trace backend store, and then those traces being retrieved and visualized in a tool like Grafana.

Grafana Tempo acts in two major capacities:

Ingesting trace spans, sorting the span resources and attributes into columns in an Apache Parquet schema, before sending them to object storage for long-term retention.
Retrieving trace data from storage, either by specific trace ID or by search parameters via TraceQL.

Tempo has a microservices-based architecture. All components are compiled into the same binary, and the -target parameter controls which component is started. This allows Tempo to run in two modes:

Monolithic mode: All components run in a single process. Recommended for getting started.
Microservices mode: Each component runs as a separate process, allowing independent scaling.

Refer to the example setups or deployment options for help deploying.

Components

Tempo is made up of the following components.

Distributor

The distributor accepts spans in multiple formats including Jaeger, OpenTelemetry, Zipkin. It uses the receiver layer from the OpenTelemetry Collector. For best performance, it’s recommended to ingest OpenTelemetry Proto. For this reason, Grafana Alloy uses the OTLP exporter/receiver to send spans to Tempo.

Once data is received, the distributor validates the request against limits, shards traces by hashing the traceID, and writes records to Kafka. The distributor uses the partition ring to determine which partitions are active and routes data accordingly. A write is only acknowledged to the client after Kafka has confirmed receipt of the data.

Kafka-compatible system

Tempo uses a Kafka-compatible system (such as Apache Kafka or WarpStream) as a durable queue between distributors and downstream components. This serves as a write-ahead log (WAL) that enables the decoupling of write and read paths.

When distributors receive trace data, they shard traces by trace ID and write records to Kafka partitions. Once Kafka acknowledges the write, the distributor returns a response to the client. From this point, the write and read paths diverge:

Block-builders consume from Kafka to build blocks for long-term object storage.
Live-stores consume from Kafka to serve recent data queries.

Because traces are sharded by trace ID to specific partitions, both block-builders and live-stores consume unique data without requiring deduplication. This enables Tempo to operate with a replication factor of 1 (RF1) while maintaining durability through Kafka.

Block-builder

The Block-builder is responsible for building blocks which are sent to long-term storage. It does so by consuming from one or more Kafka partitions where distributors have sent trace spans to. The Block-builder organizes spans into blocks based on a configurable time window, for example, 5 minutes.

Live-store

The live-store is a read-path component responsible for serving recent trace data. It consumes trace data from Kafka and makes it available for queries, typically covering the last 30 minutes to 1 hour of data depending on configuration.

Live-stores own the partition lifecycle within Tempo. While Kafka has its own concept of partitions, Tempo maintains a separate partition ring that tracks which Tempo partitions are active and which live-stores own them. Tempo partitions map to Kafka partitions, but they are logically distinct. The partition ring lets Tempo control partition states and ownership independently of Kafka’s internal partition management.

Partitions have three states:

Pending: No writes or reads. This is the initial state when a new partition is created, waiting for live-stores to be added as owners.
Active: Normal read-write mode. Distributors send data to active partitions, and queriers read from them.
Inactive: Read-only mode. Inactive partitions are eventually deleted after a configured period of time.

When a live-store starts, it checks if its partition already exists. If it does, the live-store joins as an owner. If not, it creates a new partition in pending state, waits for memberlist to propagate, then switches it to active. Scaling down requires first marking the partition as inactive while the live-store is still running. After enough time has passed for data to be available in object storage, the partition and live-store can be removed.

For high availability, live-stores are typically deployed across multiple availability zones. Each Tempo partition is owned by one live-store per zone, so if a live-store in one zone becomes unavailable, the live-store in the other zone can continue serving queries. While this setup replicates data across zones (RF2), the read quorum is 1—queriers only need a response from one live-store per partition. This provides high availability without requiring data deduplication on the read path.

Query Frontend

The Query Frontend is the component called by, for example, Grafana when a user wishes to retrieve a specific trace via a trace ID, or carry out a search via a TraceQL filter (for example, all incoming requests for traces). The Query Frontend is responsible for calling one or more Queriers to carry out examination of potential blocks of data where span data for matching traces may exist in parallel, to speed up result response time (dependent on multiple Queriers being configured). Requests by the Query Frontend can be split across multiple Queriers, which all work in parallel to retrieve results quickly. The more Queriers available, generally the quicker a result response. When Queriers have returned enough responses, the Query Frontend is responsible for concatenating all of the span data returned by each individual Querier together to send a response to the requester.

The Query Frontend is responsible for sharding the search space for an incoming query.

A simple HTTP endpoint exposes traces: GET /api/traces/<traceID>

Internally, the Query Frontend splits the blockID space into a configurable number of shards and queues these requests. Queriers connect to the Query Frontend via a streaming gRPC connection to process these sharded queries.

Querier

Queriers carry out the actual examination of block data in object storage to find any relevant span data based on the trace ID or TraceQL query given. As well as object storage, they also query live-stores for any recent span data that may have been processed but not yet sent to object storage. This ensures that both historical and recent trace data is always available.

The querier finds the requested trace ID in either the live-stores or the backend storage. Depending on parameters, the querier queries the live-stores for recently ingested traces and pulls the bloom filters and indexes from the backend storage to efficiently locate the traces within object storage blocks.

The querier exposes an HTTP endpoint at: GET /querier/api/traces/<traceID>, but it’s not intended for direct use.

Queries should be sent to the Query Frontend.

Compactor

The compactor is responsible for ensuring that the stored data is both compressed and deduplicated (more on deduplication in the advanced course). The compactor is also responsible for expiring data after the retention period for that data has been reached. Compactors run on scheduled frequent intervals to deal with data that is not compacted and has been stored by the block-builders. Compaction takes into account the data stored for specific traces to minimize search space on queries.

Backend scheduler and worker

The scheduler is a forward-looking re-architecture of the compaction process, with the goal of improving the determinism and removing the duplication present in the current compaction process. The scheduler is responsible for the scheduling and tracking jobs which are assigned to workers for processing. The scheduler component, in combination with the worker, will eventually replace the current compactor component.

The worker connects to the scheduler via gRPC to receive jobs for processing. Workers are responsible for executing jobs assigned to them by the scheduler and updating the job status back to the scheduler. These jobs currently include compaction and retention, but will likely include other kinds of jobs in the future.

The workers currently have the additional responsibility of maintaining the blocklist for all tenants, which was previously handled by the compactor. The determination of which tenants to poll is coordinated through the ring, just as it is with the compactor.

When transitioning from the compactor to the scheduler and worker architecture, some considerations need to be kept in mind:

Only one scheduler should be running at a time.
Workers should be scaled up at the same time the compactor is scaled to 0. This is to avoid conflicts between the two systems attempting to compact the same blocks.

Since the worker is taking the role of the compactor when it comes to polling, some documentation may reference the compactor, and can instead be interpreted as the worker for environments where the worker has replaced the compactor.

Object storage

Tempo uses object storage for storing all tracing data. It supports three major object storage APIs:

Amazon Simple Storage Service (S3)
Google Cloud Storage (GCS)
Microsoft Azure Storage (AS)

Metrics-generator

This is an optional component that derives metrics from ingested traces and writes them to a metrics storage. Like live-stores, metrics-generators consume trace data from Kafka. Refer to the metrics-generator documentation to learn more.

Lifecycle of a write

Trace data is sent from the tracing pipeline (OpenTelemetry Collector, Grafana Alloy) to a distributor.
The distributor validates the request, shards traces by trace ID, and writes to Kafka.
Kafka acknowledges the write
And the distributor returns a response to the client.
Live-stores consume from Kafka and make data available for recent queries.
Block-builders consume from Kafka and build blocks for long-term object storage.

Lifecycle of a read

The query frontend receives a TraceQL query and shards it into jobs for recent data and long-term storage.
Queriers retrieve recent data from live-stores.
For older data, queriers fetch blocks from object storage.
Results are aggregated and returned to the user.

Things to keep in mind

There are three fundamentally important things to keep in mind with traces:

There is no concept of the ’end’ of a trace. A trace can start with any span which holds a unique trace ID that hasn’t been seen by Tempo previously. However, spans can be continually added at any point in the future.
When a trace ID is queried in Tempo, it returns all of the currently stored/ingested spans that belong to that trace and present them in a mapped response (that is, a graph of parent/child/sibling relationships between spans) that allow, for example, Grafana to then visually render those traces.
TraceQL queries return all matching traces from Tempo for the filters presented to it. This does not necessarily mean in chronological order, as some Queriers may return results more promptly than others. When the max trace limit for a TraceQL query has been hit, the Query Frontend will return a response with all the current traces and their spans at that point and ignore/cancel all outstanding Queriers.

Glossary

Thu, 28 May 2026 17:50:33 +0100

Glossary

The following terms are often used when discussing traces.

Active series: A time series that receives new data points or samples.
Cardinality: The total combination of key/value pairs, such as labels and label values for a given metric series or log stream, and how many unique combinations they generate.

Child span: A span that is nested within a parent span. Each child span represents a sub-operation or downstream call within the broader operation represented by its parent. A child span can itself be a parent of other child spans.
Data source: A basic storage for data such as a database, a flat file, or even live references or measurements from a device. A file, database, or service that provides data. For example, trace data is imported into Grafana by configuring and enabling a Tempo data source.

Event attribute: A key-value pair attached to a span event, which represents a unique point in time during the span’s duration. For more information, refer to Span events in the OpenTelemetry documentation.
Exemplar: Any data that serves as a detailed example of one of the observations aggregated into a metric. An exemplar contains the observed value together with an optional timestamp and arbitrary trace IDs, which are typically used to reference a trace.

Intrinsics: The core, built-in fields that are fundamental to the identity and lifecycle of spans and traces. Intrinsic fields include name, duration, status, and kind. These fields are defined by the OpenTelemetry specification and are always present.

Link attribute: A key-value pair attached to a span link. A span link associates one span with one or more causally related spans. For more information, refer to Span Links in the OpenTelemetry documentation.
Log: Chronological events, usually text-based, allowing for the diagnosis of problems. Logs can provide informational context, such as detailed records of all events during user interactions, for example, when events happen, who used the system, status messages, etc.
Metric: A number that helps an operator understand the state of a system, such as the number of active users, error count, average response time, and more.

Resource attribute: A key-value pair that represents information about the entity producing a span, such as the cluster name, namespace, Pod, or container name.

Root span: The first span in a trace, representing the initial request or operation. A root span has no parent span and serves as the top of the span tree.

Semantic attribute: A standardized naming scheme for attributes shared across languages, frameworks, and runtimes, as defined by the OpenTelemetry specification. For more information, refer to Trace Semantic Conventions in the OpenTelemetry documentation.
Span: A unit of work done within a trace.

Span attribute: A key-value pair that contains metadata to annotate a span with information about the operation it tracks. For example, a span tracking an “add to cart” operation might include the user ID, item ID, and cart ID as span attributes.
Spanset: The group of spans for each individual trace returned from a TraceQL query.
Trace: A trace represents the whole journey of a request or an action as it moves through all the nodes of a distributed system, especially containerized applications or microservices architectures. Traces are composed of spans. For more information, refer to What is Distributed Tracing?.