Manage Tempo on Grafana Labs

Monitor Tempo

Thu, 09 Apr 2026 14:59:14 +0000

Monitor Tempo

Tempo is instrumented to expose metrics, logs, and traces. Furthermore, the Tempo repository has a mixin that includes a set of dashboards, rules, and alerts. Together, these can be used to monitor Tempo in production.

Instrumentation

Metrics, logs, and traces from Tempo can be collected to observe its services and functions.

Metrics

Tempo is instrumented with Prometheus metrics and emits RED metrics for most services and backends. RED metrics are a standardized format for monitoring microservices, where R stands for requests, E stands for errors, and D stands for duration.

The Tempo mixin provides several dashboards using these metrics.

Query frontend inspected bytes

Use tempo_query_frontend_bytes_inspected_total to monitor how many bytes the query frontend reads from disk and object storage. This counter is emitted per tenant and op (traces, search, metadata, metrics). Because cached responses from queriers are excluded, it reflects actual storage and network I/O.

For PromQL examples and alerting guidance, refer to Query query IO and time stamp distance.

Logs

Tempo emits logs in the key=value (logfmt) format.

Traces

Tempo uses the OpenTelemetry SDK for tracing instrumentation. The complete read path and some parts of the write path of Tempo are instrumented for tracing.

You can configure the tracer using environment variables. To enable tracing, set one of the following: OTEL_EXPORTER_OTLP_ENDPOINT or OTEL_EXPORTER_OTLP_TRACES_ENDPOINT.

The OpenTelemetry SDK uses OTLP/HTTP by default, which can be configured with OTEL_EXPORTER_OTLP_PROTOCOL.

Polling

Tempo maintains knowledge of the state of the backend by polling it on regular intervals. There are currently only two components that need this knowledge and, consequently, only two that poll the backend: compactors and queriers.

Refer to Use polling to monitor the backend status for Tempo.

Dashboards

The Tempo mixin has four Grafana dashboards in the yamls folder that you can download and import into your Grafana UI. These dashboards work well when you run Tempo in a Kubernetes (k8s) environment and metrics scraped have the cluster and namespace labels.

Tempo Reads dashboard

This is available as tempo-reads.json.

The Reads dashboard gives information on Requests, Errors, and Duration (RED) on the query path of Tempo. Each query touches the Gateway, Tempo-Query, Query-Frontend, Queriers, Ingesters, the backend, and Cache, if present.

Use this dashboard to monitor the performance of each of the mentioned components and to decide the number of replicas in each deployment.

Tempo Writes dashboard

This is available as tempo-writes.json.

The Writes dashboard gives information on RED on the write/ingest path of Tempo. A write query touches the Gateway, Distributors, Ingesters, and the backend. This dashboard also gives information on the number of operations performed by the Compactor to the backend.

Use this dashboard to monitor the performance of each of the mentioned components and to decide the number of replicas in each deployment.

Tempo Resources dashboard

This is available as tempo-resources.json.

The Resources dashboard provides information on CPU, Container Memory, and Go Heap Inuse. This dashboard is useful for resource provisioning for the different Tempo components.

Use this dashboard to see if any components are running close to their assigned limits.

Tempo Operational dashboard

This is available as tempo-operational.json.

The Tempo Operational dashboard deserves special mention because it is probably a stack of dashboard anti-patterns. It’s big and complex, doesn’t use jsonnet, and displays far too many metrics in one place. For just getting started, the RED dashboards are great places to learn how to monitor Tempo in an opaque way.

This dashboard is included in the Tempo repository for two reasons:

The dashboard provides a stack of metrics for other operators to consider monitoring while running Tempo.
We want the dashboard in our internal infrastructure and we vendor the tempo-mixin to do this.

Rules and alerts

The Rules and Alerts are available as YAML files in the compiled mixin on the repository.

To set up alerting, download the provided JSON files and configure them for use on your Prometheus monitoring server.

Check the runbook to understand the various steps that can be taken to fix firing alerts.

Dedicated attribute columns

Thu, 09 Apr 2026 14:59:14 +0000

Dedicated attribute columns

Dedicated attribute columns improve query performance by storing the most frequently used attributes in their own columns, rather than in the generic attribute key-value list.

Introduced with vParquet3, dedicated attribute columns are available when using vParquet3 or later storage formats. Even though vParquet3 is deprecated, this feature is available when using the current value of vParquet4.

With vParquet5, dedicated attribute columns gain support for array-valued attributes, event-scoped attributes, and blob attributes using the options field.

Configuration

You can configure dedicated attribute columns in the storage block or via overrides.

# Storage configuration for traces
storage:
  trace:
    block:
      # Default dedicated columns for all blocks
      parquet_dedicated_columns:
        - name: <string> # name of the attribute
          type: <string> # type of the attribute. options: string, int
          scope: <string> # scope of the attribute. options: resource, span, event
          options: [<string>] # optional, vParquet5 only. options: array, blob

overrides:
  # Global overrides for dedicated columns configuration
  parquet_dedicated_columns:
    - name: <string> # name of the attribute
      type: <string> # type of the attribute. options: string, int
      scope: <string> # scope of the attribute. options: resource, span, event
      options: [<string>] # optional, vParquet5 only. options: array, blob

  per_tenant_override_config: /conf/overrides.yaml
---
# /conf/overrides.yaml
# Tenant-specific overrides configuration
overrides:
  "<tenant id>":
    parquet_dedicated_columns:
      - name: <string> # name of the attribute
        type: <string> # type of the attribute. options: string, int
        scope: <string> # scope of the attribute. options: resource, span, event
        options: [<string>] # optional, vParquet5 only. options: array, blob

  # A "wildcard" override can be used that will apply to all tenants if a match is not found.
  "*":
    parquet_dedicated_columns:
      - name: <string> # name of the attribute
        type: <string> # type of the attribute. options: string, int
        scope: <string> # scope of the attribute. options: resource, span, event
        options: [<string>] # optional, vParquet5 only. options: array, blob

Priority is given to the most specific configuration, so tenant-specific overrides will take precedence over global overrides. Similarly, default overrides take precedence over storage block configuration.

Usage

Dedicated attribute columns are limited to 10 string attributes and 5 integer attributes per scope (span, resource, and event). As a rule of thumb, good candidates for dedicated attribute columns are attributes that contribute the most to the block size, even if they aren’t frequently queried. Reducing the generic attribute key-value list size significantly improves query performance.

Integer attribute selection

Integer dedicated columns are most effective when the attribute is present on at least 5% of rows in a scope. The tempo-cli analyse block command shows the percentage of rows for each integer attribute and marks candidates that meet this threshold with a recommendation. Sparse integer attributes (below 5% prevalence) typically perform better in the generic attribute storage.

Array-valued attributes

Dedicated attribute columns support array-valued attributes. This is useful for attributes that can have multiple values, such as HTTP headers or custom tags.

To enable array support, add options: ["array"] to the dedicated attribute column configuration. The options field is only available in vParquet5 and later. Earlier versions ignore this field and only support single-valued attributes.

When options: ["array"] is set, the dedicated column stores multiple values per attribute. Without this option (default), only single values are supported.

storage:
  trace:
    block:
      version: vParquet5
      parquet_dedicated_columns:
        # Single-valued string attribute (default behavior)
        - name: http.method
          type: string
          scope: span

        # Array-valued string attribute
        - name: http.request.header.accept
          type: string
          scope: span
          options: ["array"]

        # Array-valued integer attribute
        - name: custom.retry.counts
          type: int
          scope: span
          options: ["array"]

Blob attributes

Blob attributes are designed for high-cardinality or high-length string values where dictionary encoding becomes inefficient. Examples include UUIDs, stack traces, request bodies, or any attribute with many unique values.

When an attribute’s dictionary size per row group exceeds a threshold (default 4MiB), the column should be marked as a blob. The tempo-cli analyse block command can help identify blob candidates by showing the estimated dictionary size per row group.

To enable blob mode, add options: ["blob"] to the dedicated attribute column configuration. Blob columns use zstd compression instead of dictionary encoding, which is more efficient for high-cardinality data.

storage:
  trace:
    block:
      version: vParquet5
      parquet_dedicated_columns:
        # Standard dedicated column (uses dictionary encoding)
        - name: http.method
          type: string
          scope: span

        # Blob column for high-cardinality data
        - name: db.statement
          type: string
          scope: span
          options: ["blob"]

        # Blob column for stack traces
        - name: exception.stacktrace
          type: string
          scope: event
          options: ["blob"]

Use the tempo-cli analyse block command with the --blob-threshold option to identify attributes that should be configured as blobs. Attributes exceeding the threshold are marked as “(blob)” in the output.

Event-scoped attributes

With vParquet5, dedicated columns support event-scoped attributes in addition to span and resource scopes. Event-scoped columns are useful for frequently queried event attributes such as exception details or custom event data.

storage:
  trace:
    block:
      version: vParquet5
      parquet_dedicated_columns:
        # Event-scoped string attribute
        - name: exception.message
          type: string
          scope: event

        # Event-scoped attribute with blob option
        - name: exception.stacktrace
          type: string
          scope: event
          options: ["blob"]

Tempo-cli

You can use the tempo-cli tool to find good candidates for dedicated attribute columns. The tempo-cli provides the commands analyse block <tenant-id> <block-id> and analyse blocks <tenant-id> that will output the top N attributes by size for a given block or all blocks in a tenant.

Example:

tempo-cli analyse blocks --backend=local --bucket=./cmd/tempo-cli/test-data/ single-tenant

Refer to the tempo-cli documentation for more information.

Tune search performance

Thu, 09 Apr 2026 14:59:14 +0000

Tune search performance

Regardless of whether or not you are using TraceQL or the original search API, Tempo searches all of the blocks in the specified time range. Depending on your volume, this may result in slow queries.

This document explains how read path in Tempo works, what controls you have to tune the read path, details of the core configuration options, and how to tune these configuration options to get the most out of your Tempo cluster.

The general advice is to scale your compactors and queriers. Additional queriers can more effectively run jobs in parallel while additional compactors more aggressively reduce the length of your blocklist and copies of data (if using RF=3).

Note
All forms of search (TraceQL and tags based) are only supported on the vParquet and forward blocks. v2 blocks can only be used for trace by id lookup.

Tempo 2.3 and higher support Dedicated attribute columns, another great method to improve search performance.

Before you begin

You should understand the basic Tempo architecture.

For more information, refer to the Tempo architecture to learn about how Tempo works.

Glossary

query: A search query, issued by end user. For example, { traceDuration > 1s } is a TraceQL query.
job: A shard of a search query, the lowest unit of work. A query is broken down into multiple jobs and processed.
batch: A group of jobs is called a batch.
frontend: Another name for query-frontend
querier: Responsible for executing (processing) the jobs, and sending the results back to query-frontend.

Tempo query path

Tempo has a query path made up of a query-frontend, querrier, ingester, metrics-generator, and backend.

You can think about each component in a query path as a generic producer and worker model:

The query-frontend is a producer and it has multiple workers connected to it. The query-frontend takes a single query and shards it into multiple jobs (units of work).
The queriers are workers. They enqueue work from a queue, process it, and send the results back to the producer (query-frontend).
The querier either reads data from backend and processes the query, or it delegates the query to ingesters and metrics generators based on the type of job, time range, and query type.

Lifecycle of a query

A search request comes from the user, and it is split into multiple jobs in the query-frontend.

These jobs are then added to a queue and queriers pick up jobs in batches to process.

A single search query can create hundreds of jobs based on the time range and other variables. These jobs are batched and sent to queriers.

The querier receives the batch and processes the jobs in the batch, builds results, and sends the results of the whole batch back to the query-frontend.

The query-frontend merges and deduplicates the results, and sends them back to the client.

When querier starts, it opens multiple connections to each query-frontend.

Job batches are sent over these connections, and return the results. This process is synchronized so a single job batch can block a connection.

The number of connections control the number of batches a querier processes concurrently. The number of connections is controlled by querier.max_concurrent_queries OR frontend_worker.parallelism.

General guidelines

Tuning the search pipeline can be difficult as it requires balancing a number of different configuration parameters. The below tips can you get your head around the general problem, but the specifics require experimentation.

Review the query-frontend logs for lines like the following to get a feeling for how many jobs your queries are creating:

level=info ts=2023-07-19T19:38:01.354220385Z caller=searchsharding.go:236 msg="sharded search query request stats and SearchMetrics" ...

For a single TraceQL query the maximum number of parallel jobs is constrained by:
- query_frontend.search.concurrent_jobs: This is the maximum number of jobs the frontend dispatches for one TraceQL query.
- # queriers * querier.max_concurrent_queries * query_frontend.max_batch_size: This is the maximum job capacity of your Tempo cluster. If a given TraceQL query produces less jobs then these two values it should be executed entirely in parallel on the queriers.
Increasing querier.max_concurrent_queries is a great way to get more out of your queriers. However, if queriers are OOMing or saturating other resources then this should be lowered. Lowering query_frontend.max_batch_size will also reduce the total work attempted by one querier.

Querier and query-frontend configuration

Queriers and query-frontends have additional configuration related to search of the backend datastore.

Querier

querier:
  # Control the amount of work each querier will attempt. The total number of
  # jobs a querier will attempt this is this value * query_frontend.max_batch_size
  max_concurrent_queries: 20

Query-frontend

Query frontend lists all configuration options.

These suggestions help deal with scaling issues.

server:
  # At larger scales, searching starts to feel more like a batch job.
  # Increase the server timeout intervals.
  http_server_read_timeout: 2m
  http_server_write_timeout: 2m

query_frontend:
  # When increasing concurrent_jobs, also increase the queue size per tenant,
  # or search requests will be cause 429 errors. This is the total number of jobs
  # per tenant allowed in the queue.
  max_outstanding_per_tenant: 2000

  # The number of jobs the query-frontend will batch together when passing jobs to the queriers. This value
  # This value * querier.max_concurrent_queries is your the max number of jobs a given querier will try at once.
  max_batch_size: 3

  search:
    # At larger scales, increase the number of jobs attempted simultaneously,
    # per search query.
    concurrent_jobs: 2000

    # The query-frontend will attempt to divide jobs up by an estimate of job size. The smallest possible
    # job size is a single parquet row group. Increasing this value will create fewer, larger jobs. Decreasing
    # it will create more, smaller jobs.
    target_bytes_per_job: 50_000_000

Settings that are safe to increase without major impact.

Scaling up queriers is a safe way to add more query capacity. At Grafana Labs, we prefer to scale queries horizontally by adding more replicas. If you see out of memory (OOM) errors, it might be worth scaling the queriers vertically.

We recommend running at least two replicates of query-frontend. These replicas should be scaled vertically instead of horizontally. If you need to scale, scale query-frontends vertically by adding more CPU and RAM. Currently, query-frontends aren’t scaled horizontally, but this might change in the future.

The reason we keep only two replicas is because each query-frontend has its own request queue and it also impacts the amount of jobs sent to each querier. If you add more query-frontends, you need to tune other configuration parameters to account for the change. We decided to keep query-frontend to two replicas for now.

In a dedicated cluster, you can increase query_frontend.max_outstanding_per_tenant because the cluster is dedicated to a single customer. In a shared cluster, you need to use more caution when increasing querier.max_outstanding_per_tenant

Parameter interactions

The query_frontend configuration options control how Tempo shards a query into multiple jobs. These options also control the size of a single job, number of jobs, and the number of jobs added to the work queue at once (concurrently) for queriers to be picked up. The max_batch_size option controls how many jobs are sent to queriers at once in a single batch.

Querier configuration options control processing of these jobs, queries operate at a job level. They’re workers: they pick a job from the queue and execute the job, and return the results back to the query-frontend. Querier options control how many jobs it processes at once by opening concurrent connections to query-frontends, and each connection processes a single.

Guidelines on key configuration parameters

The following sections provide recommendations for key configuration parameters.

`query_frontend.max_outstanding_per_tenant` parameter

The query_frontend.max_outstanding_per_tenant parameter decides the maximum number of outstanding requests per tenant per frontend; requests beyond this error with HTTP 429.

This configuration controls how much load a single tenant can put on a cluster. A return of 429 indicates users to slow down.

This configuration can be used to maintain quality of service (QoS) for tenants by asking tenants to slow down by sending 429 to the tenant that’s overloading the system.

Guidelines

In a dedicated cluster with one big tenant, it’s okay to increase the number for query_frontend.max_outstanding_per_tenant.
In a shared cluster with a lot of small tenants, keep the number small.
If a single tenant is overwhelming the whole cluster, you should decrease this parameter. It reduces the amount of work the tenant can enqueue at once. It starts returning 429 to the tenant.
If your tenants are complaining about getting 429, you might want to increase this parameter and scale out queries to handle the query load.
If you increase other configurations that increase the query throughput or scale queriers, you might want to reduce this parameter to control the amount of work that can be enqueued.

`query_frontend.max_batch_size` parameter

The number of jobs to batch together in one HTTP request to the querier. Large search queries over a longer time range produce thousands of jobs. Instead of sending jobs one by one, batch the jobs and send them at once. The querier picks up and process them.

Batching pushes jobs faster to the queriers, and reduce the time spent waiting around for jobs to be picked up by the querier.

Guidelines

Default value of max_batch_size is set to 7.
We DO NOT recommend changing the batch size from the default value of 7. Based on testing at Grafana Labs, 7 is a good default.
If the batch size is big, you push more jobs at once but it takes longer for the querier to process the batch and return the results back.
Big batch size will increase the latency of the querier requests, and they might start hitting timeouts of 5xx, which will increase the rate of retries.
Bigger max_batch_size results in pushing too many jobs to the queriers. The jobs then have to be canceled if a query is exited early.

`query_frontend.search.concurrent_jobs` parameter

The number of concurrent jobs to execute when searching the backend. This controls how much work is produced concurrently.

To put it another way, if a search job is sharded into 5000 jobs, and concurrent_jobs is set to 1000, then Tempo only executes 1000 jobs concurrently. The other 4000 jobs are processed one by one as the first 1000 jobs return the responses.

If Tempo manages to answer the search query without executing all 5000 jobs, Tempo exits early and cancels the jobs that weren’t started.

Guidelines

This params controls the numbers of concurrent jobs to the number of jobs to put in the queue at once for processing a query.
If this number is low, big queries that create lots of jobs processed slowly.
If this number is high, the frontend pushes work faster to queries, and the frontend needs to cancel those jobs if the search exits early because the query is fulfilled.
If you see high load, spikes, or OOMs in the queriers, you can lower this configuration for a fair QoS and stability.
Having this number to a high value will allow a single tenant to push lots of work at once and overwhelm the querier. So keeping it to a lower number in a shared cluster allows for fair scheduling.
This parameter controls how fast jobs are pushed into the queue for queries to be picked up by queries
Adjusting this can impact your batch size so that metric should be watched while making changes to this configuration, and make sure that actual batch size is around the max_batch_size.
We recommend keeping it high in dedicated clusters with a single big tenant.
We recommend keeping it low in shared clusters because it ensures that a single tenant can’t overwhelm the read path.

`query_frontend.max_retries` parameter

This option controls the number of times to retry a request sent to a querier. We only retry on 5xx from a queriers (translated from the equivalent gRPC error).

Guidelines

This option controls the retries from query-frontend to queriers.
High max_retries amplifies the slow queriers and result in a lot more load then required.
Low max_retries can lead to users having bad experience due failed queriers, because a job failed and wasn’t retried.
If there are too many retries in a cluster, it can be a sign that queriers are struggling to finish work and needs to be scaled up.
Big jobs (high target_bytes_per_job or max_batch_size) can result in increased retries. Make the jobs smaller by reducing the target_bytes_per_job or max_batch_size to reduce the retries. If not, scale up the queriers.
Don’t set query_frontend.max_retries to a high number. The default of 2 is a good default.
Higher retries impact the backend when you have a big query that’s generating lots of jobs.
High max_retries can overwhelm the queriers when they’re under heavy load and failing to process the jobs. Retries can snowball, and degrade the query performance.

`query_frontend.search.target_bytes_per_job` parameter

The target number of bytes for each job to handle when performing a backend search.

This parameter controls the size of a single job, and size of a job dictates how much data a single job reads. You can tune this to make big or small jobs.

This option controls the upper limit on the size of a job, and can be used as a proxy for how much data a single search job scans or the amount of work a querier needs to do to process a job.

Guidelines

Setting this to a small value produces too many jobs, and results in more overhead, setting it too high produces big jobs. Queriers might struggle to finish those jobs and it can lead to high latency.
In testing at Grafana Labs, 100MB to 200MB is a good range for this configuration, and works across different sizes of clusters.
We recommend keeping this fixed within the recommended range.

`querier.frontend_worker.parallelism` parameter

Number of simultaneous queries to process per query-frontend or query-scheduler. This configuration controls the number of concurrent requests per query-frontend a querier process.

If parallelism is set to 5 and there are two query-frontends running, then a querier process opens 5 connections to each query-frontend, and in total of 10 connections.

One batch (or job, if batching is disabled) is processed per connection, so this controls the maximum number of concurrent jobs processed.

A single batch is processed in sync and size of the batch is controlled by max_batch_size, a connection is blocked until it’s process and returns the result of a batch.

You can also disable it and use max_concurrent_connections but we use this to make sure we queriers are picking up work from both query-frontends and jobs are being scheduled fairly.

If you want to make sure you always have connections defined in this config, you should set match_max_concurrent: false in your configuration to ensure you are not limited by max_concurrent_connections.

Guidelines

Maximum number of jobs a querier processes when using parallelism is equal to parallelism * query_frontend replicas * max_batch_size.
As you add more queriers, connections to an individual frontend increase. Query-frontend has a shared lock on these connections so if you see issues when you scale out queriers, lower the parallelism to reduce contention.
It’s recommended to set match_max_concurrent: false and not set max_concurrent_queries when using parallelism.

`querier.max_concurrent_queries` parameter

This controls the maximum number of jobs a querier processes concurrently. It doesn’t distinguish between the types of queries. These jobs can be from two different queries, or different tenants as well.

Querier processes these in sync, blocks the connection, and once it returns the results, it picks up a new job from the query-frontend queue.

Guidelines

The max_concurrent_queries parameter controls the number of queries that are processed by querier at once (concurrently).
if queriers are under-utilized, increase this value to process more jobs concurrently and process the queries faster.
If queriers are over utilized, seeing spikes, or OOMs, you can reduce this config to do less work per queriers and scale out queriers to add more capacity to the cluster.
This setting depends on the resources, in small queriers, it should be set to a lower number.
If you have large queriers, this configuration should be set to a higher value to fully utilize the capacity of the querier.

Querier memory sizing

If you are seeing that your queries are using more memory then you prefer, reduce the amount of work a querier is doing concurrently. That reduces resource usage.

If you want your queries to use more memory then they’re currently using, increase the amount of work a querier is doing concurrently and it increases the resource usage.

You can tune query_frontend.search.target_bytes_per_job, querier.frontend_worker.parallelism, and querier.max_concurrent_queries to tune the amount of work a querier is doing concurrently.

Queriers memory request roughly translates to job size times the concurrent work and some buffer.

Scaling cache

Tempo can be configured to use multiple caches for different types of data. When configured, Tempo uses these caches to improve the performance of queries.

Here are few general heuristics to know when to scale a cache:

Look at cache latency. If cache latency is hitting cache_timeout, that means that cache is under scaled and it’s taking too long to read or write, scale the cache.
Scale the cache if a cache has a high eviction rate. The cache might be under provisioned.
Lower level cache like bloom cache, parquet-page cache, parquet-footer cache sees higher hit rates (usually around 90% of above). If you have a consistent query traffic and these lower level caches have low hit rate, they’re undervalued and needs to be scaled up.
Higher level cache like frontend-search cache has a low hit rate and is only useful when the same query is being repeated. Size these according to the amount of data you want you want to cache
Cache sizes are also dictated by how much data you want to cache at each tier, it’s better to cache more at lower level caches because they have higher hit rate and is useful across queries.

Improve performance with caching

Thu, 09 Apr 2026 14:59:14 +0000

Improve performance with caching

Caching is mainly used to improve query performance by storing bloom filters of all backend blocks which are accessed on every query.

Tempo uses an external cache to improve query performance. Tempo supports Memcached and Redis.

For information about search performance, refer to Tune search performance.

Memcached

Memcached is one of the cache implementations supported by Tempo. It’s used by default in the Tanka and Helm examples. Refer to Deploying Tempo.

Connection limit

As a cluster grows in size, the number of instances of Tempo connecting to the cache servers also increases. By default, Memcached has a connection limit of 1024. Memcached refuses connections when this limit is surpassed. You can resolve this issue by increasing the connection limit of Memcached.

You can use the tempo_memcache_request_duration_seconds_count metric to observe these errors. For example, by using the following query:

sum by (status_code) (
  rate(tempo_memcache_request_duration_seconds_count{}[$__rate_interval])
)

This metric is also shown in the monitoring dashboards (the left panel):

Note that the already open connections continue to function. New connections are refused.

Additionally, Memcached logs the following errors when it can’t accept any new requests:

accept4(): No file descriptors available
Too many open connections
accept4(): No file descriptors available
Too many open connections

When using the memcached_exporter, you can observe the number of open connections at memcached_current_connections.

Cache size control

Tempo querier accesses bloom filters of all blocks while searching for a trace. This essentially mandates the size of cache to be at-least the total size of the bloom filters (the working set). However, in larger deployments, the working set might be larger than the desired size of cache. When that happens, eviction rates on the cache grow high, and hit rate drop.

Tempo provides two configuration parameters to filter down on the items stored in cache.

        # Min compaction level of block to qualify for caching bloom filter
        # Example: "cache_min_compaction_level: 2"
        [cache_min_compaction_level: <int>]

        # Max block age to qualify for caching bloom filter
        # Example: "cache_max_block_age: 48h"
        [cache_max_block_age: <duration>]

Using a combination of these configuration options, you can narrow down on which bloom filters are cached, thereby reducing the cache eviction rate, and increasing the cache hit rate.

In order to decide the values of these configuration parameters, you can use a cache summary command in the tempo-cli that prints a summary of bloom filter shards per day and per compaction level. The result looks something like this:

This image shows the bloom filter shards over 14 days and 6 compaction levels. This can be used to decide the configuration parameters.

Manage advanced systems

Thu, 09 Apr 2026 14:59:14 +0000

Manage advanced systems

This section provides resources for managing and tuning advanced Tempo configurations.

Tempo CLI

Thu, 09 Apr 2026 14:59:14 +0000

Tempo CLI

Tempo CLI is a separate executable that contains utility functions related to the Tempo software. Although it’s not required for a working installation, Tempo CLI can be helpful for deeper analysis or for troubleshooting.

Tempo CLI command syntax

The general syntax for commands in Tempo CLI is:

tempo-cli command [subcommand] [options] [arguments...]

--help or -h displays the help for a command or subcommand.

Example:

tempo-cli -h
tempo-cli command [subcommand] -h

Run Tempo CLI

Tempo CLI is currently available as source code.

To build Tempo CLI, you need a working Go installation and a build environment. You can compile it to a native binary and execute normally, or you can execute it using the go run command. You can also package it as a binary in a Docker container using make docker-tempo-cli.

Example:

./tempo-cli [arguments...]
go run ./cmd/tempo-cli [arguments...]

make docker-tempo-cli
docker run docker.io/grafana/tempo-cli [arguments...]

Backend options

Tempo CLI connects directly to the storage backend for some commands, meaning that it requires the ability to read from S3, GCS, Azure or file-system storage. You can configure the backend using the following options:

Load an existing tempo configuration file using the --config-file (-c) option. This is the recommended option for frequent usage. Refer to Configuration documentation for more information.
Specify individual settings:
- --backend <value> The storage backend type, one of s3, gcs, azure, and local.
- --bucket <value> The bucket name. The meaning of this value is backend-specific. Refer to Configuration documentation for more information.
- --s3-endpoint <value> The S3 API endpoint (i.e. s3.dualstack.us-east-2.amazonaws.com).
- --s3-user <value>, --s3-pass <value> The S3 user name and password (or access key and secret key). Optional, as Tempo CLI supports the same authentication mechanisms as Tempo. Refer to S3 permissions documentation for more information.
- --insecure-skip-verify skip TLS verification, only applies to S3 and GCS.

Each option applies only to the command in which it’s used. For example, --backend <value> doesn’t permanently change where Tempo stores data. It only changes it for command in which you apply the option.

Query API command

Trace ID

Call the Tempo API and retrieve a trace by ID.

tempo-cli query api trace-id <api-endpoint> <trace-id>

Arguments:

api-endpoint URL for the Tempo API.
trace-id Trace ID as a hexadecimal string.

Options:

--org-id <value> Organization ID (for use in multi-tenant setup).
--v1 use v1 API (use /api/traces endpoint to fetch traces, default: /api/v2/traces).

Example:

tempo-cli query api trace-id http://tempo:3200 f1cfe82a8eef933b

Search

Call the Tempo API and search using TraceQL.

tempo-cli query api search <host-port> <trace-ql> [<start> <end>]

Arguments:

host-port A host/port combination for Tempo. The scheme is inferred from the options.
trace-ql TraceQL query.
start Start of the time range to search: (YYYY-MM-DDThh:mm:ss)
end End of the time range to search: (YYYY-MM-DDThh:mm:ss)

Options:

--org-id <value> Organization ID (for use in multi-tenant setup).
--use-grpc Use GRPC streaming
--spss <value> Number of spans to return for each spanset
--limit <value> Number of results to return
--path-prefix <value> String to prefix search paths with
--secure Use HTTPS or gRPC with TLS

Note
Set the stream_over_http_enabled flag to true in the Tempo configuration to enable streaming over HTTP. For more information, refer to Tempo GRPC API documentation.

Search tags

Call the Tempo API and search attribute names.

tempo-cli query api search-tags <host-port> [<start> <end>]

Arguments:

host-port A host/port combination for Tempo. The scheme will be inferred based on the options provided.
start Start of the time range to search: (YYYY-MM-DDThh:mm:ss)
end End of the time range to search: (YYYY-MM-DDThh:mm:ss)

Options:

--org-id <value> Organization ID (for use in multi-tenant setup).
--use-grpc Use GRPC streaming
--path-prefix <value> String to prefix search paths with
--secure Use HTTPS or gRPC with TLS

Note
Set the stream_over_http_enabled flag to true in the Tempo configuration to enable streaming over HTTP. For more information, refer to Tempo GRPC API documentation.

Search tag values

Call the Tempo API and search attribute values.

tempo-cli query api search-tag-values <host-port> <tag> [<start> <end>]

Arguments:

host-port A host/port combination for Tempo. The scheme is inferred from the options.
tag The fully qualified TraceQL tag to search for. For example, resource.service.name.
start Start of the time range to search: (YYYY-MM-DDThh:mm:ss)
end End of the time range to search: (YYYY-MM-DDThh:mm:ss)

Options:

--org-id <value> Organization ID (for use in multi-tenant setup).
--use-grpc Use GRPC streaming
--path-prefix <value> String to prefix search paths with
--secure Use HTTP or gRPC with TLS

Note
Set the stream_over_http_enabled flag to true in the Tempo configuration to enable streaming over HTTP. For more information, refer to Tempo GRPC API documentation.

Metrics

Call the Tempo API and generate metrics from traces using TraceQL.

tempo-cli query api metrics <host-port> <trace-ql metrics query> [<start> <end>]

Arguments:

host-port A host/port combination for Tempo. The scheme will be inferred based on the options provided.
trace-ql metrics query TraceQL metrics query.
start Start of the time range to search: (YYYY-MM-DDThh:mm:ss)
end End of the time range to search: (YYYY-MM-DDThh:mm:ss)

Options:

--org-id <value> Organization ID (for use in multi-tenant setup).
--use-grpc Use GRPC streaming
--path-prefix <value> String to prefix search paths with
--secure Use HTTPS or gRPC with TLS

Note
Set the stream_over_http_enabled flag to true in the Tempo configuration to enable streaming over HTTP. For more information, refer to Tempo GRPC API documentation.

Query blocks command

Iterate over all backend blocks and dump all data found for a given trace id.

tempo-cli query blocks <trace-id> <tenant-id>

Note
This can be intense as it downloads every bloom filter and some percentage of indexes/trace data.

Arguments:

trace-id Trace ID as a hexadecimal string.
tenant-id Tenant to search.

Options: See backend options above.

Example:

tempo-cli query blocks f1cfe82a8eef933b single-tenant

Query trace summary command

Iterate over all backend blocks and dump a summary for a given trace id.

The summary includes:

number of blocks the trace is found in
span count
trace size
trace duration
root service name
root span info
top frequent service names

tempo-cli query trace-summary <trace-id> <tenant-id>

Note: can be intense as it downloads every bloom filter and some percentage of indexes/trace data.

Arguments:

trace-id Trace ID as a hexadecimal string.
tenant-id Tenant to search.

Options: See backend options above.

Example:

tempo-cli query trace-summary f1cfe82a8eef933b single-tenant

List blocks

Lists information about all blocks for the given tenant, and optionally perform integrity checks on indexes for duplicate records.

tempo-cli list blocks <tenant-id>

Arguments:

tenant-id The tenant ID. Use single-tenant for single tenant setups.

Options:

--include-compacted Include blocks that have been compacted. Default behavior is to display only active blocks.

Output: Explanation of output:

ID Block ID.
Lvl Compaction level of the block.
Objects Number of objects stored in the block.
Size Data size of the block after any compression.
Encoding Block encoding (compression algorithm).
Vers Block version.
Window The window of time that was considered for compaction purposes.
Start The earliest timestamp stored in the block.
End The latest timestamp stored in the block.
DurationDuration between the start and end time.
Age The age of the block.
Cmp Whether the block has been compacted (present when –include-compacted is specified).

Example:

tempo-cli list blocks -c ./tempo.yaml single-tenant

List block

Lists information about a single block, and optionally, scan its contents.

tempo-cli list block <tenant-id> <block-id>

Arguments:

tenant-id The tenant ID. Use single-tenant for single tenant setups.
block-id The block ID as UUID string.

Options:

--scan Also load the block data, perform integrity check for duplicates, and collect statistics. Note: can be intense.

Example:

tempo-cli list block -c ./tempo.yaml single-tenant ca314fba-efec-4852-ba3f-8d2b0bbf69f1

List compaction summary

Summarizes information about all blocks for the given tenant based on compaction level. This command is useful to analyze or troubleshoot compactor behavior.

tempo-cli list compaction-summary <tenant-id>

Arguments:

tenant-id The tenant ID. Use single-tenant for single tenant setups.

Example:

tempo-cli list compaction-summary -c ./tempo.yaml single-tenant

List cache summary

Prints information about the number of bloom filter shards per day per compaction level. This command is useful to estimate and fine-tune cache storage. Read the caching topic for more information.

tempo-cli list cache-summary <tenant-id>

Arguments:

tenant-id The tenant ID. Use single-tenant for single tenant setups.

Example:

tempo-cli list cache-summary -c ./tempo.yaml single-tenant

List index

Lists basic index info for the given block.

tempo-cli list index <tenant-id> <block-id>

Arguments:

tenant-id The tenant ID. Use single-tenant for single tenant setups.
block-id The block ID as UUID string.

Example:

tempo-cli list index -c ./tempo.yaml single-tenant ca314fba-efec-4852-ba3f-8d2b0bbf69f1

View index

View the index contents for the given block.

tempo-cli view index <tenant-id> <block-id>

Arguments:

tenant-id The tenant ID. Use single-tenant for single tenant setups.
block-id The block ID as UUID string.

Example:

tempo-cli view index -c ./tempo.yaml single-tenant ca314fba-efec-4852-ba3f-8d2b0bbf69f1

Generate bloom filter

To generate the bloom filter for a block if the files were deleted/corrupted.

Note: ensure that the block is in a local backend in the expected directory hierarchy, i.e. path / tenant / blocks.

Arguments:

tenant-id The tenant ID. Use single-tenant for single tenant setups.
block-id The block ID as UUID string.
bloom-fp The false positive to be used for the bloom filter.
bloom-shard-size The shard size to be used for the bloom filter.

Example:

tempo-cli gen bloom --backend=local --bucket=./cmd/tempo-cli/test-data/ single-tenant b18beca6-4d7f-4464-9f72-f343e688a4a0 0.05 100000

The bloom filter will be generated at the required location under the block folder.

Generate index

To generate the index/bloom for a block if the files were deleted/corrupted.

Note: ensure that the block is in a local backend in the expected directory hierarchy, i.e. path / tenant / blocks.

Arguments:

tenant-id The tenant ID. Use single-tenant for single tenant setups.
block-id The block ID as UUID string.

Example:

tempo-cli gen index --backend=local --bucket=./cmd/tempo-cli/test-data/ single-tenant b18beca6-4d7f-4464-9f72-f343e688a4a0

The index will be generated at the required location under the block folder.

Search blocks command

Search blocks in a given time range for a specific key/value pair.

tempo-cli search blocks <name> <value> <start> <end> <tenant-id>

Note: can be intense as it downloads all relevant blocks and iterates through them.

Arguments:

name Name of the attribute to search for e.g. http.post
value Value of the attribute to search for e.g. GET
start Start of the time range to search: (YYYY-MM-DDThh:mm:ss)
end End of the time range to search: (YYYY-MM-DDThh:mm:ss)
tenant-id Tenant to search.

Options: See backend options above.

Example:

tempo-cli search blocks http.post GET 2021-09-21T00:00:00 2021-09-21T00:05:00 single-tenant --backend=gcs --bucket=tempo-trace-data

Parquet convert command

Converts a parquet file from its existing schema to the one currently in the repository. This utility command is useful when attempting to determine the impact of changing compression or encoding of columns.

tempo-cli parquet convert <in file> <out file>

Arguments:

in file Filename of an existing parquet file containing Tempo trace data
out file File to write to. (Existing file is overwritten.)

Example:

tempo-cli parquet convert data.parquet out.parquet

Parquet convert A to B command

Converts a vParquet file (actual data.parquet) of format A to a block of newer format B with an optional list of dedicated attribute columns. Actual supported versions for A and B vary by Tempo release. This utility command is useful when testing the impact of different combinations of dedicated columns.

Convert vParquet3 to vParquet4

tempo-cli parquet convert-3-to-4 <in file> [<out path>] [<list of dedicated columns>]

Arguments:

in file Path to an existing vParquet3 block directory.
out path Path to write the vParquet4 block to. The default is ./out.
list of dedicated columns Optional list of columns to make dedicated. Columns use TraceQL syntax with scope. For example, span.db.statement, resource.namespace.

Example:

tempo-cli parquet convert-3-to-4 data.parquet ./out span.db.statement span.db.name

Convert vParquet4 to vParquet5

Converts a vParquet4 block to vParquet5 format with an optional list of dedicated attribute columns.

tempo-cli parquet convert-4-to-5 <in file> [<out path>] [<list of dedicated columns>]

Arguments:

in file Path to an existing vParquet4 block directory.
out path Path to write the vParquet5 block to. The default is ./out.
list of dedicated columns Optional list of columns to make dedicated. Columns use TraceQL syntax with scope. For example, span.http.method, resource.namespace, event.exception.message.

Column prefixes:

int/ marks the column as an integer type. For example, int/span.http.status_code.
blob/ marks the column for blob encoding. For example, blob/span.db.statement.

Example:

tempo-cli parquet convert-4-to-5 ./block-in ./block-out "span.http.method" "int/span.http.status_code" "blob/span.db.statement"

Migrate tenant command

Copy blocks from one backend and tenant to another. Blocks can be copied within the same backend or between two different backends. The data format isn’t converted but the tenant ID in meta.json is rewritten.

tempo-cli migrate tenant <source tenant> <dest tenant>

Arguments:

source tenant Tenant to copy blocks from
dest tenant Tenant to copy blocks into

Options:

--source-config-file <value> Configuration file for the source backend
--config-file <value> Configuration file for the destination backend

Example:

tempo-cli migrate tenant --source-config source.yaml --config-file dest.yaml my-tenant my-other-tenant

Migrate overrides config command

Migrate overrides config from inline format (legacy) to idented YAML format (new).

tempo-cli migrate overrides-config <source config file>

Arguments:

source config file Configuration file to migrate

Options:

--config-dest <value> Destination file for the migrated config. If not specified, config is printed to stdout.
--overrides-dest <value> Destination file for the migrated overrides. If not specified, overrides are printed to stdout.

Example:

tempo-cli migrate overrides-config config.yaml --config-dest config-tmp.yaml --overrides-dest overrides-tmp.yaml

Analyse block

Analyses a block and outputs a summary of the block’s generic attributes.

It’s of particular use when trying to determine candidates for dedicated attribute columns in vParquet3+. The output includes span, resource, and event attributes with cardinality and size information.

Arguments:

tenant-id The tenant ID. Use single-tenant for single tenant setups.
block-id The block ID as UUID string.

Options:

Backend options
--num-attr <value> Number of attributes to output (default: 15)
--num-int-attr <value> Number of integer attributes to display. If set to 0, uses the --num-attr value (default: 5)
--blob-threshold <value> Mark attributes as blob candidates when their dictionary size per row group exceeds this value. Set to 0 to disable. (default: 4MiB)
--include-well-known Include well-known attributes in the analysis. Enable when generating dedicated columns for vParquet5 or higher. (default: false)
--generate-jsonnet Generate Jsonnet overrides for dedicated columns
--generate-cli-args Generate command-line arguments for the parquet conversion command
--int-percent-threshold <value> Threshold for integer attributes in dedicated columns (default: 0.05)
--simple-summary Print only a single line of top attributes (default: false)
--print-full-summary Print full summary of the analysed block (default: true)

Example:

tempo-cli analyse block --backend=local --bucket=./cmd/tempo-cli/test-data/ single-tenant b18beca6-4d7f-4464-9f72-f343e688a4a0

Example with blob detection:

tempo-cli analyse block --blob-threshold=4MiB --generate-jsonnet --backend=local --bucket=./cmd/tempo-cli/test-data/ single-tenant b18beca6-4d7f-4464-9f72-f343e688a4a0

Analyse blocks

Analyses all blocks in a given time range and outputs a summary of the blocks’ generic attributes.

Arguments:

tenant-id The tenant ID. Use single-tenant for single-tenant setups.

Options:

Backend options
--num-attr <value> Number of attributes to output (default: 15)
--num-int-attr <value> Number of integer attributes to display. If set to 0, uses the --num-attr value (default: 5)
--min-compaction-level <value> Minimum compaction level to include in the analysis (default: 3)
--max-blocks <value> Maximum number of blocks to analyze (default: 10)
--max-start-time <value> Oldest start time for a block to be processed. RFC3339 format (default: disabled)
--min-start-time <value> Newest start time for a block to be processed. RFC3339 format (default: disabled)
--blob-threshold <value> Mark attributes as blob candidates when their dictionary size per row group exceeds this value. Set to 0 to disable. (default: 4MiB)
--include-well-known Include well-known attributes in the analysis. (default: false)
--jsonnet Generate Jsonnet overrides for dedicated columns
--cli Generate command-line arguments for the parquet conversion command
--int-percent-threshold <value> Threshold for integer attributes in dedicated columns (default: 0.05)
--simple-summary Print only a single line of top attributes (default: false)
--print-full-summary Print full summary of the analysed block (default: true)

Example:

tempo-cli analyse blocks --backend=local --bucket=./cmd/tempo-cli/test-data/ single-tenant

Example with blob detection and Jsonnet output:

tempo-cli analyse blocks --blob-threshold=4MiB --jsonnet --backend=local --bucket=./cmd/tempo-cli/test-data/ single-tenant

Drop traces by ID

Rewrites all blocks for a tenant that contain a specific trace IDs. The traces are dropped from the new blocks and the rewritten blocks are marked compacted so they will be cleaned up.

Arguments:

tenant-id The tenant ID. Use single-tenant for single tenant setups.
trace-ids The comma-separated trace IDs to drop (also supports single trace ID)

Options:

Backend options
--drop-traces By default, this command runs in dry run mode. Supplying this argument causes it to actually rewrite blocks with the traces dropped.

Examples

Drop one trace:

tempo-cli rewrite-blocks drop-trace --backend=local --bucket=./cmd/tempo-cli/test-data/ single-tenant 04d5f549746c96e4f3daed6202571db2

Drop multiple traces:

tempo-cli rewrite-blocks drop-trace --backend=local --bucket=./cmd/tempo-cli/test-data/ single-tenant 04d5f549746c96e4f3daed6202571db2,111fa1850042aea83c17cd7e674210b8

Apache Parquet schema

Thu, 09 Apr 2026 14:59:14 +0000

Apache Parquet schema

Starting with Tempo 2.0, Apache Parquet is used as the default column-formatted block format. Refer to the Parquet configuration options for more information.

This document describes the schema used with the Parquet block format.

Version applicability

Tempo 2.10 defaults to the vParquet4 schema. vParquet5 is production-ready and differs in some schema details. Unless otherwise noted, the sections below describe vParquet4.

The following sections apply to both vParquet4 and vParquet5:

Fully nested versus span-oriented schema
Static vs dynamic columns (see vParquet5 differences for changes to dedicated columns)
Compression and encoding
Bloom filters

Fully nested versus span-oriented schema

There are two overall approaches to a columnar schema: fully nested or span-oriented. Span-oriented means a flattened schema where traces are destructured into rows of spans. A fully nested schema means the current trace structures such as Resource/Scope/Spans/Events are preserved (nested data is natively supported in Parquet). In both cases, individual leaf values such as span name and duration are individual columns.

We chose the nested schema for several reasons:

The block size is much smaller for the nested schema. This is due to the high data duplication incurred when flattening resource-level attributes such as service.name to each individual span.
A flat schema is not truly “flat” because each span still contains nested data such as attributes and events.
Nested schema is much faster to search for resource-level attributes because the resource-level columns are very small (1 row for each batch).
Translation to and from the OpenTelemetry Protocol Specification (OTLP) is straightforward.
Easily add computed columns (for example, trace duration) at multiple levels such as per-trace, per-batch, etc.

Static vs dynamic columns

Dynamic vs static columns add another layer to the schema. A dynamic schema stores each attribute such as service.name and http.status_code as its own column and the columns in each parquet file can be different. A static schema is unresponsive to the shape of the data, and all attributes are stored in generic key/value containers.

The dynamic schema is the ultimate dream for a columnar format but it is too complex for a first release. However, the benefits of that approach are also too good to pass up, so we propose a hybrid approach. It is primarily a static schema but with some dynamic columns extracted from trace data based on some heuristics of frequently queried attributes. We plan to continue investing in this direction to implement a fully dynamic schema where trace attributes are blown out into independent Parquet columns at runtime.

For more information, refer to the Parquet design document.

Schema details

The adopted Parquet schema is mostly a direct translation of OTLP but with some key differences.

The table below uses these abbreviations:

rs - resource spans
ss - scope spans


Name	Type	Description
TraceID	byte array	The trace ID in 16-byte binary form.
TraceIDText	string	The trace ID in hexadecimal text form.
StartTimeUnixNano	int64	Start time of the first span in the trace, in nanoseconds since unix epoch.
EndTimeUnixNano	int64	End time of the last span in the trace, in nanoseconds since unix epoch.
DurationNano	int64	Total trace duration in nanoseconds, computed as difference between EndTimeUnixNano and StartTimeUnixNano.
RootServiceName	string	The resource-level `service.name` attribute (rs.Resource.ServiceName) from the root span of the trace if one exists, else empty string.
RootSpanName	string	The name (rs.ss.Spans.Name) of the root span if one exists, else empty string.
ServiceStats	map	Per-service counts keyed by service name. Values include span count and error count.
rs		Short-hand for ResourceSpans
rs.Resource.ServiceName	string	A dedicated column for the resource-level `service.name` attribute if present. https://opentelemetry.io/docs/reference/specification/resource/semantic_conventions/#service
rs.Resource.Cluster	string	A dedicated column for the resource-level `cluster` attribute if present and of string type. Values of other types will be stored in the generic attribute columns.
rs.Resource.Namespace	string	A dedicated column for the resource-level `namespace` attribute if present and of string type. Values of other types will be stored in the generic attribute columns.
rs.Resource.Pod	string	A dedicated column for the resource-level `pod` attribute if present and of string type. Values of other types will be stored in the generic attribute columns.
rs.Resource.Container	string	A dedicated column for the resource-level `container` attribute if present and of string type. Values of other types will be stored in the generic attribute columns.
rs.Resource.K8sClusterName	string	A dedicated column for the resource-level `k8s.cluster.name` attribute if present and of string type. Values of other types will be stored in the generic attribute columns. https://opentelemetry.io/docs/reference/specification/resource/semantic_conventions/k8s/#cluster
rs.Resource.K8sNamespaceName	string	A dedicated column for the resource-level `k8s.namespace.name` attribute if present and of string type. Values of other types will be stored in the generic attribute columns. https://opentelemetry.io/docs/reference/specification/resource/semantic_conventions/k8s/#namespace
rs.Resource.K8sPodName	string	A dedicated column for the resource-level `k8s.pod.name` attribute if present and of string type. Values of other types will be stored in the generic attribute columns. https://opentelemetry.io/docs/reference/specification/resource/semantic_conventions/k8s/#pod
rs.Resource.K8sContainerName	string	A dedicated column for the resource-level `k8s.container.name` attribute if present and of string type. Values of other types will be stored in the generic attribute columns. https://opentelemetry.io/docs/reference/specification/resource/semantic_conventions/k8s/#container
rs.Resource.DroppedAttributesCount	int	Number of resource attributes that were dropped.
rs.Resource.Attrs.Key	string	All resource attributes that do not have a dedicated column are stored as a key value pair in these columns. The Key column stores the name, and then one of the Value columns is populated according to the attribute’s data type. The other value columns will contain null.
rs.Resource.Attrs.IsArray	bool	Indicates if the attribute is stored as an array.
rs.Resource.Attrs.Value	string	The attribute value if string type (or array of strings), else null.
rs.Resource.Attrs.ValueInt	int	The attribute value if integer type (or array of integers), else null.
rs.Resource.Attrs.ValueDouble	float	The attribute value if float type (or array of floats), else null.
rs.Resource.Attrs.ValueBool	bool	The attribute value if boolean type (or array of booleans), else null.
rs.Resource.Attrs.ValueUnsupported	string	JSON-encoded AnyValue for unsupported or mixed-type values.
rs.Resource.DedicatedAttributes		Group containing spares for dedicated attribute columns with resource scope.
rs.Resource.DedicatedAttributes.String01 … String10	string	10 spare columns for dedicated attribute columns.
rs.ss		Shorthand for ResourceSpans.ScopeSpans
rs.ss.Scope		Shorthand for ResourceSpans.ScopeSpans.Scope
rs.ss.Scope.Name	string	Scope name if present, else empty string. https://opentelemetry.io/docs/specs/otel/glossary/#instrumentation-scope
rs.ss.Scope.Version	string	The Scope version if present, else empty string. https://opentelemetry.io/docs/specs/otel/glossary/#instrumentation-scope
rs.ss.Scope.Attrs		Scope attributes, using the same columns as rs.Resource.Attrs.*
rs.ss.Scope.DroppedAttributesCount	int	Number of scope attributes that were dropped.
rs.ss.Spans.SpanID	byte array	Span unique ID.
rs.ss.Spans.ParentSpanID	byte array	The unique ID of the span’s parent. For root spans without a parent this is null.
rs.ss.Spans.ParentID	int32	Trace local numeric parent ID.
rs.ss.Spans.NestedSetLeft	int32	Left bound of the nested set model. Also used as a trace local numeric span ID.
rs.ss.Spans.NestedSetRight	int32	Right bound of the nested set model.
rs.ss.Spans.Name	string	Span name.
rs.ss.Spans.StartTimeUnixNano	int64	Start time the span in nanoseconds since unix epoch.
rs.ss.Spans.DurationNano	int64	Span duration in nanoseconds.
rs.ss.Spans.Kind	int	The span’s kind. Defined values: 0. Unset; 1. Internal; 2. Server; 3. Client; 4. Producer; 5. Consumer; https://opentelemetry.io/docs/reference/specification/trace/api/#spankind
rs.ss.Spans.StatusCode	int	The span status. Defined values: 0: Unset; 1: OK; 2: Error. https://opentelemetry.io/docs/reference/specification/trace/api/#set-status
rs.ss.Spans.StatusMessage	string	Optional message to accompany Error status.
rs.ss.Spans.HttpMethod	string	A dedicated column for the span-level `http.method` attribute if present and of string type, else null. Values of other types will be stored in the generic attribute columns. https://opentelemetry.io/docs/reference/specification/trace/semantic_conventions/http/#common-attributes
rs.ss.Spans.HttpStatusCode	int	A dedicated column for the span-level `http.status_code` attribute if present and of integer type, else null. Values of other types will be stored in the generic attribute columns. https://opentelemetry.io/docs/reference/specification/trace/semantic_conventions/http/#common-attributes
rs.ss.Spans.HttpUrl	string	A dedicated column for the span-level `http.url` attribute if present and of string type, else null. Values of other types will be stored in the generic attribute columns. https://opentelemetry.io/docs/reference/specification/trace/semantic_conventions/http/#http-client
rs.ss.Spans.DroppedAttributesCount	int	Number of attributes that were dropped
rs.ss.Spans.Attrs		Span attributes, using the same columns as rs.Resource.Attrs.*
rs.ss.Spans.DedicatedAttributes		Group containing spares for dedicated attribute columns with span scope
rs.ss.Spans.DedicatedAttributes.String01 … String10	string	10 spare columns used for dedicated attributes
rs.ss.Spans.DroppedEventsCount	int	The number of events that were dropped
rs.ss.Spans.Events.TimeSinceStartNano	int64	The event timestamp in nanoseconds, relative to the span start time.
rs.ss.Spans.Events.Name	string	The event name or message.
rs.ss.Spans.Events.DroppedAttributesCount	int	The number of event attributes that were dropped.
rs.ss.Spans.Events.Attrs		Event attributes, using the same columns as rs.Resource.Attrs.*
rs.ss.Spans.DroppedLinksCount	int	The number of links that were dropped.
rs.ss.Spans.Links		Repeated link records with TraceID, SpanID, TraceState, Attrs, and DroppedAttributesCount.
rs.ss.Spans.TraceState	string	The span’s TraceState value if present, else empty string. https://opentelemetry.io/docs/reference/specification/trace/api/#tracestate

To increase the readability the table omits the groups list.element that are added for nested list types in Parquet. For maps (for example, ServiceStats), Parquet also inserts map-specific group levels that are omitted here.

message Trace {
  required binary TraceID;
  required binary TraceIDText (STRING);
  required int64 StartTimeUnixNano (INTEGER(64,false));
  required int64 EndTimeUnixNano (INTEGER(64,false));
  required int64 DurationNano (INTEGER(64,false));
  required binary RootServiceName (STRING);
  required binary RootSpanName (STRING);
  optional group ServiceStats (MAP) {
    repeated group key_value {
      required binary key (STRING);
      required group value {
        required int32 SpanCount (INTEGER(32,false));
        required int32 ErrorCount (INTEGER(32,false));
      }
    }
  }
  required group rs (LIST) {
    repeated group list {
      required group element {
        required group Resource {
          required group Attrs (LIST) {
            repeated group list {
              required group element {
                required binary Key (STRING);
                required boolean IsArray;
                required group Value (LIST) {
                  repeated group list {
                    required binary element (STRING);
                  }
                }
                required group ValueInt (LIST) {
                  repeated group list {
                    required int64 element (INTEGER(64,true));
                  }
                }
                required group ValueDouble (LIST) {
                  repeated group list {
                    required double element;
                  }
                }
                required group ValueBool (LIST) {
                  repeated group list {
                    required boolean element;
                  }
                }
                optional binary ValueUnsupported (STRING);
              }
            }
          }
          required int32 DroppedAttributesCount (INTEGER(32,true));
          required binary ServiceName (STRING);
          optional binary Cluster (STRING);
          optional binary Namespace (STRING);
          optional binary Pod (STRING);
          optional binary Container (STRING);
          optional binary K8sClusterName (STRING);
          optional binary K8sNamespaceName (STRING);
          optional binary K8sPodName (STRING);
          optional binary K8sContainerName (STRING);
          required group DedicatedAttributes {
            optional binary String01 (STRING);
            optional binary String02 (STRING);
            optional binary String03 (STRING);
            optional binary String04 (STRING);
            optional binary String05 (STRING);
            optional binary String06 (STRING);
            optional binary String07 (STRING);
            optional binary String08 (STRING);
            optional binary String09 (STRING);
            optional binary String10 (STRING);
          }
        }
        required group ss (LIST) {
          repeated group list {
            required group element {
              required group Scope {
                required binary Name (STRING);
                required binary Version (STRING);
                required group Attrs (LIST) {
                  repeated group list {
                    required group element {
                      required binary Key (STRING);
                      required boolean IsArray;
                      required group Value (LIST) {
                        repeated group list {
                          required binary element (STRING);
                        }
                      }
                      required group ValueInt (LIST) {
                        repeated group list {
                          required int64 element (INTEGER(64,true));
                        }
                      }
                      required group ValueDouble (LIST) {
                        repeated group list {
                          required double element;
                        }
                      }
                      required group ValueBool (LIST) {
                        repeated group list {
                          required boolean element;
                        }
                      }
                      optional binary ValueUnsupported (STRING);
                    }
                  }
                }
                required int32 DroppedAttributesCount (INTEGER(32,true));
              }
              required group Spans (LIST) {
                repeated group list {
                  required group element {
                    required binary SpanID;
                    required binary ParentSpanID;
                    required int32 ParentID (INTEGER(32,true));
                    required int32 NestedSetLeft (INTEGER(32,true));
                    required int32 NestedSetRight (INTEGER(32,true));
                    required binary Name (STRING);
                    required int64 Kind (INTEGER(64,true));
                    required binary TraceState (STRING);
                    required int64 StartTimeUnixNano (INTEGER(64,false));
                    required int64 DurationNano (INTEGER(64,false));
                    required int64 StatusCode (INTEGER(64,true));
                    required binary StatusMessage (STRING);
                    required group Attrs (LIST) {
                      repeated group list {
                        required group element {
                          required binary Key (STRING);
                          required boolean IsArray;
                          required group Value (LIST) {
                            repeated group list {
                              required binary element (STRING);
                            }
                          }
                          required group ValueInt (LIST) {
                            repeated group list {
                              required int64 element (INTEGER(64,true));
                            }
                          }
                          required group ValueDouble (LIST) {
                            repeated group list {
                              required double element;
                            }
                          }
                          required group ValueBool (LIST) {
                            repeated group list {
                              required boolean element;
                            }
                          }
                          optional binary ValueUnsupported (STRING);
                        }
                      }
                    }
                    required int32 DroppedAttributesCount (INTEGER(32,true));
                    required group Events (LIST) {
                      repeated group list {
                        required group element {
                          required int64 TimeSinceStartNano (INTEGER(64,false));
                          required binary Name (STRING);
                          required group Attrs (LIST) {
                            repeated group list {
                              required group element {
                                required binary Key (STRING);
                                required boolean IsArray;
                                required group Value (LIST) {
                                  repeated group list {
                                    required binary element (STRING);
                                  }
                                }
                                required group ValueInt (LIST) {
                                  repeated group list {
                                    required int64 element (INTEGER(64,true));
                                  }
                                }
                                required group ValueDouble (LIST) {
                                  repeated group list {
                                    required double element;
                                  }
                                }
                                required group ValueBool (LIST) {
                                  repeated group list {
                                    required boolean element;
                                  }
                                }
                                optional binary ValueUnsupported (STRING);
                              }
                            }
                          }
                          required int32 DroppedAttributesCount (INTEGER(32,true));
                        }
                      }
                    }
                    required int32 DroppedEventsCount (INTEGER(32,true));
                    required group Links (LIST) {
                      repeated group list {
                        required group element {
                          required binary TraceID;
                          required binary SpanID;
                          required binary TraceState (STRING);
                          required group Attrs (LIST) {
                            repeated group list {
                              required group element {
                                required binary Key (STRING);
                                required boolean IsArray;
                                required group Value (LIST) {
                                  repeated group list {
                                    required binary element (STRING);
                                  }
                                }
                                required group ValueInt (LIST) {
                                  repeated group list {
                                    required int64 element (INTEGER(64,true));
                                  }
                                }
                                required group ValueDouble (LIST) {
                                  repeated group list {
                                    required double element;
                                  }
                                }
                                required group ValueBool (LIST) {
                                  repeated group list {
                                    required boolean element;
                                  }
                                }
                                optional binary ValueUnsupported (STRING);
                              }
                            }
                          }
                          required int32 DroppedAttributesCount (INTEGER(32,true));
                        }
                      }
                    }
                    required int32 DroppedLinksCount (INTEGER(32,true));
                    optional binary HttpMethod (STRING);
                    optional binary HttpUrl (STRING);
                    optional int64 HttpStatusCode (INTEGER(64,true));
                    required group DedicatedAttributes {
                      optional binary String01 (STRING);
                      optional binary String02 (STRING);
                      optional binary String03 (STRING);
                      optional binary String04 (STRING);
                      optional binary String05 (STRING);
                      optional binary String06 (STRING);
                      optional binary String07 (STRING);
                      optional binary String08 (STRING);
                      optional binary String09 (STRING);
                      optional binary String10 (STRING);
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

For the authoritative schema, refer to tempodb/encoding/vparquet4/schema.go and tempodb/encoding/vparquet5/schema.go.

Summary of vParquet5 differences

Resource-level dedicated columns (Cluster/Namespace/Pod/Container/K8s*) and span HTTP columns are removed; vParquet5 relies on dynamically assigned dedicated columns only.
Dedicated attribute columns expand to include integer spares, array attributes, and optional blob configuration for selected columns.
Additional fields exist for optimization, including span ChildCount, rounded start time buckets, and trace-level ServiceStats as a list with explicit service names.

Trace-level attributes

For speed and ease-of-use, we are projecting several values to columns at the trace-level:

Trace ID - Don’t store on each span.
Root service/span names/StartTimeUnixNano - These are selected properties of the root span in each trace (if there is one). These are used for displaying results in the Grafana UI. These properties are computed at ingest time and stored once for efficiency, so we don’t have to find the root span.
DurationNano - The total trace duration, computed at ingest time. This powers the min/max duration filtering in the current Tempo search and is more efficient than scanning the spans duration column. However, it may go away with TraceQL or we could decide to change it to span-level duration filtering too.
ServiceStats - Per-service span and error counts for each trace.

“Any”-type Attributes

OTLP attributes have variable data types, which is simpler in formats like protocol-buffers, but doesn’t translate directly to Parquet. Each column must have a concrete type. There are several possibilities here but we chose to have optional values for each concrete type. Unsupported or mixed-type values are stored as JSON-encoded AnyValue in ValueUnsupported.

Only scalar values are stored in the dedicated attribute columns. Arrays and unsupported types remain in the generic attribute columns.

repeated group Attrs {
required binary Key (STRING);
required boolean IsArray;
repeated binary Value (STRING);
repeated int64 ValueInt (INTEGER(64,true));
repeated double ValueDouble;
repeated boolean ValueBool;
optional binary ValueUnsupported (STRING);
}

Compression and encoding

Parquet has robust support for many compression algorithms and data encodings. We’ve found excellent combinations of storage size and performance with the following:

Snappy Compression - Enable on all columns
Dictionary encoding - Enable on all string columns. Most strings are very repetitive so this works well to optimize storage size. However, you can greatly speed up search by inspecting the dictionary first and eliminating pages with no matches.
Time and duration UNIX nanos - Delta encoding
Rarely used columns such as DroppedAttributesCount - These columns are usually all zeroes, RLE works well.

Bloom filters

Parquet has native support for bloom filters. However, Tempo doesn’t use them at this time. Tempo already has sophisticated support for sharding and caching bloom filters.

Manage authentication

Thu, 09 Apr 2026 14:59:14 +0000

Manage authentication

Grafana Tempo does not come with any included authentication layer. You must run an authenticating reverse proxy in front of your services.

We recommend that in all deployment modes you add a reverse proxy to be deployed in front of Tempo, to direct client API requests to the various components.

A list of open-source reverse proxies you can use:

HAProxy
NGINX using their guide on restricting access with HTTP basic authentication
OAuth2 proxy
Pomerium, which has a guide for securing Grafana

Note
When using Tempo in multi-tenant mode, Tempo requires the HTTP header X-Scope-OrgID to be set to a string identifying the tenant. It’s assumed that clients setting X-Scope-OrgID are trusted clients, and the responsibility of populating this value should be handled by the authenticating reverse proxy. For more information, read the multi-tenancy documentation.

Manage Tempo on Grafana Labs

Monitor Tempo

Monitor Tempo

Instrumentation

Metrics

Query frontend inspected bytes

Logs

Traces

Polling

Dashboards

Tempo Reads dashboard

Tempo Writes dashboard

Tempo Resources dashboard

Tempo Operational dashboard

Rules and alerts

Dedicated attribute columns

Dedicated attribute columns

Configuration

Usage

Integer attribute selection

Array-valued attributes

Blob attributes

Event-scoped attributes

Tempo-cli

Tune search performance

Tune search performance

Before you begin

Glossary

Tempo query path

Lifecycle of a query

General guidelines

Querier and query-frontend configuration

Querier

Query-frontend

Settings that are safe to increase without major impact.

Parameter interactions

Guidelines on key configuration parameters

query_frontend.max_outstanding_per_tenant parameter

Guidelines

query_frontend.max_batch_size parameter

Guidelines

query_frontend.search.concurrent_jobs parameter

Guidelines

query_frontend.max_retries parameter

Guidelines

query_frontend.search.target_bytes_per_job parameter

Guidelines

querier.frontend_worker.parallelism parameter

Guidelines

querier.max_concurrent_queries parameter

Guidelines

Querier memory sizing

Scaling cache

Improve performance with caching

Improve performance with caching

Memcached

Connection limit

Cache size control

Manage advanced systems

Manage advanced systems

Tempo CLI

Tempo CLI

Tempo CLI command syntax

Run Tempo CLI

Backend options

Query API command

Trace ID

Search

Search tags

Search tag values

Metrics

Query blocks command

Query trace summary command

List blocks

List block

List compaction summary

List cache summary

List index

View index

Generate bloom filter

`query_frontend.max_outstanding_per_tenant` parameter

`query_frontend.max_batch_size` parameter

`query_frontend.search.concurrent_jobs` parameter

`query_frontend.max_retries` parameter

`query_frontend.search.target_bytes_per_job` parameter

`querier.frontend_worker.parallelism` parameter

`querier.max_concurrent_queries` parameter