Set up Tempo on Grafana Labs

Plan your Tempo deployment

Thu, 09 Apr 2026 14:59:14 +0000

Plan your Tempo deployment

To plan your Grafana Tempo deployment, you should:

Identify your use case Decide what you need from Tempo. For example, consider if you want a basic installation, horizontal scalability, multi-tenancy, or advanced data storage. Your use case determines the deployment architecture and resource requirements.
Estimate tracing data volume Assess the amount of tracing data your environment will generate. This helps you size your cluster and storage. Refer to the Size your cluster section for guidance on calculating storage and performance needs.
Choose a deployment mode. Tempo supports monolithic and microservices deployment modes.
- Use the monolithic mode for simple, single-tenant environments or smaller workloads.
- Use the microservices mode for large-scale, multi-tenant, or highly available deployments. Review the Deployment modes documentation to compare options and select the best fit for your use case.

By following these recommendations, you can plan a Tempo deployment that matches your operational requirements and scales with your tracing data.

Next steps

Once you’ve planned your deployment, you can Deploy your Tempo instance using the deployment mode you selected.

Deploy Tempo

Thu, 09 Apr 2026 14:59:14 +0000

Deploy Tempo

Tempo can be easily deployed through a number of tools, including Helm, Tanka, Kubernetes, and Docker.

The following procedures provide example Tempo deployments that you can use as a starting point.

Tempo can be deployed in a number of ways, depending on your needs and environment. You can deploy Tempo in a monolithic mode or in a microservices mode.

You can also use Docker to deploy Tempo using the Docker examples.

Note
Grafana Tempo does not come with any included authentication layer. You must run an authenticating reverse proxy in front of your services to prevent unauthorized access to Tempo (for example, nginx). Manage authentication for more details

Deploy locally

Monolithic mode (single binary) is commonly used for a local installation, testing, or small-scale deployments. This mode can be deployed using a pre-compiled binary, OS-specific packaging, or Docker image. While it’s possible to deploy monolithic mode in a Kubernetes cluster, it is not recommended for production use.

You can deploy Tempo locally using the monolithic mode. You can use the Docker examples or you can use the Linux example (monolithic) to deploy Tempo on a Linux host.

For more information, refer Deploy Tempo locally.

Deploy using Kubernetes

Kubernetes deployment examples:

Deploy with Helm (microservices and monolithic)
Deploy with Tempo Operator (microservices)
Deploy on Kubernetes using Tanka (microservices)

Note
The Tanka and Helm examples are equivalent. They are both provided for people who prefer different configuration mechanisms.

For more information, refer to Deploy Tempo on Kubernetes.

Example setups

Thu, 09 Apr 2026 14:59:14 +0000

Example setups

The following examples show various deployment and configuration options using trace generators so you can get started experimenting with Tempo without an existing application.

For more information about Tempo setup and configuration, see:

If you are interested in instrumentation, refer to Tempo instrumentation.

Docker Compose

The docker-compose examples are simpler and designed to show minimal configuration.

Some of the examples include:

Trace discovery with Loki
Basic Grafana Alloy/OpenTelemetry Setup
Various Backends (S3/GCS/Azure)
K6 with Traces

This is a great place to get started with Tempo and learn about various trace discovery flows.

Helm

The Helm example shows a complete microservice based deployment. There are monolithic mode and microservices examples.

To install Tempo on Kubernetes, use the Deploy on Kubernetes using Helm procedure.

Tanka

To view an example of a complete microservice-based deployment, this Jsonnet based example shows a complete microservice based deployment. There are monolithic mode and microservices examples.

To learn how to set up a Tempo cluster, refer to Deploy on Kubernetes with Tanka.

Introduction to Metrics, Logs, Traces, and Profiles example

The Introduction to Metrics, Logs, Traces, and Profiles in Grafana provides a self-contained environment for learning about Mimir, Loki, Tempo, Pyroscope, and Grafana. It includes detailed explanations of each component and annotated configurations for each component.

The README.md file explains how to download and start the environment, including instructions for using Grafana Cloud and Grafana Alloy collector.

Validate deployment

Thu, 09 Apr 2026 14:59:14 +0000

Validate your Tempo deployment

To test your Tempo deployment, select one of the procedures below to test your Tempo deployment:

Validate your local Tempo deployment
Instructions for validating your local Tempo deployment.
Validate Kubernetes deployment using a test application
Validate your Tempo deployment on Kubernetes.

Upgrade your Tempo installation

Thu, 09 Apr 2026 14:59:14 +0000

Upgrade your Tempo installation

You can upgrade a Tempo installation to the next version. However, any release has the potential to have breaking changes. We recommend testing in a non-production environment prior to rolling these changes to production.

The upgrade process changes for each version, depending upon the changes made for the subsequent release.

This upgrade guide applies to on-premise installations and not for Grafana Cloud.

For detailed information about any release, refer to the Release notes.

Tip
You can check your configuration options using the status API endpoint in your Tempo installation.

Upgrade to Tempo 2.10

When upgrading to Tempo 2.10, be aware of these considerations and breaking changes.

Busybox removed from Tempo image

The Tempo container image no longer includes busybox. This change reduces the image size and attack surface, preventing future busybox-related vulnerabilities from affecting Tempo deployments. [PR 5717]

The image switched from gcr.io/distroless/static-debian12:debug, which includes busybox, to gcr.io/distroless/static-debian12, which doesn’t. The busybox shell and utilities are no longer available inside the running container.

You can no longer exec into the Tempo container with a shell. Commands like kubectl exec -it <pod> -- sh or docker exec -it <container> sh will fail.

To debug a running Tempo container, use one of these alternatives:

Kubernetes ephemeral debug containers (kubectl debug)
Docker Desktop or other container runtime tools that support shell injection for distroless images

If you have custom Docker Compose files or scripts that use the Tempo image for shell operations (such as running chown in an init container), update them to use a separate busybox:latest image for those tasks.

Tempo’s runtime behavior, configuration options, and APIs are unchanged.

vParquet format changes

This release includes breaking changes to vParquet block format support and deprecates older formats ahead of Tempo 3.0.

In preparation for Tempo 3.0, make sure you’re using vParquet4 or higher.

Enable vParquet5 (optional)

vParquet5 is now production-ready and available as an optional upgrade. vParquet4 remains the default block format. [PR 6219]

To enable vParquet5, set the block version in your storage configuration:

storage:
  trace:
    block:
      version: vParquet5

Once enabled, Tempo writes new blocks in vParquet5 format while continuing to read existing vParquet4 blocks. No migration of existing data is required.

vParquet2 removed

vParquet2 encoding has been completely removed from Tempo 2.10. Tempo can no longer read vParquet2 blocks. [PR 6071]

No action is required if you’ve used default settings. The default block format migrated away from vParquet2 several releases ago.

Action is required if your storage configuration explicitly specifies vParquet2. Before upgrading, verify your configuration doesn’t specify vParquet2. If it does:

Update to a previous Tempo release (2.9 or earlier) configured for vParquet3 or higher
Wait for all existing vParquet2 blocks to expire and be deleted from backend storage
Then upgrade to Tempo 2.10

Upgrading to 2.10 while vParquet2 blocks still exist in storage will cause read errors.

vParquet3 and vParquet2 schemas deprecated

TempoDB schemas vParquet3 and vParquet2 are now deprecated and will be removed in Tempo 3.0. [PR 6198]

If you’re using vParquet3, plan your migration to vParquet4 or higher before upgrading to Tempo 3.0.

Breaking changes

Be aware of these breaking changes when upgrading to Tempo 2.10:

Tenant ID validation: Tempo now validates tenant IDs in the frontend and distributor components, using the same validation rules as Grafana Mimir. Previously, Tempo accepted any tenant ID value. Valid tenant IDs may only contain alphanumeric characters (a-z, A-Z, 0-9) and special characters (!, -, _, ., *, ', (, )), with a maximum length of 150 characters. Empty tenant IDs and the values . or .. are rejected. Requests with invalid tenant IDs return an error. [PR 5786]
SearchTagsV2WithRange API change: The Tempo HTTP client SearchTagsV2WithRange function signature has changed. The function previously accepted only start and end parameters. It now requires two additional parameters: scope and query. This change affects Go code that imports the pkg/httpclient package. To maintain existing behavior, pass empty strings for the new parameters: SearchTagsV2WithRange("", "", start, end). [PR 6088]
Go version upgrade: Tempo 2.10 upgrades to Go 1.25.5, which may affect custom builds or deployments with specific Go version requirements. (PRs #5939 #6001, #6096, #6089)

Upgrade to Tempo 2.9

When upgrading to Tempo 2.9, be aware of these considerations and breaking changes.

RPM and DEB packages discontinued

Tempo no longer publishes RPM and DEB packages due to an internal change to the handling of signing keys. This can be restored if customers need these packages. (PR 5684)

Migrated Vulture and integration tests to OTLP exporter

In this release, we’ve migrated Tempo Vulture and Integration Tests from the deprecated Jaeger agent/exporter to the standard OTLP exporter. Vulture now pushes traces to the Tempo OTLP GRCP endpoint. (PR 5058)

Bucket calculation changes

TraceQL metrics buckets are now calculated based on data in the past instead of the future, which aligns behavior with Prometheus. This resolves issues with empty last buckets that were previously encountered. (PR 5366)

This change may cause differences in existing dashboards and alerts that rely on TraceQL metrics bucket calculations. Review your monitoring and alerting configurations after upgrading.

Series label handling improvements

Fixed incorrect TraceQL metrics results when series labels include both strings and integers with the same textual representation (such as "500" vs 500). The prom_labels field has been removed from TraceQL metrics responses as it was the source of these errors. (PR 5659)

There may be brief interruptions to TraceQL metrics queries during rollout while components run different versions. Plan for a coordinated rollout to minimize impact.

vParquet5 block format

Tempo 2.9 introduces a new experimental vParquet5 block format, designed to improve query performance and reduce storage requirements.vParquet5 has two previews: vParquet5-preview1, low-resolution timestamp columns, and vParquet5-preview2, dedicated integer columns.

Breaking changes are expected before the final release. (PR 5495, PR 5639)

Go version upgrade

Tempo 2.9 upgrades to Go 1.25.0, which may affect custom builds or deployments with specific Go version requirements. (PR 5548)

If you’re building Tempo from source or using custom Docker images, ensure your build environment supports Go 1.25.0.

Upgrade to Tempo 2.8

When upgrading to Tempo 2.8, be aware of these considerations and breaking changes.

Changed the default listening port

With Tempo 2.8, the default http_listen_port changes from 80 to 3200. Check the configuration options for the server: block in your Tempo configuration file.

server:
    # HTTP server listen host
    [http_listen_address: <string>]

    # HTTP server listen port
    [http_listen_port: <int> | default = 3200]

Refer to issue 4945 for more information for the rationale. PR 4960

Removed Tempo serverless

Tempo serverless has been removed. The following configuration options are no longer valid and should be removed from your Tempo configuration. [PR 4599)

querier:
  search:
    prefer_self: <int>
    external_hedge_requests_at: <duration>
    external_hedge_requests_up_to: <duration>
    external_backend: <string>
    google_cloud_run: <string>
    external_endpoints: <array>

In addition, these Tempo serverless related metrics have been removed: tempo_querier_external_endpoint_duration_seconds, tempo_querier_external_endpoint_hedged_roundtrips_total, and tempo_feature_enabled.

Updated, removed, or renamed configuration parameters

Parameter	Comments
`max_span_attr_byte`	Renamed to `max_attribute_bytes`. (PR 4633)
`tempo_discarded_spans_total`	Removed `internal_error` as a reason from `tempo_discarded_spans_total`. (PR 4554)
`tempo_receiver_accepted_span` and `tempo_receiver_refused_spans`	The `name` dimension from `tempo_receiver_accepted_span` and `tempo_receiver_refused_spans` changes from `tempo/jaeger_receiver` to `jaeger/jaeger_receiver`. (PR 4893)

Other upgrade considerations

Upgrade OTEL Collector to v0.122.1. The name dimension from tempo_receiver_accepted_span and tempo_receiver_refused_spans changes from tempo/jaeger_receiver to jaeger/jaeger_receiver. (PR 4893)
Replace opentracing-contrib/go-grpc by otelgrpc in Tempo query. (PR 4958)
Enforce max attribute size at event, link, and instrumentation scope. The configuration is now per-tenant. Renamed max_span_attr_byte to max_attribute_bytes. (PR 4633)
Converted SLO metric query_frontend_bytes_processed_per_second from a histogram to a counter as it’s more performant. (PR 4748)
Removed the OpenTelemetry Jaeger exporter, which has been deprecated. (PR 4926)

Upgrade to Tempo 2.7

When upgrading to Tempo 2.7, be aware of these considerations and breaking changes.

OpenTelemetry Collector receiver listens on `localhost` by default

After this change, the OpenTelemetry Collector receiver defaults to binding on localhost rather than 0.0.0.0. Tempo installations running in Docker or other container environments must update their listener address to continue receiving data. (#4465)

Most Tempo installations use the receivers with the default configuration:

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
        http:

This used to work fine since the receivers defaulted to 0.0.0.0:4317 and 0.0.0.0:4318 respectively. With the changes to replace unspecified addresses, the receivers now default to localhost:4317 and localhost:4318.

As a result, connections to Tempo running in a Docker container won’t work anymore.

To workaround this, you need to specify the address you want to bind to explicitly. For instance, if Tempo is running in a container with hostname tempo, this should work:

# ...
http:
  endpoint: 'tempo:4318'

You can also explicitly bind to 0.0.0.0 still, but this has potential security risks:

# ...
http:
  endpoint: '0.0.0.0:4318'

Maximum spans per span set

A new max_spans_per_span_set limit is enabled by default and set to 100. Set it to 0 to restore the old behavior (unlimited). Otherwise, spans beyond the configured max are dropped. (#4275)

query_frontend:
  search:
      max_spans_per_span_set: 0

Tempo serverless deprecation

Tempo serverless is officially deprecated and will be removed in an upcoming release. Prepare to migrate any serverless workflows to alternative deployments. (#4017)

There are no changes to this release for serverless. However, you’ll need to remove these configurations before the next release.

Anchored regular expressions matchers in TraceQL

Regex matchers in TraceQL are now fully anchored using Prometheus’s fast regexp. For instance, span.foo =~ "bar" is interpreted as span.foo =~ "^bar$". Adjust existing queries accordingly. (#4329)

For more information, refer to the Comparison operators TraceQL documentation.

Migration from OpenTracing to OpenTelemetry

The use_otel_tracer option is removed. Configure your spans via standard OpenTelemetry environment variables. For Jaeger exporting, set OTEL_TRACES_EXPORTER=jaeger.For more information, refer to the OpenTelemetry documentation. (#3646)

gRPC compression set to snappy

Tempo 2.7.1 set gRPC compression between all components to be snappy. Using snappy provides a balanced approach to compression between components that will work for most installations.

If you prefer a different balance of CPU/Memory and bandwidth, consider disabling compression or using zstd.

For a discussion on alternatives, refer to this discussion thread. (#4696).

gRPC compression disabled

Tempo 2.7.0 release disabled gRPC compression in the querier and distributor for performance reasons. (#4429). Our benchmark suggests that without compression, queriers and distributors use less CPU and memory.

Tempo 2.7.1 changed the default value to snappy for internal components.

Note
This change may increase data usage and network traffic, which can impact cloud billing.

If you notice increased network traffic or issues, check the gRPC compression settings.

For instructions how to enable gRPC compression, refer to gRPC compression configuration for more information.

Added, updated, removed, or renamed configuration parameters

Parameter	Comments
`querier_forget_delay`	Removed. The `querier_forget_delay` setting provided no effective functionality and has been dropped. (#3996)
`use_otel_tracer`	Removed. Configure your spans via standard OpenTelemetry environment variables. For Jaeger exporting, set `OTEL_TRACES_EXPORTER=jaeger`. (#3646)
`max_spans_per_span_set`	Added to query-frontend configuration. (#4275)
`use_otel_tracer`	The `use_otel_tracer` option is removed. Configure your spans via standard OpenTelemetry environment variables. For Jaeger exporting, set `OTEL_TRACES_EXPORTER=jaeger`. (#3646)

Other upgrade considerations

The Tempo CLI now targets the /api/v2/traces endpoint by default. Use the --v1 flag if you still rely on the older /api/traces endpoint. (#4127)
If you already set the X-Scope-OrgID header in per-tenant overrides or global Tempo configuration, it is now honored and not overwritten by Tempo. This may change behavior if you previously depended on automatic injection. (#4021)
The AWS Lambda build output changes from main to bootstrap. Follow the AWS migration steps to ensure your Lambda functions continue to work. (#3852)

Upgrade to Tempo 2.6

Tempo 2.6 has several considerations for any upgrade:

Operational change for TraceQL metrics
vParquet4 is now the default block format
Updated, removed, or renamed parameters

For a complete list of changes, refer to the Tempo 2.6 CHANGELOG.

Operational change for TraceQL metrics

We’ve changed to an RF1 (Replication Factor 1) pattern for TraceQL metrics as we were unable to hit performance goals for RF3 deduplication. This requires some operational changes to query TraceQL metrics.

TraceQL metrics are still considered experimental, but we hope to mark them GA soon when we productionize a complete RF1 write-read path. [PRs 3628, 3691, 3723, 3995]

For recent data

The local-blocks processor must be enabled to start using metrics queries like { } | rate(). If not enabled metrics queries fail with the error localblocks processor not found. Enabling the local-blocks processor can be done either per tenant or in all tenants.

Per-tenant in the per-tenant overrides:

overrides:
  'tenantID':
    metrics_generator_processors:
      - local-blocks

By default, for all tenants in the main configuration:

overrides:
  defaults:
    metrics_generator:
      processors: [local-blocks]

Add this configuration to run TraceQL metrics queries against all spans (and not just server spans):

metrics_generator:
  processor:
    local_blocks:
      filter_server_spans: false

For historical data

To run metrics queries on historical data, you must configure the local-blocks processor to flush RF1 blocks to object storage:

metrics_generator:
  processor:
    local_blocks:
      flush_to_storage: true

Transition to vParquet4

vParquet4 format is now the default block format. It’s production ready and we highly recommend switching to it for improved query performance. [PR 3810]

Upgrading to Tempo 2.6 modifies the Parquet block format. You don’t need to do anything with Parquet to go from 2.5 to 2.6. If you used vParquet2 or vParquet3, all of your old blocks remain and can be read by Tempo 2.6. Tempo 2.6 creates vParquet4 blocks by default, which enables the new TraceQL features.

Although you can use Tempo 2.6 with vParquet2 or vParquet3, you can only use vParquet4 with Tempo 2.5 and later. If you are using 2.5 with vParquet4, you’ll need to upgrade to Tempo 2.6 to use the new TraceQL features.

You can also use the tempo-cli analyse blocks command to query vParquet4 blocks. PR 3868] Refer to the Tempo CLIdocumentation for more information.

For information on upgrading, refer to Choose a different block format.

Updated, removed, or renamed configuration parameters

Parameter	Comments
`storage:` `azure:` `use_v2_sdk:`	Removed. Azure v2 is the only and primary Azure backend [PR #3875]
`autocomplete_filtering_enabled`	The feature flag option has been removed. The feature is always enabled. [PR #3729]
`completedfilepath` and `blocksfilepath`	Removed unused WAL configuration options. [PR #3911]
`compaction_disabled`	New. Allow compaction disablement per-tenant. [PR #3965, /configuration/#overrides">documentation]
`Storage:` `s3:` `[enable_dual_stack: <bool>]`	Boolean flag to activate or deactivate dualstack mode on the Storage block configuration for S3. [PR #3721, /configuration/#standard-overrides">documentation]

`tempo-query` is a standalone server

With Tempo 2.6.1, tempo-query is no longer a Jaeger instance with grpcPlugin. It’s now a standalone server. Serving a gRPC API for Jaeger on 0.0.0.0:7777 by default. [PR 3840]

Upgrade to Tempo 2.5

Tempo 2.5 has several considerations for any upgrade:

Docker image runs as a new UID
Support for vParquet format removed
Experimental vParquet4 block format
Removed configuration parameters

For a complete list of changes, enhancements, and bug fixes, refer to the Tempo 2.5 CHANGELOG.

Docker image runs as new UID

The Tempo process in the official Docker image used to run as root. The Tempo process now runs as UID 10001 in the Docker image.

Components such as ingesters and metrics generators that maintain files on disk won’t come up cleanly without intervention. The new user 10001 won’t have access to the old files created by root.

The ownership of /var/tempo changed from root:root to tempo:tempo with the UID/GID of 10001.

The ingester and metrics-generator statefulsets may need to run chown to change ownership to start properly.

Refer to PR 2265 to see a Jsonnet example of an init container.

This change doesn’t impact you if you used the Helm chart with the default security context set in the chart. All data should be owned by the tempo user already. The UID won’t impact Helm chart users.

Support for vParquet format removed

The original vParquet format has been removed from Tempo 2.5. Direct upgrades from Tempo 2.1 to Tempo 2.5 aren’t possible. You will need to upgrade to an intermediate version and wait for the old vParquet blocks to fall out of retention before upgrading to 2.5. PR 3663]

vParquet(1) won’t be recognized as a valid encoding and any remaining vParquet(1) blocks won’t be readable.

Installations running with historical defaults should not require any changes as the default has been migrated for several releases. Installations with storage settings pinned to vParquet must run a previous release configured for vParquet2 or higher until all existing vParquet(1) blocks have expired and been deleted from the backend, or else you’ll encounter read errors after upgrading to this release.

Experimental vParquet4 block format

The vParquet4 block format is required for querying links, events, and arrays and improves query performance relative to previous formats. vParquet4 will be the default block format in the next release. [PR 3368]

While you can use vParquet4, keep in mind that it’s experimental. If you choose to use vParquet4 and then opt to revert to vParquet3, any vParquet4 blocks would not be readable by vParquet3.

To try vParquet4, refer to Choose a block format.

Removed configuration parameters

Parameter	Comments
`[hedge_requests_at: <duration> \| default = 2s ]` `[hedge_requests_up_to: <int> \| default = 2 ]`	Removed options from the configuration. [PR #3522]

Additional considerations

Updating to OTLP 1.3.0 removes the deprecated InstrumentationLibrary and InstrumentationLibrarySpan from the OTLP receivers. PR 3649]
Removes the addition of a tenant in multi-tenant trace id lookup. PR 3522]

Upgrade to Tempo 2.4

Tempo 2.4 has several considerations for any upgrade:

vParquet3 is now the default backend
Caches configuration was refactored
Updated, removed, and renamed configuration parameters

For a complete list of changes, enhancements, and bug fixes, refer to the Tempo 2.4 CHANGELOG.

Transition to vParquet3 as default block format

vParquet3 format is now the default block format. It is production ready and we highly recommend switching to it for improved query performance and dedicated attribute columns.

Upgrading to Tempo 2.4 modifies the Parquet block format. Although you can use Tempo 2.3 with vParquet2 or vParquet3, you can only use Tempo 2.4 with vParquet3.

With this release, the first version of our Parquet backend, vParquet, is being deprecated. Tempo 2.4 still reads vParquet1 blocks. However, Tempo will exit with error if they are manually configured. [PR 3377]

For information on changing the vParquet version, refer to Choose a different block format.

Cache configuration refactored

The major cache refactor lets you configure multiple role-based caches. [PR 3166] This change resulted in several fields being deprecated (refer to the old configuration).

These fields have all been migrated to a top level cache: field.

For more information about the configuration, refer to the Cache section.

The old configuration block looked like this:

storage:
  trace:
    cache:
    search:
      cache_control:
    background_cache:
    memcached:
    redis:

With the new configuration, you create your list of caches, with either redis or memcached cluster with your configuration, and then define the types of data and roles.

Simple configuration example:

cache:
  caches:
    - memcached:
        host: <some memcached cluster>
      roles:
        - bloom
        - parquet-footer
    - memcached:
        host: <some memcached cluster>
      roles:
        - frontend-search

Updated, removed, or renamed configuration parameters

Comments

Parameter
`distributor.log_received_traces`	Use the `distributor.log_received_spans` configuration block instead. [PR #3008]
`tempo_query_frontend_queries_total{op="searchtags\|metrics"}`	Removed deprecated frontend metrics configuration option

The distributor now returns 200 for any batch containing only trace_too_large and max_live_traces errors. The number of discarded spans are still reflected in the tempo_discarded_spans_total metrics.

Upgrade to Tempo 2.3

Tempo 2.3 has several considerations for any upgrade:

vParquet3 is available as a stable, production-read block format
Configuration option to use Azure SDK v2
New defaults block in Overrides module configuration
Several configuration parameters have been renamed or removed.

For a complete list of changes, enhancements, and bug fixes, refer to the Tempo 2.3 CHANGELOG.

Production-ready vParquet3 block format

vParquet3 provides improved query performance and dedicated attribute columns.

This block format is required for using dedicated attribute columns.

While vParquet2 remains the default backend for Tempo 2.3, vParquet3 is available as a stable option. Both work with Tempo 2.3.

Upgrading to Tempo 2.3 doesn’t modify the Parquet block format.

Note
Tempo 2.2 can’t read data stored in vParquet3.

Recommended update process:

Upgrade your Tempo installation to version 2.3, remaining on vParquet2.
Verify the upgrade is stable and performs as expected. If you notice any issues, you can downgrade to version 2.2, and data remains readable.
Change the block format to vParquet3.

If you notice any issues on step 3 using the new block format, you can downgrade to vParquet2. All your data remains readable in Tempo 2.3. However, if you have vParquet3 blocks and have to downgrade to Tempo 2.2, you will have data loss.

Use Azure SDK v2

If you are using Azure storage, we recommend using the v2 SDK, azure-sdk-for-go. You can use the use_v2_sdk configure option for switching.

For more information, refer to the Storage block configuration example documentation.

New `defaults` block in Overrides module configuration

The Overrides module has a new defaults block for configuring global or per-tenant settings. The Overrides format now includes changes to indented syntax. For more information, read the Overrides configuration documentation.

You can also use the Tempo CLI to migrate configurations. Refer to the tempo-cli documentation.

The old configuration block looked like this:

overrides:
  ingestion_rate_strategy: local
  ingestion_rate_limit_bytes: 12345
  ingestion_burst_size_bytes: 67890
  max_search_duration: 17s
  forwarders: ['foo']
  metrics_generator_processors: [service-graphs, span-metrics]

The new configuration block looks like this:

overrides:
  defaults:
    ingestion:
      rate_strategy: local
      rate_limit_bytes: 12345
      burst_size_bytes: 67890
    read:
      max_search_duration: 17s
    forwarders: ['foo']
    metrics_generator:
      processors: [service-graphs, span-metrics]

Removed or renamed configuration parameters

Parameter	Comments
`distributor.log_received_traces`	Use the `distributor.log_received_spans` configuration block instead. [PR 3008]
`tempo_query_frontend_queries_total{op="searchtags\|metrics"}`	Removed deprecated frontend metrics configuration option

Upgrade to Tempo 2.2

Tempo 2.2 has several considerations for any upgrade:

vParquet2 is now the default block format
Several configuration parameters have been renamed or removed.

For a complete list of changes, enhancements, and bug fixes, refer to the Tempo 2.2 CHANGELOG.

Default block format changed to vParquet2

While not a breaking change, upgrading to Tempo 2.2 by default changes Tempo’s block format to vParquet2.

To stay on a previous block format, read the Parquet configuration documentation. We strongly encourage upgrading to vParquet2 as soon as possible as this is required for using structural operators in your TraceQL queries and provides query performance improvements, in particular on queries using the duration intrinsic.

Updated JSonnet supports `statefulset` for the metrics-generator

Tempo 2.2 updates the microservices JSonnet to support a statefulset for the metrics_generator component.

Note
This update is important if you use the experimental local-blocks processor.

To support a new processor, the metrics-generator has been converted from a deployment into a statefulset with a PVC. This requires manual intervention to migrate successfully and avoid downtime. Note that currently both a deployment and a statefulset will be managed by the JSonnet for a period of time, after which we will delete the deployment from this repository and you will need to delete user-side references to the tempo_metrics_generator_deployment, as well as delete the deployment itself.

Refer to the PR for seamless migration instructions. [PRs 2533, 2467]

Removed or renamed configuration parameters

The following fields were removed or renamed.

Parameter	Comments
query_frontend: tolerate_failed_blocks:	Remove support for `tolerant_failed_blocks` [PR 2416]
storage: trace: s3: insecure_skip_verify: true // renamed to tls_insecure_skip_verify	Renamed `insecure_skip_verify` to `tls_insecure_skip_verify` [PR 2407]

Upgrade to Tempo 2.1

Tempo 2.1 has two major considerations for any upgrade:

Support for search on v2 block is removed
Breaking changes to metric names

For more information on other enhancements, read the Tempo 2.1 release notes.

Remove support for Search on v2 blocks

Users can no longer search blocks in v2 format. Only the Parquet formats support search. These search configuration options were removed from the overrides section:

overrides:
  max_search_bytes_per_trace:
  search_tags_allow_list:
  search_tags_deny_list:

The following metrics configuration was also removed:

tempo_ingester_trace_search_bytes_discarded_total

Upgrade path to maintain search from Tempo 1.x to 2.1

Removing support for search on v2 blocks means that if you upgrade directly from 1.9 to 2.1, you will not be able to search your v2 blocks. To avoid this, upgrade to 2.0 first, since 2.0 supports searching both v2 and vParquet blocks. You can let your old v2 blocks gradually age out while Tempo creates new vParquet blocks from incoming traces. Once all of your v2 blocks have been deleted and you only have vParquet format-blocks, you can upgrade to Tempo 2.1. All of your blocks will be searchable.

Parquet files are no longer cached when carrying out searches.

Breaking changes to metric names exposed by Tempo

All Prometheus metrics exposed by Tempo on its /metrics endpoint that were previously prefixed with cortex_ have now been renamed to be prefixed with tempo_ instead. (PR 2204)

Tempo now includes SLO metrics to count where queries are returned within a configurable time range. (PR 2008)

The query_frontend_result_metrics_inspected_bytes metric was removed in favor of query_frontend_bytes_processed_per_second.

Upgrade from Tempo 1.5 to 2.0

Tempo 2.0 marks a major milestone in Tempo development. When planning your upgrade, consider these factors:

Breaking changes:
- Renamed, removed, and moved configurations are described in section below.
- The TempoRequestErrors alert was removed from mixin. Any Jsonnet users relying on this alert should copy this into their own environment.
Advisory:
- Changed defaults. Are these updates relevant for your installation?
- TraceQL editor needs to be enabled in Grafana to use the query editor.
- Resource requirements have changed for Tempo 2.0 with the default configuration.

Once you upgrade to Tempo 2.0, there is no path to downgrade.

Note
There is a potential issue loading experimental Parquet storage blocks. You may see errors or even panics in the compactors. We have only been able to reproduce this with interim commits between 1.5 and 2.0, but if you experience any issues, report them.

Check Tempo installation resource allocation

Parquet provides faster search and is required to enable TraceQL. However, the Tempo installation requires additional CPU and memory resources to use Parquet efficiently. Parquet is more costly due to the extra work of building the columnar blocks, and operators should expect at least 1.5x increase in required resources to run a Tempo 2.0 cluster. Most users find these extra resources are negligible compared to the benefits that come from the additional features of TraceQL and from storing traces in an open format.

You can continue using the previous v2 block format using the instructions provided in the Parquet configuration documentation. Tempo continues to support trace by id lookup on the v2 format for the foreseeable future.

Enable TraceQL in Grafana

TraceQL is enabled by default in Tempo 2.0. The TraceQL query editor requires Grafana 9.3.2 and later.

The TraceQL query editor is in beta in Grafana 9.3.2 and needs to be enabled with the traceqlEditor feature flag.

Check configuration options for removed and renamed options

The following tables describe the parameters that have been removed or renamed.

Removed and replaced

Parameter	Comments
query_frontend: query_shards:	Replaced by `trace_by_id.query_shards`.
querier: query_timeout:	Replaced by two different settings: `search.query_timeout` and `trace_by_id.query_timeout`.
ingester: use_flatbuffer_search:	Removed and automatically determined based on block format.
`search_enabled`	Removed. Now defaults to true.
`metrics_generator_enabled`	Removed. Now defaults to true.

Renamed

The following compactor configuration parameters were renamed.

Parameter	Comments
compaction: chunk_size_bytes:	Renamed to `v2_in_buffer_bytes`
compaction: flush_size_bytes:	Renamed to `v2_out_buffer_bytes`
compaction: iterator_buffer_size:	Renamed to `v2_prefetch_traces_count`

The following storage configuration parameters were renamed.

Parameter	Comments
wal: encoding:	Renamed to `v2_encoding`
block: index_downsample_bytes:	Renamed to `v2_index_downsample_bytes`
block: index_page_size_bytes:	Renamed to `v2_index_page_size_bytes`
block: encoding:	Renamed to `v2_encoding`
block: row_group_size_bytes:	Renamed to `parquet_row_group_size_bytes`

The Azure Storage configuration section now uses snake case with underscores (_) instead of dashes (-). Example of using snake case on Azure Storage config:

# config.yaml
storage:
  trace:
    azure:
      storage_account_name:
      storage_account_key:
      container_name:

Command line flags

Thu, 09 Apr 2026 14:59:14 +0000

Command line flags

Tempo provides various command-line flags to configure its behavior when starting the binary. This document serves as a reference for these flags.

Global flags

Flag	Description	Default
`--version`	Print this build’s version information and exit	`false`
`--mem-ballast-size-mbs`	Size of memory ballast to allocate in MBs	`0`
`--mutex-profile-fraction`	Override default mutex profiling fraction	`0`
`--block-profile-threshold`	Override default block profiling threshold	`0`
`--config.file`	Configuration file to load
`--config.expand-env`	Whether to expand environment variables in config file	`false`
`--config.verify`	Verify configuration and exit	`false`

Target flag

The deployment mode is determined by the runtime configuration target, or by using the -target flag on the command line. The default target is all, which is the monolithic deployment mode.

Flag	Description	Default
`--target`	Target module to run	`all`

Refer to the Plan your Tempo deployment documentation for information on deployment modes.

Authentication and multitenancy

Flag	Description	Default
`--auth.enabled`	Set to true to enable auth (deprecated: use multitenancy.enabled)	`false`
`--multitenancy.enabled`	Set to true to enable multitenancy	`false`

HTTP and API settings

Flag	Description	Default
`--http-api-prefix`	String prefix for all HTTP API endpoints	`""`
`--enable-go-runtime-metrics`	Set to true to enable all Go runtime metrics	`false`
`--shutdown-delay`	How long to wait between SIGTERM and shutdown	`0`

Server settings

Flag	Description	Default
`--server.http-listen-port`	HTTP server listen port	`80`
`--server.grpc-listen-port`	gRPC server listen port	`9095`

Memberlist settings

Flag	Description	Default
`--memberlist.host-port`	Host port to connect to memberlist cluster
`--memberlist.bind-port`	Port for memberlist to communicate on	`7946`
`--memberlist.message-history-buffer-bytes`	Size in bytes for the message history buffer	`0`

Module configuration

You can use additional flags to configuring individual Tempo modules, such as the distributor, ingester, querier, and their components. These flags follow a pattern like --<module>.<setting> and are extensively documented in the configuration file format.

Use the configuration file approach described in the Configuration documentation. The documentation has a comprehensive list of all configuration options.

Usage examples

Start Tempo with a configuration file:

tempo --config.file=/etc/tempo/config.yaml

Start Tempo with a specific target:

tempo --target=distributor --config.file=/etc/tempo/config.yaml

Verify configuration without starting Tempo:

tempo --config.file=/etc/tempo/config.yaml --config.verify

Print version information:

tempo --version

Set up Tempo on Grafana Labs

Plan your Tempo deployment

Plan your Tempo deployment

Next steps

Deploy Tempo

Deploy Tempo

Deploy locally

Deploy using Kubernetes

Example setups

Example setups

Docker Compose

Helm

Tanka

Introduction to Metrics, Logs, Traces, and Profiles example

Validate deployment

Validate your Tempo deployment

Upgrade your Tempo installation

Upgrade your Tempo installation

Upgrade to Tempo 2.10

Busybox removed from Tempo image

vParquet format changes

Enable vParquet5 (optional)

vParquet2 removed

vParquet3 and vParquet2 schemas deprecated

Breaking changes

Upgrade to Tempo 2.9

RPM and DEB packages discontinued

Migrated Vulture and integration tests to OTLP exporter

Bucket calculation changes

Series label handling improvements

vParquet5 block format

Go version upgrade

Upgrade to Tempo 2.8

Changed the default listening port

Removed Tempo serverless

Updated, removed, or renamed configuration parameters

Other upgrade considerations

Upgrade to Tempo 2.7

OpenTelemetry Collector receiver listens on localhost by default

Maximum spans per span set

Tempo serverless deprecation

Anchored regular expressions matchers in TraceQL

Migration from OpenTracing to OpenTelemetry

gRPC compression set to snappy

gRPC compression disabled

Added, updated, removed, or renamed configuration parameters

Other upgrade considerations

Upgrade to Tempo 2.6

Operational change for TraceQL metrics

Transition to vParquet4

Updated, removed, or renamed configuration parameters

tempo-query is a standalone server

Upgrade to Tempo 2.5

Docker image runs as new UID

Support for vParquet format removed

Experimental vParquet4 block format

Removed configuration parameters

Additional considerations

Upgrade to Tempo 2.4

Transition to vParquet3 as default block format

Cache configuration refactored

Updated, removed, or renamed configuration parameters

Upgrade to Tempo 2.3

Production-ready vParquet3 block format

Use Azure SDK v2

New defaults block in Overrides module configuration

Removed or renamed configuration parameters

Upgrade to Tempo 2.2

Default block format changed to vParquet2

Updated JSonnet supports statefulset for the metrics-generator

Removed or renamed configuration parameters

Upgrade to Tempo 2.1

Remove support for Search on v2 blocks

Upgrade path to maintain search from Tempo 1.x to 2.1

Breaking changes to metric names exposed by Tempo

Upgrade from Tempo 1.5 to 2.0

Check Tempo installation resource allocation

Enable TraceQL in Grafana

Check configuration options for removed and renamed options

Removed and replaced

OpenTelemetry Collector receiver listens on `localhost` by default

`tempo-query` is a standalone server

New `defaults` block in Overrides module configuration

Updated JSonnet supports `statefulset` for the metrics-generator