Set up Tempo on Grafana Labs

Plan your Tempo deployment

Thu, 28 May 2026 17:50:33 +0100

Plan your Tempo deployment

Before you deploy Tempo, you should consider how you plan to use traces and Tempo as well as any special requirements for Tempo 3.0.

Considerations for Tempo 3.0

Before planning your Tempo deployment, be aware of the following infrastructure requirements:

Object storage: Tempo stores trace data in object storage such as S3, GCS, or Azure Storage. Object storage is required for microservices deployments and recommended for production. Monolithic deployments can use local filesystem storage for development and test environments.
Kafka-compatible system: Microservices mode requires a Kafka-compatible system, such as Apache Kafka, Redpanda, or WarpStream, as the durable queue that decouples the write and read paths. Monolithic mode doesn’t use Kafka.

Include these infrastructure components in your deployment planning and resource estimation.

Develop a deployment plan

To plan your Grafana Tempo deployment, you should:

Identify your use case Decide what you need from Tempo. For example, consider if you want a basic installation, horizontal scalability, multi-tenancy, or advanced data storage. Your use case determines the deployment architecture and resource requirements.
Estimate tracing data volume Assess the amount of tracing data your environment will generate. This helps you size your cluster and storage. Refer to the Size your cluster section for guidance on calculating storage and performance needs.
Choose a deployment mode. Tempo supports monolithic and microservices deployment modes.
- Use monolithic mode for getting started, development, or smaller workloads. No Kafka is required.
- Use microservices mode for production, high-volume, or highly available deployments. Requires a Kafka-compatible system.
Review the Deployment modes documentation to compare options and select the best fit for your use case.

By following these recommendations, you can plan a Tempo deployment that matches your operational requirements and scales with your tracing data.

Next steps

Once you’ve planned your deployment, you can Deploy your Tempo instance using the deployment mode you selected.

Deploy Tempo

Thu, 28 May 2026 17:50:33 +0100

Deploy Tempo

Tempo can be easily deployed through a number of tools, including Helm, Tanka, Kubernetes, and Docker.

The following procedures provide example Tempo deployments that you can use as a starting point.

Tempo can be deployed in a number of ways, depending on your needs and environment. You can deploy Tempo in a monolithic mode or in a microservices mode.

You can also use Docker to deploy Tempo using the Docker examples.

Note
Grafana Tempo does not come with any included authentication layer. You must run an authenticating reverse proxy in front of your services to prevent unauthorized access to Tempo (for example, nginx). Manage authentication for more details

Kafka requirement

Tempo 3.0 requires a Kafka-compatible message queue as part of its architecture. Distributors write incoming trace data to Kafka, and downstream components such as block-builders and live-stores consume from it. Any Kafka-compatible system works.

Deploy locally

Monolithic mode (single binary) is commonly used for a local installation, testing, or small-scale deployments. This mode can be deployed using a pre-compiled binary, OS-specific packaging, or Docker image. While it’s possible to deploy monolithic mode in a Kubernetes cluster, it is not recommended for production use.

Check out the Docker examples for more information.

Deploy using Kubernetes

Kubernetes deployment examples:

Deploy with Helm (microservices and monolithic)
Deploy with Tempo Operator (microservices)
Deploy on Kubernetes using Tanka (microservices)

Note
The Tanka and Helm examples are equivalent. They are both provided for people who prefer different configuration mechanisms.

For more information, refer to Deploy Tempo on Kubernetes.

Example setups

Thu, 28 May 2026 17:50:33 +0100

Example setups

The following examples show various deployment and configuration options using trace generators so you can get started experimenting with Tempo without an existing application.

For more information about Tempo setup and configuration, see:

If you are interested in instrumentation, refer to Tempo instrumentation.

Docker Compose

The docker-compose examples are simpler and designed to show minimal configuration.

Some of the examples include:

Trace discovery with Loki
Basic Grafana Alloy/OpenTelemetry Setup
Various Backends (S3/GCS/Azure)
K6 with Traces

This is a great place to get started with Tempo and learn about various trace discovery flows.

Helm

The Helm example shows a complete microservice based deployment. There are monolithic mode and microservices examples.

To install Tempo on Kubernetes, use the Deploy on Kubernetes using Helm procedure.

Tanka

The Jsonnet based examples include complete monolithic mode and microservices deployments.

To learn how to set up a Tempo cluster, refer to Deploy on Kubernetes with Tanka.

Introduction to Metrics, Logs, Traces, and Profiles example

The Introduction to Metrics, Logs, Traces, and Profiles in Grafana provides a self-contained environment for learning about Mimir, Loki, Tempo, Pyroscope, and Grafana. It includes detailed explanations of each component and annotated configurations for each component.

The README.md file explains how to download and start the environment, including instructions for using Grafana Cloud and Grafana Alloy collector.

Validate deployment

Thu, 28 May 2026 17:50:33 +0100

Validate your Tempo deployment

To test your Tempo deployment, select one of the procedures below to test your Tempo deployment:

Validate your local Tempo deployment
Validate your local Grafana Tempo deployment by sending traces and verifying they are stored and queryable.
Validate Kubernetes deployment using a test application
Validate your Tempo deployment on Kubernetes.

Upgrade your Tempo installation

Thu, 28 May 2026 17:50:33 +0100

Upgrade your Tempo installation

You can upgrade a Tempo installation to the next version. However, any release has the potential to have breaking changes. Before promoting an upgrade to production, test in a non-production environment.

The upgrade process changes for each version, depending upon the changes made for the subsequent release.

This upgrade guide applies to self-managed installations and not for Grafana Cloud.

For information about updating to Tempo 2.x, refer to Upgrade to Tempo 2.x in the Tempo 2.10 documentation.

For detailed information about any release, refer to the Release notes.

Tip
You can check your configuration options using the status API endpoint in your Tempo installation.

Upgrade to Tempo 3.0

Tempo 3.0 is a major release that replaces the ingester-based architecture with a new design that separates the read and write paths. Block-builders, live-stores, and a backend scheduler replace ingesters and the compactor. For a detailed description of the new architecture, refer to the Tempo architecture reference.

Warning
Tempo 3.0 requires vParquet4 or later as the block format. If your storage configuration specifies vParquet3 or earlier, upgrade the block format before migrating. Refer to Change the block format version.

The migration path depends on your deployment mode:

Monolithic mode: Update your configuration (remove ingester, ingester_client, compactor, and metrics_generator_client blocks), then upgrade the binary. No Kafka is required. For step-by-step instructions, refer to Migrate a monolithic deployment. For a reference monolithic configuration, refer to Deploy Tempo locally.
Microservices mode: Requires a Kafka-compatible system. Deploy Tempo 3.0 alongside your existing 2.x deployment, switch traffic, then decommission 2.x. Refer to the full Migrate from Tempo 2.x to 3.0 guide.

You can automate configuration migration using the tempo-cli migrate config command, which removes obsolete blocks and adds the required ingest configuration for microservices mode.

When upgrading to Tempo 3.0, also be aware of these breaking changes:

No downgrade path: There is no supported downgrade path from 3.0 to 2.x.
Scalable monolithic mode (SSB) removed: The scalable-single-binary target is no longer available. Use either microservices or monolithic (target: all) instead. Refer to Deployment modes.
Deployment manifests: Update Helm, Tanka, and other deployment manifests to include the new components and Kafka infrastructure.

Legacy overrides disabled by default

Tempo now refuses to start if it detects legacy (flat, unscoped) overrides in the main configuration or the per-tenant overrides file. [PR 6741]

To resolve this, either migrate to the scoped defaults format (recommended) or temporarily opt back in.

Option 1: Migrate to the scoped format

Convert your overrides from the legacy flat format to the scoped defaults format. For example:

Before (legacy):

overrides:
  ingestion_rate_limit_bytes: 20000000
  ingestion_burst_size_bytes: 20000000
  max_bytes_per_trace: 30000000
  max_traces_per_user: 100000

After (scoped):

overrides:
  defaults:
    ingestion:
      rate_limit_bytes: 20000000
      burst_size_bytes: 20000000
      max_traces_per_user: 100000
    global:
      max_bytes_per_trace: 30000000

You can automate the migration using the Tempo CLI. Refer to the tempo-cli migrate overrides-config command.

For the full field mapping between legacy and scoped formats, refer to the Upgrade to Tempo 2.3 section.

Option 2: Temporarily opt back in

Set enable_legacy_overrides: true in the overrides configuration block or pass -config.enable-legacy-overrides=true on the CLI. A deprecation warning is logged on startup and each time per-tenant overrides are loaded. This is a temporary escape hatch. Legacy overrides are removed in a future release.

overrides:
  enable_legacy_overrides: true

`mem-ballast-size-mbs` flag removed

The -mem-ballast-size-mbs command-line flag has been removed. This flag is no longer needed in Go 1.19 and later, which use GOMEMLIMIT instead. [PR 6403]

If your deployment scripts, Helm values, or Tanka/Jsonnet configurations pass -mem-ballast-size-mbs, remove it. Tempo fails to start with an unrecognized flag error.

Metrics-generator configuration changes

The metrics-generator gRPC endpoint and push path have been removed. In Tempo 3.0, the metrics-generator consumes directly from Kafka rather than receiving spans through gRPC from the distributor. [PR 6618]

If your configuration includes a top-level metrics_generator_client block, you can safely remove it. Tempo 3.0 ignores this block, and it is deprecated. It is removed in a future release.

Block configuration centralized to `storage.trace.block`

Block and WAL configuration for the block-builder and live-store is now always sourced from storage.trace.block. Per-module block configuration fields have been removed. [PR 6647]

If your configuration sets block-level options such as version, parquet_dedicated_columns, or parquet_row_group_size_bytes under block_builder.block or live_store.block_config, move them to storage.trace.block.

Before:

block_builder:
  block:
    version: "vParquet5"
    parquet_dedicated_columns:
      - { scope: resource, name: service.name, type: string }

live_store:
  block_config:
    version: "vParquet5"
    parquet_dedicated_columns:
      - { scope: resource, name: service.name, type: string }

After:

storage:
  trace:
    block:
      version: "vParquet5"
      parquet_dedicated_columns:
        - { scope: resource, name: service.name, type: string }

`partition_ring_live_store` removed

Tempo 3.0 removes the top-level partition_ring_live_store setting. Tempo now uses a single partition ring, so this configuration toggle is no longer needed. [PR 6981]

If your 2.x configuration still includes this field, remove it before or during your 3.0 upgrade.

Before:

partition_ring_live_store: true

After:

# Remove partition_ring_live_store

Live-store and query defaults reduced

The default values for several live-store and query-frontend settings have been reduced to produce smaller WAL blocks, release completed blocks sooner, and align the metrics query backend boundary with search.

Setting	Previous default	New default
`live_store.flush_check_period`	`10s`	`5s`
`live_store.max_block_duration`	`30m`	`30s`
`live_store.max_block_bytes`	`100 MiB`	`50 MiB`
`live_store.complete_block_timeout`	`1h`	`20m`
`query_frontend.metrics.query_backend_after`	`30m`	`15m`

If you explicitly set these values in your configuration, no action is needed.

Ingester removal

The ingester module is removed entirely. All ingester-related configuration fields, CLI flags, alerts, and dashboard panels must be removed from your deployment. The write path is now handled by the block-builder and live-store.

Removed configuration sections: ingester, ingester_client, compactor, metrics_generator_client. The ingest.enabled field is also removed, but the ingest block itself is still required for microservices mode (for example, ingest.kafka).

For step-by-step migration instructions, refer to Migrate from Tempo 2.x to 3.0.

(PRs #6959, #6504, #6667, #6873)

Compactor removal and CLI flag changes

The compactor component and the v2 block encoding are removed. Compaction is now handled by the backend scheduler and worker, which track job progress centrally and automatically reschedule failed jobs.

Remove all compactor-related configuration, alerts, and dashboard panels from your deployment. The following tempo-cli commands are also removed because they were specific to the v2 format: list block, list index, view index, gen index, and gen bloom.

The compaction CLI flags drop their duplicate compaction. prefix. Update these flags in your configuration:

compaction.compaction.block-retention → compaction.block-retention
compaction.compaction.max-objects-per-block → compaction.max-objects-per-block
compaction.compaction.max-block-bytes → compaction.max-block-bytes
compaction.compaction.compaction-window → compaction.compaction-window

(PRs #6273, #6369, #6909)

`RetryInfo` enabled by default

The distributor.retry_after_on_resource_exhausted setting now defaults to 5s (previously 0). OTLP clients receive a retry hint on ResourceExhausted errors from the distributor. [PR 7088]

To disable cluster-wide, set the value to 0. To disable for a single tenant, set the per-tenant override ingestion.retry_info_enabled: false.

TraceQL array matching changes

The TraceQL AST optimization changes the semantics of != and !~ operators when used with array attributes. != now means NOT IN (previously CONTAINS NOT EQUAL) and !~ now means MATCH NONE (previously CONTAINS NON-MATCH). Regex operands must be of type string or string array. [PR 6353]

If you have queries that depend on the previous behavior, disable the optimization with the query hint skip_optimization=true.

Other breaking changes

The all target is now 3.0-compatible and the scalable-single-binary target is removed. Refer to Deployment modes. [PR 6283]
The OpenCensus receiver is removed. Migrate to OTLP. [PR 6523]
SpanMetricsSummary is removed and querier code simplified. (PRs #6496, #6510)
The querier.query_live_store configuration is removed. [PR 7048]
query_frontend.search.query_ingesters_until is removed in favor of query_frontend.search.query_backend_after. Refer to the query-frontend component reference. [PR 6507]
The tempo-cli query search command no longer accepts timestamps without a timezone (for example, 2024-01-01T00:00:00). Use RFC3339 format (for example, 2024-01-01T00:00:00Z) or relative time (for example, now-1h). Refer to the Tempo CLI documentation. [PR 6458]
Tempo 3.0 upgrades to Go 1.26.2. [PR 6443]

Migrate from Tempo 2.x to 3.0

Thu, 28 May 2026 17:50:33 +0100

Migrate from Tempo 2.x to 3.0

Grafana Tempo 3.0 introduces a new architecture that replaces ingesters with a Kafka-based ingest path. Distributors write trace data to Kafka, and two new components consume from it: block-builders create blocks for long-term object storage, and live-stores serve recent-data queries. A backend-scheduler and backend-worker replace the compactor for block maintenance.

This guide walks you through migrating a self-managed Grafana Tempo deployment from 2.x to 3.0. The migration path depends on your deployment mode:

Monolithic mode users have a simpler path: update the configuration and upgrade the binary. See Migrate a monolithic deployment.
Microservices mode users follow a parallel-deployment migration: deploy 3.0 alongside 2.x, switch traffic, then decommission. The bulk of this guide covers that path.

Warning
There’s no in-place downgrade from 3.0 to 2.x. During the migration you can route traffic back to 2.x (see Roll back), but once you decommission the 2.x deployment, plan to stay on 3.0.

Note
Running two Tempo deployments in parallel increases infrastructure costs for the duration of the migration. Plan to complete the migration and decommission the 2.x deployment promptly.

Before you begin

Confirm the following before you start:

Your Tempo 2.x deployment uses vParquet4 or later as the block format. Tempo 3.0 doesn’t support vParquet3 or earlier. If you’re using an older format, upgrade your block format before migrating. Refer to Choose a different block format.
Microservices mode only: You have a running Kafka-compatible system (for example, Apache Kafka or Redpanda). Monolithic mode does not require Kafka.
You have access to the same object storage bucket or container used by your 2.x deployment.
If you’re running scalable monolithic mode (SSB), plan to switch to either monolithic or microservices mode. SSB has been removed in Tempo 3.0.
You’ve reviewed the Upgrade your Tempo installation page for breaking changes.

Migrate legacy overrides format

Tempo 3.0 disables the legacy (flat, unscoped) overrides format by default. If your main config or per-tenant overrides file uses the legacy format, Tempo refuses to start with an error like:

DEPRECATED: legacy overrides config format detected but legacy overrides are disabled by default.
Migrate your overrides config to the new scoped format, or set -config.enable-legacy-overrides=true
(or enable_legacy_overrides: true in YAML) to continue using legacy overrides temporarily

As a temporary workaround, set enable_legacy_overrides: true in the overrides block or pass -config.enable-legacy-overrides=true on the CLI. Legacy overrides will be removed in a future release.

To migrate, use the tempo-cli migrate overrides-config command, or manually rewrite your overrides to the new scoped format. Refer to the overrides configuration reference for the full schema.

Architecture changes

In Tempo 2.x, distributors forward spans to ingesters and metrics-generators over gRPC. Ingesters batch spans, build blocks, and flush them to object storage. Compactors maintain blocks in storage. For more information, refer to the Tempo 2.x architecture.

In Tempo 3.0, distributors write spans to Kafka instead. Block-builders consume from Kafka and build blocks for object storage. Live-stores consume from Kafka and serve recent-data queries (typically the last 30 minutes to 1 hour). A backend-scheduler and backend-worker replace the compactor for block compaction and retention. Because Kafka provides durability, Tempo 3.0 operates with a replication factor of 1 (RF1), eliminating the need for ingester replication. Ingesters, the scalable single binary (SSB) mode, and the compactor target are removed.

For a detailed description of the new architecture, refer to Tempo architecture.

Migrate a monolithic deployment

Monolithic deployments don’t require Kafka or any new components beyond the binary upgrade. The distributor pushes spans directly to the in-process metrics-generator (the same as in 2.x), and the live-store handles both serving recent queries and flushing blocks to storage.

To migrate a monolithic deployment:

Migrate your configuration. The simplest way is to use the tempo-cli migrate config command:
Bash
```
tempo-cli migrate config --mode=monolithic old-config.yaml > new-config.yaml
```
Or update the config manually: remove the ingester:, ingester_client:, compactor:, and metrics_generator_client: blocks. If your metrics-generator uses the local_blocks processor, remove it (block building is handled internally by the live-store). All other blocks carry over unchanged.
Upgrade the binary to Tempo 3.0 with the new configuration.
Validate the deployment with Tempo Vulture, or by querying historical traces directly to confirm the new deployment can read from your object storage.

Tempo 3.0 reads both 2.x blocks (RF3) and 3.0 blocks (RF1) automatically, so historical queries work without configuration. TraceQL metrics queries only read RF1 blocks — if your 2.x deployment didn’t use the local-blocks processor, metrics queries only return results for data ingested after the upgrade.

The rest of this guide covers microservices mode.

Prepare for migration

Before deploying Tempo 3.0, configure a Kafka topic and prepare your new configuration.

Configure the Kafka topic

Tempo uses a single Kafka topic for all trace data, configured through ingest.kafka.topic. You can let Tempo auto-create the topic or create it manually. For more information, refer to the Kafka topic configuration reference.

To let Tempo create the topic automatically:

ingest:
  kafka:
    address: <KAFKA_BROKER_ADDRESS>
    topic: <KAFKA_TOPIC_NAME>
    auto_create_topic_enabled: true
    auto_create_topic_default_partitions: <PARTITION_COUNT>

Your Kafka broker must also have auto.create.topics.enable set to true.

Warning
auto_create_topic_default_partitions sets the broker-wide num.partitions default, which affects all auto-created topics on the broker, not just the Tempo topic. If other services share the same Kafka cluster, consider creating the topic manually instead.

To create the topic manually, set the partition count based on your expected parallelism. As a starting point, plan for approximately 10 MB/s of peak ingestion throughput per partition.

Review configuration changes

Tempo 3.0 removes ingester configuration and adds new configuration blocks. The simplest way to migrate your config is to use the tempo-cli migrate config command, which takes your 2.x config and outputs a valid 3.0 config:

tempo-cli migrate config --kafka-address=<KAFKA_BROKER_ADDRESS> --kafka-topic=<KAFKA_TOPIC> old-config.yaml > new-config.yaml

The command removes ingester, ingester_client, compactor, and metrics_generator_client blocks; adds the ingest: block in microservices mode; removes local_blocks processor config; and sets compaction_disabled: true in the defaults overrides for parallel operation. Review the output before deploying.

If you prefer to migrate your config manually:

Remove from your 2.x configuration:

The ingester: block and all its settings
Any ingester_client: configuration
The compactor: block (replaced by backend_scheduler: and backend_worker:)
Any metrics_generator_client: configuration
Any local_blocks processor configuration

Add to your 3.0 configuration:

ingest: block to connect Tempo to Kafka

Your existing server:, distributor:, query_frontend:, storage:, memberlist:, and overrides: blocks carry over largely unchanged.

The following example shows the minimum new configuration to add for microservices mode:

ingest:
  kafka:
    address: <KAFKA_BROKER_ADDRESS>
    topic: <KAFKA_TOPIC>

Each block-builder instance consumes from exactly one Kafka partition based on its ordinal: block-builder-0 consumes partition 0, block-builder-1 consumes partition 1, and so on. This means the number of block-builder replicas must equal the number of Kafka partitions.

To keep block-builder replicas in sync with the partition count, scale them to match the live-store replica count (live-stores also run one replica per partition). You can do this with:

A KEDA autoscaler that mirrors the live-store pod count. The Tempo Jsonnet library includes this configuration.
The Grafana rollout-operator mirroring feature, which keeps one StatefulSet’s replica count in sync with another.
A static replica count set to the number of Kafka partitions if your partition count is fixed.

The live_store: block uses sensible defaults and doesn’t require overrides for most deployments.

If your Kafka cluster requires authentication, add SASL credentials to the ingest block:

ingest:
  kafka:
    address: <KAFKA_BROKER_ADDRESS>
    topic: <KAFKA_TOPIC>
    sasl_username: <USERNAME>
    sasl_password: <PASSWORD>

For the full list of options, refer to the Ingest, Block-builder, and Live-store configuration reference.

Disable compaction during parallel operation

Only one compaction system can safely write to shared storage at a time. Disable compaction in the 3.0 deployment while the 2.x compactors are still running.

Set compaction_disabled: true in the defaults and in every per-tenant override in your 3.0 configuration. Overrides don’t inherit properties, so each tenant entry must include it explicitly:

overrides:
  defaults:
    compaction:
      compaction_disabled: true
  "tenant-a":
    compaction:
      compaction_disabled: true
  # Repeat for all per-tenant overrides

You re-enable compaction by removing this override after the 2.x deployment is decommissioned. Refer to Clean up the Tempo 2.x deployment.

Querying historical data

Tempo 2.x ingesters wrote blocks with replication factor 3 (RF3). Tempo 3.0 block-builders write blocks with replication factor 1 (RF1). After migration, your object storage contains both types. Tempo 3.0 automatically queries the correct blocks based on their replication factor — no additional configuration is needed.

TraceQL and trace-by-ID queries work across your full history, covering both old RF3 blocks from 2.x and new RF1 blocks from 3.0.

TraceQL metrics queries only read RF1 blocks. If your 2.x deployment used the local-blocks processor in the metrics-generator, those RF1 blocks are already in storage and metrics queries cover your full history. If you didn’t use the local-blocks processor, metrics queries only return results for data ingested after the switch to 3.0.

Metrics-generator

In Tempo 3.0, the metrics-generator consumes spans from Kafka in microservices mode. It automatically uses the top-level ingest: block — no additional Kafka configuration is needed for it.

The local-blocks processor has been removed — block building is now handled by the block-builder component. Remove any local_blocks configuration from your metrics-generator settings.

For more information, refer to Metrics-generator.

User-configurable overrides config `metrics_generator.processors` behavior change

In Tempo 2.x, metrics_generator.processors was the only user-configurable override field that OR-merged with the runtime overrides list. In Tempo 3.0, if processors is set in user-configurable overrides, that will override the runtime metrics_generator.processors config.

If you previously relied on runtime override’s merge behavior to force-enable a processor (e.g. local-blocks) for tenants that also use user-configurable overrides, you need to move that processor into each tenant’s user-configurable overrides metrics_generator.processors config.

Setting processors: [] in user-configurable overrides disables all processors for the tenant.

Deploy Tempo 3.0

Deploy a new Tempo 3.0 instance alongside your existing 2.x deployment. Both deployments must point at the same object storage bucket so the new deployment can query historical blocks.

For deployment instructions, refer to Deploy Tempo. For a complete example microservices configuration, refer to the distributed docker-compose example.

To validate the deployment:

Deploy Tempo 3.0 using your preferred method with the updated configuration.
Confirm that compaction is disabled in the 3.0 deployment, as described in Disable compaction during parallel operation.
Verify that all components start successfully and connect to Kafka. Check that tempo_live_store_ready equals 1. The log message "live-store ready to serve queries" confirms the live-store is ready. At this point, no traffic is flowing through the 3.0 distributors yet, so write metrics will be at zero.

Note
On a fresh deployment with no traffic flowing yet, the live-store can take up to live_store.readiness_max_wait (default 30 minutes) to become ready, because there’s no Kafka high-water mark to compare against. This is normal. Once traffic starts flowing in the next step, restarts are near-instant.
Validate the deployment end-to-end using Tempo Vulture, Tempo’s built-in testing tool. Vulture writes traces to the distributor and reads them back through the query frontend, validating the full write and read path. Point it at the 3.0 deployment and let it run for 10–15 minutes. Confirm that tempo_vulture_trace_error_total stays at zero and tempo_vulture_trace_total is increasing.

You can also validate manually by querying the 3.0 deployment directly — for example, using curl against the query frontend API or querying from Grafana:
Bash
```
curl http://<TEMPO_3_QUERY_FRONTEND>:3200/api/traces/<TRACE_ID>
```
Use a trace ID that you know exists in your 2.x deployment. If the query returns trace data, the new deployment is reading from storage correctly.

Before proceeding to the cutover, confirm all of the following:

tempo_live_store_ready equals 1 on every live-store replica.
rate(tempo_block_builder_fetch_errors_total[5m]) is at or near zero on every block-builder.
Vulture has been running for 10–15 minutes with tempo_vulture_trace_error_total at zero and tempo_vulture_trace_total increasing.
At least one trace-by-ID query for a trace that exists in your 2.x deployment returns data when sent to the 3.0 query frontend.

Resolve any issues so all of these conditions are satisfied. Switching traffic with an unhealthy 3.0 deployment risks data loss for newly ingested traces.

Switch traffic to Tempo 3.0

Redirect your trace clients to send data to the Tempo 3.0 deployment. Trace clients include OpenTelemetry Collectors, Grafana Alloy instances, or other exporters configured to send spans to Tempo.

To switch traffic to the 3.0 deployment:

If your routing layer supports gradual traffic shifting, start by sending a small percentage of traffic to the 3.0 deployment and monitor for errors.
Once canary traffic is stable, update your routing to send all traffic to the 3.0 deployment. This is typically a DNS record change, load balancer update, or collector configuration change.
Monitor the 3.0 deployment. Confirm:
- tempo_distributor_kafka_write_bytes_total is increasing steadily.
- tempo_block_builder_fetch_records_total is increasing and tempo_block_builder_fetch_errors_total is at zero.
- tempo_live_store_ready equals 1 and tempo_live_store_records_processed_total is increasing.
- Queries return results for newly ingested traces.
Confirm traffic has drained from the 2.x deployment. Check the 2.x distributor metrics to verify it’s no longer receiving requests.

Before proceeding to cleanup, confirm that the 3.0 deployment is handling all traffic and the 2.x deployment shows zero active requests.

Clean up the Tempo 2.x deployment

After traffic has fully moved to the 3.0 deployment, decommission the old deployment.

To clean up the 2.x deployment:

Wait for the 2.x ingesters to flush all in-memory traces to object storage. With default settings (max_block_duration of 30 minutes and complete_block_timeout of 15 minutes), this takes up to approximately 45 minutes. If you’ve customized these values, add your max_block_duration and complete_block_timeout together to estimate the wait.

To confirm the ingesters are drained, check these metrics on the 2.x deployment:
- tempo_ingester_live_traces should drop to zero (or near zero) — no traces remain in memory.
- tempo_ingester_flush_queue_length should be zero — no pending flushes.
Scale the 2.x compactors to zero and remove the compaction_disabled override from the 3.0 deployment to re-enable compaction. Running without compaction for extended periods can cause the blocklist to grow.
Once you’re confident the 3.0 deployment is stable, scale all 2.x components to zero and decommission the old infrastructure. Don’t delete the shared object storage.
Review your backend-worker replica count. The overlap period may have produced additional blocks that require compaction. Once the backlog clears, reduce backend-workers to your steady-state configuration.

Verify the migration

After completing the migration, confirm the following:

Distributors are writing to Kafka. Verify that tempo_distributor_kafka_write_bytes_total is increasing and tempo_distributor_kafka_write_latency_seconds is within an acceptable range.
Block-builders are consuming from Kafka and building blocks. Verify that tempo_block_builder_fetch_records_total is increasing and tempo_block_builder_fetch_errors_total is at zero.
Live-stores are serving recent-data queries. Verify that tempo_live_store_ready equals 1 and tempo_live_store_records_processed_total is increasing. Confirm tempo_live_store_records_dropped_total is at zero.
Kafka lag is within an acceptable range. Check tempo_ingest_group_partition_lag_seconds.
Historical queries return results from object storage.
Backend-workers are running in the 3.0 deployment and compacting blocks.
If you use the metrics-generator, it’s producing metrics from Kafka-sourced trace data.

Roll back

Rolling back depends on how far the migration has progressed.

Before switching traffic

The 2.x deployment is still active and serving all traffic. No rollback is needed. Stop the 3.0 deployment.

After switching traffic, before decommissioning 2.x

Revert the traffic routing change to point clients back to the 2.x deployment. Because the cutover is a routing change (DNS, load balancer, or collector config), this is near-instant. Both deployments share the same object storage, so historical data remains available. Traces ingested by the 3.0 deployment into Kafka may not be queryable from the 2.x deployment if block-builders haven’t flushed them to storage yet.

After decommissioning 2.x

Rolling back requires redeploying the 2.x infrastructure. Blocks written by the 3.0 block-builders remain in object storage and are readable by both versions, assuming they use a compatible block format (vParquet4 or later).

Troubleshoot the migration

This section covers common issues you might encounter during the migration.

Kafka connection failure

Tempo logs errors such as "the Kafka address has not been configured", "ping kafka; will retry", or "kafka broker not ready after 10 retries".

To resolve this issue:

Verify that ingest.kafka.address in your configuration points to the correct broker address.
Confirm the broker is reachable from the Tempo deployment. Check network connectivity and firewall rules.
If you use SASL authentication, verify that both sasl_username and sasl_password are set. Setting only one produces the error "the SASL username and password must be both configured to enable SASL authentication".

Block-builder not consuming

Tempo logs errors such as "no partitions assigned" or "failed to create kafka reader client".

To resolve this issue:

Verify the Kafka topic exists and contains data.
Check the tempo_block_builder_owned_partitions metric to confirm the block-builder has partition assignments.
Verify the block-builder can reach the Kafka broker.

Live-store not ready

Queries return errors with "live-store is starting" or the live-store logs show "failed to catch up".

The live-store needs time to catch up with Kafka after startup. To monitor progress:

Check the tempo_live_store_ready metric. A value of 1 indicates the live-store is ready to serve queries.
Check tempo_live_store_catch_up_duration_seconds to monitor catch-up progress.
The log message "live-store ready to serve queries" confirms readiness.

Historical queries return no results

Queries for traces that existed in your 2.x deployment return empty results from the 3.0 deployment.

To resolve this issue:

Verify that the 3.0 deployment uses the same object storage configuration (bucket, endpoint, credentials) as the 2.x deployment.
Confirm that the storage backend is reachable from the 3.0 deployment.

Next steps

Refer to Upgrade your Tempo installation for version-specific breaking changes and removed configuration parameters.
Refer to Tempo architecture for details on how block-builders, live-stores, and Kafka interact.
Refer to Configuration for the full configuration reference, including all ingest:, block_builder:, and live_store: parameters.
Refer to Plan your deployment for guidance on deployment modes and sizing in Tempo 3.0.

Command line flags

Thu, 28 May 2026 17:50:33 +0100

Command line flags

Tempo provides various command-line flags to configure its behavior when starting the binary. This document serves as a reference for these flags.

Global flags

Flag	Description	Default
`--version`	Print this build’s version information and exit	`false`
`--mutex-profile-fraction`	Override default mutex profiling fraction	`0`
`--block-profile-threshold`	Override default block profiling threshold	`0`
`--config.file`	Configuration file to load
`--config.expand-env`	Whether to expand environment variables in config file	`false`
`--config.verify`	Verify configuration and exit	`false`

Target flag

The deployment mode is determined by the runtime configuration target, or by using the -target flag on the command line. The default target is all, which runs all components in a single process (monolithic deployment mode).

Flag	Description	Default
`--target`	Target module to run	`all`

Valid target values:

Target	Description
`all`	Monolithic mode. Runs all components in a single process.
`distributor`	Receives and distributes trace data to downstream components.
`metrics-generator`	Generates metrics from ingested trace data.
`querier`	Queries the backend storage for traces and metrics.
`query-frontend`	Provides search API and splits queries for parallelism.
`block-builder`	Consumes data from Kafka and writes blocks to backend storage.
`backend-scheduler`	Schedules and coordinates backend query jobs across workers.
`backend-worker`	Executes query jobs assigned by the backend scheduler.
`live-store`	Serves recently ingested data from Kafka for real-time queries.

Note
In Tempo 3.0, the ingester, compactor, and scalable-single-binary targets were removed as part of the new Tempo architecture.

Refer to the Plan your Tempo deployment documentation for information on deployment modes.

Authentication and multitenancy

Flag	Description	Default
`--multitenancy.enabled`	Set to true to enable multitenancy	`false`
`--auth.enabled`	Deprecated. Use `--multitenancy.enabled` instead. Set to true to enable auth. This flag will be removed in a future release.	`false`

HTTP and API settings

Flag	Description	Default
`--http-api-prefix`	String prefix for all HTTP API endpoints	`""`
`--enable-go-runtime-metrics`	Set to true to enable all Go runtime metrics	`false`
`--shutdown-delay`	How long to wait between SIGTERM and shutdown. After receiving SIGTERM, Tempo reports not-ready status via the `/ready` endpoint.	`0`

Span profiling

Flag	Description	Default
`--span-profiling`	Enable span profiling via `otelpyroscope`. When enabled, Tempo attaches pprof goroutine labels (`span_id`, `span_name`) to OTel spans and adds a `pyroscope.profile.id` attribute to root spans, enabling profile-to-trace correlation in Pyroscope. Requires an OTLP exporter to be configured through environment variables (`OTEL_TRACES_EXPORTER`, `OTEL_EXPORTER_OTLP_ENDPOINT`, or `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`).	`false`

You can also set this option in the configuration file:

span_profiling: true

Health check

Flag	Description	Default
`--health`	Run a health check against the `/ready` endpoint and exit. Returns exit code `0` if healthy, `1` if unhealthy.	`false`
`--health.url`	URL to check when running a health check	`http://localhost:3200/ready`

The Tempo container image uses a distroless base image that doesn’t include a shell, curl, wget, or other utilities. This means the common Docker health check pattern HEALTHCHECK CMD curl -f http://localhost:3200/ready doesn’t work.

The --health flag provides a native alternative. It doesn’t require a Tempo configuration file, so it can be used directly in a HEALTHCHECK instruction:

HEALTHCHECK CMD ["/tempo", "--health"]

Kubernetes users typically don’t need this flag because they can configure httpGet readiness and liveness probes directly against the /ready endpoint.

Logging settings

Flag	Description	Default
`--log.level`	Only log messages with the given severity or above. Valid levels: `debug`, `info`, `warn`, `error`	`info`
`--log.format`	Output format for log messages. Valid formats: `logfmt`, `json`	`logfmt`

Server settings

Flag	Description	Default
`--server.http-listen-port`	HTTP server listen port	`3200`
`--server.grpc-listen-port`	gRPC server listen port	`9095`

Memberlist settings

Flag	Description	Default
`--memberlist.host-port`	Host port to connect to memberlist cluster
`--memberlist.bind-port`	Port for memberlist to communicate on	`7946`
`--memberlist.message-history-buffer-bytes`	Size in bytes for the message history buffer	`0`

MCP server

Flag	Description	Default
`--query-frontend.mcp-server.enabled`	Set to true to enable the MCP server	`false`

Tempo includes an MCP (Model Context Protocol) server that provides AI assistants and Large Language Models (LLMs) with direct access to distributed tracing data through TraceQL queries and other endpoints.

Refer to the Model Context Protocol (MCP) Server documentation for more information.

Module configuration

You can use additional flags to configure individual Tempo modules, such as the distributor, block-builder, live-store, querier, backend-scheduler, backend-worker, and their components. These flags follow a pattern like --<module>.<setting> and are documented in the configuration file format.

Use the configuration file approach described in the Configuration documentation. The documentation has a comprehensive list of all configuration options.

Usage examples

Start Tempo with a configuration file:

tempo --config.file=/etc/tempo/config.yaml

Start Tempo with a specific target:

tempo --target=distributor --config.file=/etc/tempo/config.yaml

Verify configuration without starting Tempo:

tempo --config.file=/etc/tempo/config.yaml --config.verify

Print version information:

tempo --version

Start Tempo with debug-level logging for troubleshooting:

tempo --config.file=/etc/tempo/config.yaml --log.level=debug

Use environment variable expansion in the config file, which lets you inject secrets or environment-specific values at startup:

tempo --config.file=/etc/tempo/config.yaml --config.expand-env

Start the distributor component on a custom HTTP port with JSON-formatted logs, typical for a microservices deployment behind a load balancer:

tempo --target=distributor \
  --config.file=/etc/tempo/config.yaml \
  --server.http-listen-port=3200 \
  --log.format=json

Start Tempo in monolithic mode with multitenancy enabled and a graceful shutdown delay of 30 seconds, allowing in-flight requests to complete during rolling updates:

tempo --config.file=/etc/tempo/config.yaml \
  --multitenancy.enabled \
  --shutdown-delay=30s

Run a health check against a custom URL:

tempo --health --health.url=http://localhost:3200/ready

Set up Tempo on Grafana Labs

Plan your Tempo deployment

Plan your Tempo deployment

Considerations for Tempo 3.0

Develop a deployment plan

Next steps

Deploy Tempo

Deploy Tempo

Kafka requirement

Deploy locally

Deploy using Kubernetes

Example setups

Example setups

Docker Compose

Helm

Tanka

Introduction to Metrics, Logs, Traces, and Profiles example

Validate deployment

Validate your Tempo deployment

Upgrade your Tempo installation

Upgrade your Tempo installation

Upgrade to Tempo 3.0

Legacy overrides disabled by default

Option 1: Migrate to the scoped format

Option 2: Temporarily opt back in

mem-ballast-size-mbs flag removed

Metrics-generator configuration changes

Block configuration centralized to storage.trace.block

partition_ring_live_store removed

Live-store and query defaults reduced

Ingester removal

Compactor removal and CLI flag changes

RetryInfo enabled by default

TraceQL array matching changes

Other breaking changes

Migrate from Tempo 2.x to 3.0

Migrate from Tempo 2.x to 3.0

Before you begin

Migrate legacy overrides format

Architecture changes

Migrate a monolithic deployment

Prepare for migration

Configure the Kafka topic

Review configuration changes

Disable compaction during parallel operation

Querying historical data

Metrics-generator

User-configurable overrides config metrics_generator.processors behavior change

Deploy Tempo 3.0

Switch traffic to Tempo 3.0

Clean up the Tempo 2.x deployment

Verify the migration

Roll back

Before switching traffic

After switching traffic, before decommissioning 2.x

After decommissioning 2.x

Troubleshoot the migration

Kafka connection failure

Block-builder not consuming

Live-store not ready

Historical queries return no results

Next steps

Command line flags

Command line flags

Global flags

Target flag

Authentication and multitenancy

HTTP and API settings

Span profiling

Health check

Logging settings

Server settings

Memberlist settings

MCP server

Module configuration

Usage examples

`mem-ballast-size-mbs` flag removed

Block configuration centralized to `storage.trace.block`

`partition_ring_live_store` removed

`RetryInfo` enabled by default

User-configurable overrides config `metrics_generator.processors` behavior change