This is documentation for the next version of Grafana Tempo documentation. For the latest stable release, go to the latest version.

Documentation Grafana Tempo Manage Tempo CLI

Tempo CLI

Tempo CLI is a separate executable that contains utility functions related to the Tempo software. Although it’s not required for a working installation, Tempo CLI can be helpful for deeper analysis or for troubleshooting.

Tempo CLI command syntax

The general syntax for commands in Tempo CLI is:

Bash Copy

tempo-cli command [subcommand] [options] [arguments...]

--help or -h displays the help for a command or subcommand.

Example:

Bash Copy

tempo-cli -h
tempo-cli command [subcommand] -h

Run Tempo CLI

Tempo CLI is available as source code and as a Docker image.

Run with Docker (published image)

Bash Copy

docker run --rm grafana/tempo-cli [arguments...]

Run directly with go run

Bash Copy

go run ./cmd/tempo-cli [arguments...]

Build and run a local binary

To build a local binary, you need a working Go installation and build environment.

Bash Copy

make tempo-cli
./bin/$(go env GOOS)/tempo-cli-$(go env GOARCH) [arguments...]

Build and run a local Docker image

Bash Copy

make docker-tempo-cli
docker run --rm tempo-cli [arguments...]

Backend options

Tempo CLI connects directly to the storage backend for some commands, meaning that it requires the ability to read from S3, GCS, Azure or file-system storage. You can configure the backend using the following options:

• Load an existing tempo configuration file using the --config-file (-c) option. This is the recommended option for frequent usage. Refer to Configuration documentation for more information.
• Specify individual settings:
  • --backend <value> The storage backend type, one of s3, gcs, azure, and local.
  • --bucket <value> The bucket name. The meaning of this value is backend-specific. Refer to Configuration documentation for more information.
  • --s3-endpoint <value> The S3 API endpoint (i.e. s3.dualstack.us-east-2.amazonaws.com).
  • --s3-user <value>, --s3-pass <value> The S3 user name and password (or access key and secret key). Optional, as Tempo CLI supports the same authentication mechanisms as Tempo. Refer to S3 permissions documentation for more information.
  • --insecure-skip-verify skip TLS verification, only applies to S3 and GCS.

Each option applies only to the command in which it’s used. For example, --backend <value> doesn’t permanently change where Tempo stores data. It only changes it for command in which you apply the option.

TLS options

All query api commands support these options for HTTPS and gRPC TLS connections. Use an https:// URL for trace-id, or --secure for the other commands.

• --tls-cert <path>, --tls-key <path> PEM client certificate and matching unencrypted private key for mutual TLS (mTLS). Both must be provided together.
• --tls-ca <path> PEM CA bundle used instead of the system trust roots for server certificate verification.
• --tls-server-name <name> Override the server name used for certificate verification and SNI.

The CA and server-name options can also be used without a client certificate.

Query API command

Trace ID

Call the Tempo API and retrieve a trace by ID.

Bash Copy

tempo-cli query api trace-id <api-endpoint> <trace-id>

Arguments:

• api-endpoint URL for the Tempo API.
• trace-id Trace ID as a hexadecimal string.

Options:

• TLS options
• --org-id <value> Organization ID (for use in multi-tenant setup).
• --header <key=value> Extra HTTP header to send with the request. Can be specified multiple times.
• --v1 use v1 API (use /api/traces endpoint to fetch traces, default: /api/v2/traces).

Example:

Bash Copy

tempo-cli query api trace-id http://tempo:3200 f1cfe82a8eef933b

Example with a custom authentication header:

Bash Copy

tempo-cli query api trace-id http://tempo:3200 f1cfe82a8eef933b --header "X-TOKEN=<API_TOKEN>"

Replace <API_TOKEN> with your authentication token.

Search

Call the Tempo API and search using TraceQL.

Bash Copy

tempo-cli query api search <host-port> <trace-ql> [<start> <end>]

Arguments:

• host-port A host/port combination for Tempo. The scheme is inferred from the options.
• trace-ql TraceQL query.
• start Start of the time range to search in RFC3339 format (e.g. 2024-01-01T00:00:00Z) or relative (e.g. now-1h)
• end End of the time range to search in RFC3339 format (e.g. 2024-01-01T01:00:00Z) or relative (e.g. now)

Options:

• TLS options
• --org-id <value> Organization ID (for use in multi-tenant setup).
• --header <key=value> Extra header to send with the request (as gRPC metadata when using --use-grpc). Can be specified multiple times.
• --use-grpc Use GRPC streaming
• --spss <value> Number of spans to return for each spanset
• --limit <value> Number of results to return
• --path-prefix <value> String to prefix search paths with
• --secure Use HTTPS or gRPC with TLS

Example searching for error spans using relative time:

Bash Copy

tempo-cli query api search localhost:3200 '{status = error}' now-1h now

Example searching for error spans using absolute time:

Bash Copy

tempo-cli query api search localhost:3200 '{status = error}' 2024-01-01T00:00:00Z 2024-01-01T01:00:00Z

Example using GRPC streaming with organization ID:

Bash Copy

tempo-cli query api search --use-grpc --org-id my-org localhost:3200 '{span.http.status_code >= 400}' now-1h now

Example with a custom authentication header over GRPC:

Bash Copy

tempo-cli query api search --use-grpc --header "X-TOKEN=<API_TOKEN>" localhost:9095 '{status = error}' now-1h now

Search tags

Call the Tempo API and search attribute names.

Bash Copy

tempo-cli query api search-tags <host-port> [<start> <end>]

Arguments:

• host-port A host/port combination for Tempo. The scheme will be inferred based on the options provided.
• start Start of the time range to search in RFC3339 format (e.g. 2024-01-01T00:00:00Z) or relative (e.g. now-1h)
• end End of the time range to search in RFC3339 format (e.g. 2024-01-01T01:00:00Z) or relative (e.g. now)

Options:

• TLS options
• --org-id <value> Organization ID (for use in multi-tenant setup).
• --header <key=value> Extra header to send with the request (as gRPC metadata when using --use-grpc). Can be specified multiple times.
• --use-grpc Use GRPC streaming
• --path-prefix <value> String to prefix search paths with
• --secure Use HTTPS or gRPC with TLS

Example:

Bash Copy

tempo-cli query api search-tags localhost:3200

Example with relative time range:

Bash Copy

tempo-cli query api search-tags localhost:3200 now-1h now

Example with absolute time range:

Bash Copy

tempo-cli query api search-tags localhost:3200 2024-01-01T00:00:00Z 2024-01-02T00:00:00Z

Search tag values

Call the Tempo API and search attribute values.

Bash Copy

tempo-cli query api search-tag-values <host-port> <tag> [<start> <end>]

Arguments:

• host-port A host/port combination for Tempo. The scheme is inferred from the options.
• tag The fully qualified TraceQL tag to search for. For example, resource.service.name.
• start Start of the time range to search in RFC3339 format (e.g. 2024-01-01T00:00:00Z) or relative (e.g. now-1h)
• end End of the time range to search in RFC3339 format (e.g. 2024-01-01T01:00:00Z) or relative (e.g. now)

Options:

• TLS options
• --org-id <value> Organization ID (for use in multi-tenant setup).
• --header <key=value> Extra header to send with the request (as gRPC metadata when using --use-grpc). Can be specified multiple times.
• --query <value> TraceQL query to filter attribute results by.
• --use-grpc Use GRPC streaming
• --path-prefix <value> String to prefix search paths with
• --secure Use HTTP or gRPC with TLS

Example to find all service names:

Bash Copy

tempo-cli query api search-tag-values localhost:3200 resource.service.name

Example with query filter to find service names that have errors:

Bash Copy

tempo-cli query api search-tag-values --query '{status = error}' localhost:3200 resource.service.name

Metrics

Call the Tempo API and generate metrics from traces using TraceQL.

Bash Copy

tempo-cli query api metrics <host-port> <trace-ql metrics query> [<start> <end>]

Arguments:

• host-port A host/port combination for Tempo. The scheme will be inferred based on the options provided.
• trace-ql metrics query TraceQL metrics query.
• start Start of the time range to search in RFC3339 format (e.g. 2024-01-01T00:00:00Z) or relative (e.g. now-1h)
• end End of the time range to search in RFC3339 format (e.g. 2024-01-01T01:00:00Z) or relative (e.g. now)

Options:

• TLS options
• --org-id <value> Organization ID (for use in multi-tenant setup).
• --header <key=value> Extra header to send with the request (as gRPC metadata when using --use-grpc). Can be specified multiple times.
• --use-grpc Use GRPC streaming
• --instant Perform an instant query instead of a range query.
• --path-prefix <value> String to prefix search paths with
• --secure Use HTTPS or gRPC with TLS

Example range query for request rates by service:

Bash Copy

tempo-cli query api metrics localhost:3200 '{} | rate() by (resource.service.name)' now-1h now

Example instant query for current error rates:

Bash Copy

tempo-cli query api metrics --instant localhost:3200 '{status = error} | rate()' now-1h now

Query trace-id command

Iterate over all backend blocks and dump all data found for a given trace ID.

Bash Copy

tempo-cli query trace-id <trace-id> <tenant-id>

Arguments:

• trace-id Trace ID as a hexadecimal string.
• tenant-id Tenant to search.

Options:

• Backend options
• --percentage <value> Percentage of blocks to scan (for example, 0.1 for 10%). Useful for sampling large datasets.

Example:

Bash Copy

tempo-cli query trace-id f1cfe82a8eef933b single-tenant

Example scanning only 10% of blocks:

Bash Copy

tempo-cli query trace-id --percentage 0.1 f1cfe82a8eef933b single-tenant

Query trace summary command

Iterate over all backend blocks and dump a summary for a given trace ID.

The summary includes:

• number of blocks the trace is found in
• span count
• trace size
• trace duration
• root service name
• root span info
• top frequent service names

Bash Copy

tempo-cli query trace-summary <trace-id> <tenant-id>

Arguments:

• trace-id Trace ID as a hexadecimal string.
• tenant-id Tenant to search.

Options:

• Backend options
• --percentage <value> Percentage of blocks to scan (for example, 0.1 for 10%). Useful for sampling large datasets.

Example:

Bash Copy

tempo-cli query trace-summary f1cfe82a8eef933b single-tenant

List blocks

Lists information about all blocks for the given tenant, and optionally perform integrity checks on indexes for duplicate records.

Bash Copy

tempo-cli list blocks <tenant-id>

Arguments:

• tenant-id The tenant ID. Use single-tenant for single tenant setups.

Options:

• --include-compacted Include blocks that have been compacted. Default behavior is to display only active blocks.

Output: Explanation of output:

• ID Block ID.
• Lvl Compaction level of the block.
• Objects Number of objects stored in the block.
• Size Data size of the block after any compression.
• Vers Block version.
• Window The window of time that was considered for compaction purposes.
• Start The earliest timestamp stored in the block.
• End The latest timestamp stored in the block.
• Duration Duration between the start and end time.
• Age The age of the block.
• Cmp Whether the block has been compacted (present when –include-compacted is specified).

Example:

Bash Copy

tempo-cli list blocks -c ./tempo.yaml single-tenant

List compaction summary

Summarizes information about all blocks for the given tenant based on compaction level. This command is useful to analyze or troubleshoot compaction behavior.

Bash Copy

tempo-cli list compaction-summary <tenant-id>

Arguments:

• tenant-id The tenant ID. Use single-tenant for single tenant setups.

Example:

Bash Copy

tempo-cli list compaction-summary -c ./tempo.yaml single-tenant

List cache summary

Prints information about the number of bloom filter shards per day per compaction level. This command is useful to estimate and fine-tune cache storage. Read the caching topic for more information.

Bash Copy

tempo-cli list cache-summary <tenant-id>

Arguments:

• tenant-id The tenant ID. Use single-tenant for single tenant setups.

Example:

Bash Copy

tempo-cli list cache-summary -c ./tempo.yaml single-tenant

List column

Lists values in a given column of a block. Useful for inspecting parquet block data directly.

Bash Copy

tempo-cli list column <tenant-id> <block-id> [column-name]

Arguments:

• tenant-id The tenant ID. Use single-tenant for single tenant setups.
• block-id The block ID as UUID string.
• column-name Column name to list values of (default: TraceID).

Options:

• Backend options

Example:

Bash Copy

tempo-cli list column -c ./tempo.yaml single-tenant ca314fba-efec-4852-ba3f-8d2b0bbf69f1 TraceID

View schema

View block metadata, parquet schema structure, and column sizes for a given block.

Bash Copy

tempo-cli view schema <tenant-id> <block-id>

Arguments:

• tenant-id The tenant ID. Use single-tenant for single tenant setups.
• block-id The block ID as UUID string.

Options:

• Backend options

The output includes:

• Block metadata
• Parquet schema structure
• Column sizes in KB

Example:

Bash Copy

tempo-cli view schema -c ./tempo.yaml single-tenant ca314fba-efec-4852-ba3f-8d2b0bbf69f1

Benchmark profile

Profile a local block for read-path benchmarking. Writes a JSON file recording what had to be measured from the block — its metadata, its row-group count, and present and absent trace IDs to look up — so that a benchmark run does not have to inspect the block, and every variant of an experiment works from the same measurements.

Bash Copy

tempo-cli benchmark profile <block-path>

Arguments:

• block-path Path to the block directory on local disk, laid out as <bucket>/<tenant-id>/<block-id>.

Options:

• --trace-ids Number of present trace IDs to sample, or all to enumerate every ID at run time rather than embedding them. Defaults to 10000. One absent ID is derived per present ID. Pass 0 to skip trace IDs, and with them the full scan they require.
• -o, --out File to write the profile to. Defaults to stdout.

Profiles built from a customer block embed real trace IDs. Treat them as local artifacts.

Example:

Bash Copy

tempo-cli benchmark profile /data/traces/single-tenant/ca314fba-efec-4852-ba3f-8d2b0bbf69f1 --trace-ids=10000 -o profile.json

Benchmark run

Run read-path benchmark queries against a local block and write the measurements as JSON. Takes a profile from benchmark profile, so the run does not inspect the block and every variant of an experiment measures the same queries.

Bash Copy

tempo-cli benchmark run <block-path> -p <profile.json>

Arguments:

• block-path Path to the block directory on local disk, laid out as <bucket>/<tenant-id>/<block-id>.

Options:

• -p, --profile Profile of the block, from benchmark profile. Required.
• -o, --out File to write the result to. Defaults to stdout.
• --repeat Passes over the query set. Defaults to 1.
• --warmup Passes to run and discard first. Defaults to 1, which pays the block’s cold-read cost outside the measurement. Setting it to 0 measures the first case cold and every later one warm.
• --target-bytes-per-request Bytes per search shard, mirroring the query frontend option of the same name. Defaults to 100MiB.
• --search-limit Traces a search returns per shard. Defaults to 20.
• --max-series Series a metrics query returns. Defaults to 1000.
• --exemplars Exemplars a metrics query collects. Defaults to 0.
• --read-buffer-size Storage read buffer size, such as 8MiB. Defaults to 0, meaning Tempo’s default. This is the knob an experiment varies.
• --backend-latency, --backend-bandwidth Simulate an object store: every backend request waits the latency plus its size over the bandwidth, a size per second such as 100MiB. Each defaults to 0, which reads the local block as is. Without them, a local read costs almost nothing, so an option that trades request count for bytes read, like --read-buffer-size, only shows up in backend.* and not in latency.

Each case records how many results it matched, and one metrics map. Two runs are only comparable if the match counts agree, so a difference there means the comparison is invalid rather than interesting.

Every measurement has the same shape whatever its source, so nothing reading a result needs a rule per source:

• total is the sum over the case for a counter, or the value left behind for a gauge.
• summary describes the per-execution values, with quantiles so a box plot needs nothing else. A total on its own hides the tail, which on the read path is usually the interesting part.

Keys are source.name:

• harness.* is what the benchmark timed itself: wallNs, cpuNs, allocBytes, allocCount.
• backend.* is object-store traffic: reads, bytes, timeNs.
• response.* is what a query API reported, under Tempo’s own metric names. A metric Tempo did not report is absent rather than zero, because the two are different claims, so a summary’s count says how many executions reported it.
• process.* is what Tempo emitted to its Prometheus registry, including the Go runtime and process collectors.

Metrics are collected rather than listed, so a metric added to Tempo appears without a change to the benchmark. Only metrics that moved are kept.

The query set covers trace lookups by ID, present and absent; an unfiltered search; rate() and rate() by (resource.service.name) as metrics range queries; and tag-name lookups in each attribute scope. Metrics queries run over the block’s whole time range, stepping at max(60s, window/30) to land about 30 points. Searches and metrics queries are split into shards of row groups, mirroring how the query frontend splits a job.

A case that fails is recorded with its error and the rest of the run continues. Benchmarking trace lookups reads the block’s bloom filters, so a partial block copy without them can still be profiled but only its search cases will run.

Example:

Bash Copy

tempo-cli benchmark profile /data/traces/single-tenant/ca314fba-efec-4852-ba3f-8d2b0bbf69f1 -o profile.json
tempo-cli benchmark run /data/traces/single-tenant/ca314fba-efec-4852-ba3f-8d2b0bbf69f1 -p profile.json -o result.json

Benchmark compare

Compare two or more results from benchmark run, and write the comparison as markdown to stdout, to read or paste into a pull request, or serve it as web pages to explore. For each metric it lays every case out as benchstat does: the baseline’s value, then each other run’s value and its change from the baseline.

Bash Copy

tempo-cli benchmark compare <result.json> <result.json>...

A change of 10% or more is in bold. A run that cannot be compared with the baseline on a case reads not comparable, with the reason under the table: its match or execution count per pass differs, or it is missing the case or failed it. When a run’s name is too long to head a column, runs are numbered, and listed with their numbers.

With --http, it serves the comparison as web pages instead, rendered on each request as pprof’s web view is. The summary page has a table per metric, with links to switch the metric and the percentile. Each case has a page with the box plots and table of every metric at once, and hovering a box plot’s row shows its numbers. A box spans the 25th to 75th percentile, with a mark at the median. Its whisker runs from the minimum to the 99th percentile, with a tick at the 90th. The axis stops near the highest 99th percentile, so a maximum far past it is marked at the edge and written out, rather than squashing every box to make room for it. Clicking a run on either page makes it the baseline. As with pprof, an address without a host, like :8080, is served on localhost only.

Arguments:

• results Results to compare, as name=path or a path. The first is the baseline.

Options:

• -m, --metric Metrics to show, as glob patterns over the metric keys. Defaults to harness.wallNs, harness.cpuNs, harness.allocBytes, backend.bytes, and backend.reads.
• -k, --case Cases to show, as glob patterns over the case IDs, for example traceid/*. Defaults to every case.
• --percentile Percentile the summaries show: min, p25, p50, p75, p90, p99, or max. Defaults to p99, since tail latency is what hurts most.
• --http Serve the comparison as web pages on this address, like :8080, instead of writing markdown.

A result given as a path is named after the settings that set it apart from the others: the run options and git SHA that differ between the runs, or, when those are all the same, the Go version, GOMAXPROCS, or host. When one setting differs, the name is its value, like 4MiB. When several do, the name lists each, like targetBytesPerRequest=2MiB readBufferSize=4MiB. Runs set up alike, such as repeats of one setup, are named after their files instead. When the one setting that differs is a number, the runs are put in its order, with those left at Tempo’s default first, so reading across the columns follows the setting as it grows. The baseline stays the run given first wherever it lands.

Above the tables, each run has a line saying how it differs from the baseline, like readBufferSize default → 4MiB. The baseline’s line shows what the others are measured from: its value of every setting that differs, then its git SHA, Go version, GOMAXPROCS, and host. A difference in where a run happened, its Go version, GOMAXPROCS, or host, is flagged with ⚠, since latencies from two environments are hard to compare.

The summaries are per execution: one trace lookup, or one shard of a search, metrics, or tag-name query. The spread of a box is across those executions, not across repeated runs, so it describes how the inputs differ, not how noisy the measurement is.

Example, where the runs are named default, 4MiB, and 16MiB from their read buffer sizes:

Bash Copy

tempo-cli benchmark compare main.json read-buffer-4mib.json read-buffer-16mib.json -k 'traceid/*' > comparison.md
tempo-cli benchmark compare main.json read-buffer-4mib.json read-buffer-16mib.json --http=:8080

Query search command

Search blocks in a given time range for a specific key/value pair.

Bash Copy

tempo-cli query search <name> <value> <start> <end> <tenant-id>

Arguments:

• name Name of the attribute to search for, for example, http.method.
• value Value of the attribute to search for, for example, GET.
• start Start of the time range to search in RFC3339 format (e.g. 2024-01-01T00:00:00Z) or relative (e.g. now-1h)
• end End of the time range to search in RFC3339 format (e.g. 2024-01-01T01:00:00Z) or relative (e.g. now)
• tenant-id Tenant to search.

Options:

• Backend options

Example using relative time:

Bash Copy

tempo-cli query search http.method GET now-1h now single-tenant --backend=gcs --bucket=tempo-trace-data

Example using absolute time:

Bash Copy

tempo-cli query search http.method GET 2024-01-01T00:00:00Z 2024-01-01T00:05:00Z single-tenant --backend=gcs --bucket=tempo-trace-data

Parquet convert A to B command

Converts a vParquet file (actual data.parquet) of format A to a block of newer format B with an optional list of dedicated attribute columns. Actual supported versions for A and B vary by Tempo release. This utility command is useful when testing the impact of different combinations of dedicated columns.

Convert vParquet3 to vParquet4

Bash Copy

tempo-cli parquet convert-3to4 <in file> [<out path>] [<list of dedicated columns>]

Arguments:

• in file Path to an existing vParquet3 block directory.
• out path Path to write the vParquet4 block to. The default is ./out.
• list of dedicated columns Optional list of columns to make dedicated. Columns use TraceQL syntax with scope. For example, span.db.statement, resource.namespace.

Example:

Bash Copy

tempo-cli parquet convert-3to4 ./block-in ./out span.db.statement span.db.name

Convert vParquet4 to vParquet5

Converts a vParquet4 block to vParquet5 format with an optional list of dedicated attribute columns.

Bash Copy

tempo-cli parquet convert-4to5 <in file> [<out path>] [<list of dedicated columns>]

Arguments:

• in file Path to an existing vParquet4 block directory.
• out path Path to write the vParquet5 block to. The default is ./out.
• list of dedicated columns Optional list of columns to make dedicated. Columns use TraceQL syntax with scope. For example, span.http.method, resource.namespace, event.exception.message.

Column prefixes:

• int/ marks the column as an integer type. For example, int/span.http.status_code.
• blob/ marks the column for blob encoding. For example, blob/span.db.statement.

Example:

Bash Copy

tempo-cli parquet convert-4to5 ./block-in ./block-out "span.http.method" "int/span.http.status_code" "blob/span.db.statement"

Migrate tenant command

Copy blocks from one backend and tenant to another. Blocks can be copied within the same backend or between two different backends. The data format isn’t converted but the tenant ID in meta.json is rewritten.

Bash Copy

tempo-cli migrate tenant <source tenant> <dest tenant>

Arguments:

• source tenant Tenant to copy blocks from
• dest tenant Tenant to copy blocks into

Options:

• --source-config-file <value> (required) Configuration file for the source backend.
• --config-file <value> Configuration file for the destination backend.

Example:

Bash Copy

tempo-cli migrate tenant --source-config-file source.yaml --config-file dest.yaml my-tenant my-other-tenant

Migrate overrides config command

Migrate the overrides section of a full Tempo config file from the legacy flat format to the new scoped format. The command reads the full config, converts any legacy overrides, and outputs only user-set values.

Bash Copy

tempo-cli migrate overrides-config <config-file>

Arguments:

• config-file Path to the full Tempo config file.

Options:

• -d, --config-dest <path> Path to write the migrated overrides section. If not specified, output is printed to stdout.

Example:

Bash Copy

tempo-cli migrate overrides-config config.yaml -d migrated-overrides.yaml

Migrate overrides per-tenant command

Migrate a per-tenant overrides file from the legacy flat format to the new scoped format. The command handles both legacy and new format entries, and outputs only tenant-specific values.

Bash Copy

tempo-cli migrate overrides-per-tenant <overrides-file>

Arguments:

• overrides-file Path to the per-tenant overrides file.

Options:

• -d, --output-dest <path> Path to write the migrated per-tenant overrides. If not specified, output is printed to stdout.

Example:

Bash Copy

tempo-cli migrate overrides-per-tenant overrides.yaml -d migrated-overrides.yaml

Migrate config command

Migrate a Tempo 2.x configuration file to a valid 3.0 configuration. The command removes obsolete configuration sections (such as ingester, ingester_client, and compactor), adds Kafka ingest configuration for microservices mode, disables compaction in overrides for parallel operation during migration, and strips the removed local-blocks metrics-generator processor.

The tool works at the YAML map level rather than rewriting the file from fully decoded Tempo structs, so environment variable references like ${VAR} are preserved. Top-level sections that are not recognized by the Tempo 3.0 configuration are dropped from the output. Unknown nested keys normally cause validation to fail, but validation is best-effort when the configuration contains ${VAR} placeholders where non-string types are expected, which may let unknown nested keys through.

Bash Copy

tempo-cli migrate config [options] <config-file>

Arguments:

• config-file Path to the 2.x Tempo configuration file.

Options:

• --kafka-address <address> Kafka broker address. Required when running in microservices mode.
• --kafka-topic <topic> Kafka topic name. Defaults to tempo.
• --mode <monolithic|microservices> Override automatic deployment mode detection. By default, the mode is detected from the target field (all or absent means monolithic, any other value means microservices).

The migrated configuration is printed to stdout. Warnings are printed to stderr.

Examples

Monolithic mode (no Kafka flags needed):

Bash Copy

tempo-cli migrate config old-config.yaml > new-config.yaml

Microservices mode:

Bash Copy

tempo-cli migrate config --kafka-address=kafka:9092 --kafka-topic=tempo-traces old-config.yaml > new-config.yaml

Analyse block

Analyses a block and outputs a summary of the block’s generic attributes.

It’s of particular use when trying to determine candidates for dedicated attribute columns in vParquet3+. The output includes span, resource, and event attributes with cardinality and size information.

Arguments:

• tenant-id The tenant ID. Use single-tenant for single tenant setups.
• block-id The block ID as UUID string.

Options:

• Backend options
• --num-attr <value> Number of attributes to output (default: 20)
• --num-int-attr <value> Number of integer attributes to display. If set to 0, uses the --num-attr value (default: 5)
• --blob-threshold <value> Mark attributes as blob candidates when their dictionary size per row group exceeds this value. Set to 0 to disable. (default: 4MiB)
• --include-well-known Include well-known attributes in the analysis. Enable when generating dedicated columns for vParquet5 or higher. (default: false)
• --generate-jsonnet Generate Jsonnet overrides for dedicated columns
• --generate-cli-args Generate command-line arguments for the parquet conversion command
• --int-percent-threshold <value> Threshold for integer attributes in dedicated columns (default: 0.05)
• --str-percent-threshold <value> Threshold for string attributes in dedicated columns (default: 0.03)
• --simple-summary Print only a single line of top attributes (default: false)
• --print-full-summary Print full summary of the analysed block (default: true)

Example:

Bash Copy

tempo-cli analyse block --backend=local --bucket=./cmd/tempo-cli/test-data/ single-tenant b18beca6-4d7f-4464-9f72-f343e688a4a0

Example with blob detection:

Bash Copy

tempo-cli analyse block --blob-threshold=4MiB --generate-jsonnet --backend=local --bucket=./cmd/tempo-cli/test-data/ single-tenant b18beca6-4d7f-4464-9f72-f343e688a4a0

Analyse blocks

Analyses all blocks in a given time range and outputs a summary of the blocks’ generic attributes.

It’s of particular use when trying to determine candidates for dedicated attribute columns in vParquet3+. The output includes span, resource, and event attributes with cardinality and size information.

Arguments:

• tenant-id The tenant ID. Use single-tenant for single-tenant setups.

Options:

• Backend options
• --num-attr <value> Number of attributes to output (default: 20)
• --num-int-attr <value> Number of integer attributes to display. If set to 0, uses the --num-attr value (default: 5)
• --min-compaction-level <value> Minimum compaction level to include in the analysis (default: 3)
• --max-blocks <value> Maximum number of blocks to analyze (default: 10)
• --max-start-time <value> Oldest start time for a block to be processed. RFC3339 format (default: disabled)
• --min-start-time <value> Newest start time for a block to be processed. RFC3339 format (default: disabled)
• --blob-threshold <value> Mark attributes as blob candidates when their dictionary size per row group exceeds this value. Set to 0 to disable. (default: 4MiB)
• --include-well-known Include well-known attributes in the analysis. (default: false)
• --jsonnet Generate Jsonnet overrides for dedicated columns
• --cli Generate command-line arguments for the parquet conversion command
• --int-percent-threshold <value> Threshold for integer attributes in dedicated columns (default: 0.05)
• --str-percent-threshold <value> Threshold for string attributes in dedicated columns (default: 0.03)
• --simple-summary Print only a single line of top attributes (default: false)
• --print-full-summary Print full summary of the analysed block (default: true)

Example:

Bash Copy

tempo-cli analyse blocks --backend=local --bucket=./cmd/tempo-cli/test-data/ single-tenant

Example with blob detection and Jsonnet output:

Bash Copy

tempo-cli analyse blocks --blob-threshold=4MiB --jsonnet --backend=local --bucket=./cmd/tempo-cli/test-data/ single-tenant

Suggest columns

Suggests dedicated columns for a tenant based on analysis of blocks. This command analyzes block data and outputs configuration recommendations in YAML or Jsonnet format that can be used to configure dedicated attribute columns.

Bash Copy

tempo-cli suggest columns <tenant-id>

Arguments:

• tenant-id The tenant ID. Use single-tenant for single tenant setups.

Options:

• Backend options
• --block-id <value> Specific block ID to analyse. If not provided, analyzes multiple blocks.
• --min-compaction-level <value> Minimum compaction level to analyse (default: 3)
• --max-blocks <value> Maximum number of blocks to analyse (default: 10)
• --num-attr <value> Number of attributes to display (default: 20)
• --num-int-attr <value> Number of integer attributes to display. If set to 0, uses the --num-attr value (default: 5)
• --int-percent-threshold <value> Threshold for integer attributes put in dedicated columns (default: 0.05)
• --str-percent-threshold <value> Threshold for string attributes in dedicated columns (default: 0.03)
• --include-well-known Include well-known attributes in the analysis (default: true)
• --blob-threshold <value> Convert column to blob when dictionary size reaches this value (default: 4MiB)
• --max-start-time <value> Oldest start time for a block to be processed. RFC3339 format.
• --min-start-time <value> Newest start time for a block to be processed. RFC3339 format.
• -o, --out <value> File to write output to. If not specified, output is printed to stdout.
• -f, --format <jsonnet|yaml> Output format (default: yaml)

Example outputting YAML to a file:

Bash Copy

tempo-cli suggest columns -c ./tempo.yaml --format yaml --out suggestions.yaml single-tenant

Example outputting Jsonnet for use in overrides:

Bash Copy

tempo-cli suggest columns -c ./tempo.yaml --format jsonnet single-tenant

Generate attribute index

Generate an attribute index for a parquet block. This creates an index file that can be used for faster attribute lookups.

Bash Copy

tempo-cli gen attr-index <input-path>

Arguments:

• input-path Path to the input parquet block directory.

Options:

• --add-intrinsics Add intrinsic attributes to the index such as name, kind, status, and others.
• --index-types <rows|codes|rows,codes> Type of index to generate (default: rows,codes)

Example:

Bash Copy

tempo-cli gen attr-index ./path/to/block

Example with intrinsic attributes:

Bash Copy

tempo-cli gen attr-index --add-intrinsics ./path/to/block

Experimental traces diff

Compare two local trace JSON files. The default trace-patch-v0 format returns the complete mechanical change list. You can instead request a compact native summary or a composed summary with a size-bounded patch.

Use this command to compare traces captured at different times or from different environments, for example, to understand how a deployment changed trace structure.

Bash Copy

tempo-cli experimental traces-diff --trace-a <BASELINE_PATH> --trace-b <COMPARISON_PATH>

Arguments:

• --trace-a <path> (required) Path to the baseline trace JSON file.
• --trace-b <path> (required) Path to the comparison trace JSON file.

Options:

• --format <value> Output format: trace-patch-v0 (default), trace-summary-v0-native, or trace-summary-v0-composed.
• -o, --out <path> File to write output to. If not specified, output is printed to stdout.
• --pretty Pretty-print JSON output.

The input files can be either raw OpenTelemetry JSON traces or Tempo TraceByIDResponse JSON responses.

The trace-summary-v0-composed format always includes a trace-summary-v0-native document. If the serialized trace-patch-v0 document is no larger than 64 KiB, it is included in patch. Otherwise, patchOmitted reports its size and the reason over_budget; rerun with --format trace-patch-v0 to retrieve the full patch.

Example:

Bash Copy

tempo-cli experimental traces-diff --trace-a baseline.json --trace-b compare.json --pretty

Example writing output to a file:

Bash Copy

tempo-cli experimental traces-diff --trace-a baseline.json --trace-b compare.json -o diff-output.json

Example producing the native summary:

Bash Copy

tempo-cli experimental traces-diff \
  --trace-a baseline.json \
  --trace-b compare.json \
  --format trace-summary-v0-native \
  --pretty

The native summary calculates latency, the sum of inclusive span durations, topology, and significant per-service duration drift from the normalized traces. It combines those values with matcher-derived change and error counts. changedServices and service rollups cover all matcher changes and significant duration drift; changedServices also includes structure-only changes, which do not have service rollups.

Warnings report partial inputs, high-cardinality span names, duplicate span IDs, ambiguous duplicate-span matching, empty traces, and invalid durations. Invalid spans contribute zero to summary duration aggregates.

Drop traces by ID

Rewrites all blocks for a tenant that contain specific trace IDs. The traces are dropped from the new blocks and the rewritten blocks are marked compacted so they will be cleaned up.

Bash Copy

tempo-cli rewrite-blocks drop-traces <tenant-id> <trace-ids>

Arguments:

• tenant-id The tenant ID. Use single-tenant for single tenant setups.
• trace-ids The comma-separated trace IDs to drop (also supports single trace ID).

Options:

• Backend options
• --drop-trace By default, this command runs in dry run mode. Supplying this argument causes it to actually rewrite blocks with the traces dropped.
• --background Run in background mode (default: false). Suppresses progress dots for use in automated scripts.

Examples

Dry run (default) to see which blocks would be affected:

Bash Copy

tempo-cli rewrite-blocks drop-traces --backend=local --bucket=./cmd/tempo-cli/test-data/ single-tenant 04d5f549746c96e4f3daed6202571db2

Drop one trace (actually perform the operation):

Bash Copy

tempo-cli rewrite-blocks drop-traces --drop-trace --backend=local --bucket=./cmd/tempo-cli/test-data/ single-tenant 04d5f549746c96e4f3daed6202571db2

Drop multiple traces:

Bash Copy

tempo-cli rewrite-blocks drop-traces --drop-trace --backend=local --bucket=./cmd/tempo-cli/test-data/ single-tenant 04d5f549746c96e4f3daed6202571db2,111fa1850042aea83c17cd7e674210b8

Redact traces

Remove traces containing personally identifiable information or other sensitive data from object storage without waiting for retention to expire.

The redact command submits a redaction request to the backend scheduler. The scheduler creates jobs that rewrite affected blocks in object storage to remove the specified traces. Unlike drop-traces, which operates directly on object storage from the CLI, redact delegates the work to the backend scheduler over gRPC.

Bash Copy

tempo-cli redact --tenant=<TENANT_ID> --trace-id=<TRACE_ID> [--trace-id=<TRACE_ID> ...] <scheduler-address>

Bash Copy

tempo-cli redact --tenant=<TENANT_ID> --query=<TRACEQL_QUERY> [--start=<START> --end=<END>] <scheduler-address>

Arguments:

• scheduler-address The backend scheduler gRPC address (host:port).

Options:

• --tenant <value> (required) Tenant ID.
• --trace-id <value> Trace ID to redact, in hex format. Repeat the flag for several traces in one request (--trace-id=<ID> --trace-id=<ID>, not comma-separated), up to 1000. Every job the redaction creates carries the whole list, so a longer list costs one copy per block; use --query instead. Mutually exclusive with --query.
• --query <value> TraceQL query selecting the traces to redact, for example { span.http.status_code = 500 }. Mutually exclusive with --trace-id. The query is restricted to a single spanset filter: = comparisons on the matched span’s own resource.* or span.* attributes, joined by && or ||. Regular expressions, != or ordered comparisons, parent.-scoped attributes, and pipelines or aggregates aren’t supported.
• --dry-run Evaluate the selector without rewriting any blocks. After the dry-run jobs complete, match counts are added to tempo_backend_scheduler_redaction_traces_found_total (mode="dry_run"). The command doesn’t print the count (default: false).
• --start <value> Start of the time window. Accepts now, a relative offset such as now-7d, or an RFC3339 timestamp. Must be given with --end, must be before --end, and cannot be combined with --trace-id. Omit both bounds to redact the whole tenant.
• --end <value> End of the time window. Same forms as --start. Must be given with --start.
• --tls Use TLS for the gRPC connection (default: false).
• --tls-server-name <value> Override the TLS server name (SNI).
• --tls-ca <value> Path to a PEM-encoded CA certificate file.

You must provide exactly one of --trace-id or --query. Providing both, or neither, returns an error before the request is submitted.

A tenant can have only one redaction in progress at a time, dry runs included. A submission made while an earlier one is still running, or still in its quiescence period, is rejected. A trace-ID list larger than the cap therefore has to be redacted as successive batches rather than several at once, which is another reason to prefer --query.

On success, the command prints the batch ID and the number of jobs created:

Copy

batch_id:     <BATCH_ID>
jobs_created: <COUNT>

When --dry-run is set, the command also prints a line indicating that no blocks were rewritten:

Copy

batch_id:     <BATCH_ID>
jobs_created: <COUNT>
mode:         dry-run (jobs will report match counts; no blocks will be rewritten)

Before you submit a redaction

Redaction only rewrites blocks that already exist in object storage when you submit. Traces still held by block-builders aren’t covered, so recently ingested traces can survive a run.

1. Stop ingesting the sensitive data at its source.
2. Wait for the current blocks to flush to object storage. Blocks flush on an interval of a few minutes; allow around 10 minutes to be safe, so the traces you want to remove land in blocks the redaction can reach.

If you use --query:

1. Check your redaction query in Grafana Explore to confirm it selects the right traces.
2. Run the same query with --dry-run. The command returns as soon as jobs are created; it doesn’t print the match count. Wait until those jobs complete.
3. Monitor progress on the /status/backendscheduler endpoint.
4. Read tempo_backend_scheduler_redaction_traces_found_total for your tenant with mode="dry_run". The metric is a counter that increments when each job finishes, so use an increase over the run or the Dry-run Blast Radius / h panel on the Tempo - Backend Work dashboard. A value of zero can mean no matches or that jobs haven’t finished yet. For the metric and dashboard, refer to Key metrics. If the count is far larger than the Explore sample, narrow the query first.

Then submit the redaction. For commands, refer to Examples.

Check your redaction query in Grafana Explore

Use this procedure when you redact with --query. Enter the same query string you pass to --query.

1. In Grafana, go to Explore and select your Tempo data source.
2. For Query type, select TraceQL, then enter your redaction query, for example { span.http.status_code = 500 }.
3. Select Run query and inspect the matching traces in the results.

For help building and running TraceQL queries, refer to TraceQL queries in Grafana.

The --query option accepts only a restricted subset of TraceQL: a single spanset filter with positive equality matchers. That subset is deliberately narrow so a redaction query can’t widen its own match set the way negation or regular expressions could. A broader query that works in Explore may be rejected at submission. Explore shows a sample; use --dry-run for the count before you apply the redaction.

Redact a time window

A redaction with no window covers every block the tenant has, which keeps the tenant’s compaction paused for the whole run and lets its block list grow. --start and --end scope a redaction to a time range, so a large tenant can be redacted in slices with compaction recovering in between.

Bash Copy

tempo-cli redact --tenant=my-tenant --query='{resource.namespace = "checkout"}' --start=now-7d --end=now-6d localhost:9095

Both bounds are inclusive and must both be supplied. Blocks whose data range overlaps the window are read, as are blocks whose recorded range is unusable — those are included rather than skipped, so that a block whose timestamps cannot be judged is never silently left behind.

Inside each block the window bounds the scan, and a trace is redacted if any part of it overlaps: a trace that starts before the window and ends inside it is removed in full, including its earlier spans.

A window cannot be combined with --trace-id. The window scopes which blocks are read and is not applied per trace, so the pair would remove each listed trace only from the blocks that happen to overlap and leave the rest of it in place while reporting success. Redact by trace ID without a window.

The window is resolved to absolute timestamps when the command runs, so a long redaction does not drift forward into data that arrived after it started. Traces outside every window you run are left in place.

Repeat the command for each slice, but expect to wait between slices. A finished redaction is held briefly before it is cleared, and a second submission for the same tenant is rejected with AlreadyExists until that completes — a couple of minutes at the default maintenance interval.

Monitor job progress through the /status/backendscheduler endpoint.

Examples

Redact a single trace:

Bash Copy

tempo-cli redact --tenant=my-tenant --trace-id=931281e2a09876de16e15f45ff86283d localhost:9095

Redact multiple traces in one request:

Bash Copy

tempo-cli redact --tenant=my-tenant --trace-id=931281e2a09876de16e15f45ff86283d --trace-id=00000000000000000000000000000001 localhost:9095

Preview the traces a query would match, without rewriting any blocks:

Bash Copy

tempo-cli redact --tenant=my-tenant --query='{ span.http.status_code = 500 }' --dry-run localhost:9095

After you preview, redact all traces matching that query:

Bash Copy

tempo-cli redact --tenant=my-tenant --query='{ span.http.status_code = 500 }' localhost:9095

With TLS and a custom CA:

Bash Copy

tempo-cli redact --tenant=my-tenant --trace-id=931281e2a09876de16e15f45ff86283d --tls --tls-ca=/path/to/ca.pem scheduler.example.com:9095