Manage Tempo on Grafana Labs

Compaction

Thu, 28 May 2026 17:50:33 +0100

Compaction

Tempo stores trace data as immutable blocks in object storage. Over time, many small blocks accumulate from frequent flushes. Compaction merges groups of small blocks into fewer, larger blocks, reducing storage costs and improving query performance by limiting the number of blocks a querier must scan.

Block states

Every block has one of two states, signaled by which metadata file is present in the backend:

Live (meta.json): the block is queryable and included in query plans.
Compacted (meta.compacted.json): the block’s data has been merged into a new block. The file is renamed from meta.json to meta.compacted.json at the time of compaction.

Blocks are never deleted immediately after compaction. Because the blocklist is polled periodically, queriers may not yet know about the new merged block when the old blocks are first marked compacted. The compacted_block_retention window keeps the old blocks readable until all pollers have caught up.

Blocklist and poller

Tempo components rely on an in-memory blocklist to know which blocks exist in object storage. They do not scan object storage directly on each request. The blocklist is maintained by the poller, a background process that reads the backend on a fixed interval controlled by blocklist_poll (default 5 minutes). The poller is how compaction results become visible to queriers and other components: newly merged blocks appear in the blocklist and compacted blocks are removed. A component can only serve data from blocks the poller has discovered. If the poller falls behind or fails, the component’s view of the backend becomes incomplete.

The poller reads a pre-built tenant index from the backend rather than scanning all objects on each cycle. If the tenant index is missing or older than blocklist_poll_stale_tenant_index, the component falls back to scanning the backend directly. Setting blocklist_poll_fallback: true (the default) enables this fallback; disabling it means a stale or missing index causes the poll to fail rather than recover.

Because the poller runs on an interval, the blocklist is always slightly out of date. In a healthy system, the blocklist will be stale by at most twice the configured blocklist_poll duration. This is because a writer may update the backend at any point within a polling cycle, and the reader may have just started its own cycle at that same moment.

Timing requirements

The 2x staleness bound drives two concrete configuration requirements.

query_backend_after must be at least 2x blocklist_poll

The query-frontend uses query_backend_after to decide where to search:

Traces more recent than query_backend_after are served from live-stores only.
Traces older than query_backend_after are served from the backend.

A newly flushed block will not appear in the querier’s blocklist until up to 2 * blocklist_poll after it was written. During that window, the block would be missed if the query-frontend routed the request to the backend. Setting query_backend_after to at least 2 * blocklist_poll ensures that such blocks are still covered by live-stores and are not yet expected to appear in the backend.

With the default blocklist_poll of 5 minutes, query_backend_after should be at least 10 minutes. The default of 15 minutes for search provides a comfortable margin.

compacted_block_retention must be at least 2x blocklist_poll

When a block is compacted, its data is merged into a new block and the old block is marked compacted. A querier with a stale blocklist may still reference the old block until it completes its next poll cycle. Setting compacted_block_retention to at least 2 * blocklist_poll ensures those queriers can still read the old block before it is deleted. The default of 1 hour is well above the minimum for typical configurations.

Block selection

Before any compaction work begins, the block selector determines which blocks to compact together.

The time window block selector groups blocks by their time range and compaction level. Blocks are only compacted with other blocks that share the same compaction window, data encoding, format version, and dedicated column configuration.

The selector divides the blocklist into two zones based on the active window (the most recent 24 hours):

Inside the active window: blocks are grouped by compaction level, then by time window. Lower compaction levels are prioritized, and within each group the smallest blocks are selected first.
Outside the active window: blocks are grouped by time window only, ignoring compaction level. This allows older data to be consolidated regardless of how many times it has already been compacted.

The selector only returns a group for compaction if it contains at least the minimum number of input blocks (default 2). Each input set is also bounded by max_block_bytes (default 100 GB). If a full group would exceed this limit, the selector picks the largest contiguous subset of blocks that fits; any remaining blocks in the group are eligible in subsequent compaction runs.

The number of input blocks per compaction job is configurable via min_input_blocks and max_input_blocks, defaulting to a minimum of 2 and a maximum of 4.

How compaction works

Compaction is handled by the backend scheduler and backend workers. The scheduler is a singleton that assigns compaction jobs to a pool of workers. Workers request jobs from the scheduler and report back when complete; the scheduler tracks active jobs and avoids handing out overlapping work.

The scheduler uses a priority queue to choose which tenant to work on next. Outstanding blocks are those the block selector has identified as eligible for compaction – blocks that meet the window, encoding, and minimum-group-size requirements. Tenants are ranked by the sum of their total blocklist length and their outstanding block count, so tenants with more overall backend work receive jobs first. A tenant with no outstanding blocks receives a priority of zero and is not scheduled.

The scheduler processes one tenant at a time, issuing jobs until the tenant’s eligible blocks are exhausted or the per-tenant job limit (max_jobs_per_tenant, default 1000) is reached, then moves to the next tenant. This cap is the mechanism that prevents a single busy tenant from monopolizing the workers.

Within a tenant, the block selector prioritizes lower compaction levels first. Higher-level blocks are still compacted, but after lower-level work is cleared.

The compaction provider waits for a poll notification before selecting new work, preventing idle spin when no blocks are eligible. A minimum cycle interval (min_cycle_interval, default 30 seconds) rate-limits tenant prioritization when the queue is empty.

Autoscaling

Outstanding block counts are measured for all tenants on a periodic ticker (measure_interval, default 1 minute), giving an aggregate view of total backend work. Autoscaling thresholds should be set against this aggregate total rather than per-instance averages. For a reference KEDA autoscaling configuration driven by outstanding block metrics, see operations/jsonnet/microservices/autoscaling.libsonnet.

Configuration reference

For a full configuration reference, refer to the configuration documentation.

The following fields are most relevant to compaction behavior.

Polling interval (`storage.trace`)

storage:
  trace:
    # How often to poll the backend for new blocks. Default: 5m.
    blocklist_poll: 5m

Compaction settings (`backend_worker.compaction`)

backend_worker:
  compaction:
    # Size of the time window used to group blocks. Default: 1h.
    compaction_window: 1h

    # Maximum size of a compacted output block in bytes. Default: 100GB.
    max_block_bytes: 107374182400

    # How long to keep a block after compaction. Must be >= 2x blocklist_poll. Default: 1h.
    compacted_block_retention: 1h

Query cutoff (must be >= 2x blocklist_poll)

query_frontend:
  search:
    # Default: 15m.
    query_backend_after: 15m

Scheduler compaction (`backend_scheduler.provider.compaction`)

backend_scheduler:
  provider:
    compaction:
      # Maximum jobs to issue per tenant before switching. Default: 1000.
      max_jobs_per_tenant: 1000

      # Minimum number of input blocks per job. Default: 2.
      min_input_blocks: 2

      # Maximum number of input blocks per job. Default: 4.
      max_input_blocks: 4

      # Minimum time between tenant selection cycles. Default: 30s.
      min_cycle_interval: 30s

Note
The backend_worker.compaction configuration block contains two fields that are not used by the current scheduled compaction path: max_time_per_tenant and compaction_cycle. These fields were used by the ring-based compaction loop, which has been removed. They are accepted by the configuration parser but have no effect.

Monitor Tempo

Thu, 28 May 2026 17:50:33 +0100

Monitor Tempo

Tempo is instrumented to expose metrics, logs, and traces. Furthermore, the Tempo repository has a mixin that includes a set of dashboards, rules, and alerts. Together, these can be used to monitor Tempo in production.

Instrumentation

Metrics, logs, and traces from Tempo can be collected to observe its services and functions.

Metrics

Tempo is instrumented with Prometheus metrics and emits RED metrics for most services and backends. RED metrics are a standardized format for monitoring microservices, where R stands for requests, E stands for errors, and D stands for duration.

The Tempo mixin provides several dashboards using these metrics.

Query frontend inspected bytes

Use tempo_query_frontend_bytes_inspected_total to monitor how many bytes the query frontend reads from disk and object storage. This counter is emitted per tenant and op (traces, search, metadata, metrics). Because cached responses from queriers are excluded, it reflects actual storage and network I/O.

For PromQL examples and alerting guidance, refer to Query query IO and time stamp distance.

Logs

Tempo emits logs in the key=value (logfmt) format.

Traces

Tempo uses the OpenTelemetry SDK for tracing instrumentation. The complete read path and some parts of the write path of Tempo are instrumented for tracing.

You can configure the tracer using environment variables. To enable tracing, set one of the following: OTEL_EXPORTER_OTLP_ENDPOINT or OTEL_EXPORTER_OTLP_TRACES_ENDPOINT.

The OpenTelemetry SDK uses OTLP/HTTP by default, which can be configured with OTEL_EXPORTER_OTLP_PROTOCOL.

Polling

Tempo maintains knowledge of the state of the backend by polling it on regular intervals. Several components need this knowledge, including schedulers, workers, queriers, and query-frontends.

Refer to Use polling to monitor the backend status for Tempo.

Dashboards

The Tempo mixin has eight Grafana dashboards in the dashboards folder that you can download and import into your Grafana UI. These dashboards work well when you run Tempo in a Kubernetes (k8s) environment and metrics scraped have the cluster and namespace labels.

Tempo Reads dashboard

This is available as tempo-reads.json.

The Reads dashboard gives information on Requests, Errors, and Duration (RED) on the query path of Tempo. Each query touches the Gateway, Query-Frontend, Queriers, Live-stores, Memcached, and the backend.

Use this dashboard to monitor the performance of each of the mentioned components and to decide the number of replicas in each deployment.

Tempo Writes dashboard

This is available as tempo-writes.json.

The Writes dashboard gives information on RED on the write/ingest path of Tempo. A write query touches the Gateway, Distributors, and Kafka. The dashboard also shows compaction activity against Memcached and the backend.

Use this dashboard to monitor the performance of each of the mentioned components and to decide the number of replicas in each deployment.

Tempo Resources dashboard

This is available as tempo-resources.json.

The Resources dashboard provides information on CPU, Container Memory, and Go Heap Inuse. This dashboard is useful for resource provisioning for the different Tempo components.

Use this dashboard to see if any components are running close to their assigned limits.

Tempo Operational dashboard

This is available as tempo-operational.json.

The Tempo Operational dashboard deserves special mention because it is probably a stack of dashboard anti-patterns. It’s big and complex, doesn’t use jsonnet, and displays far too many metrics in one place. For just getting started, the RED dashboards are great places to learn how to monitor Tempo in an opaque way.

This dashboard is included in the Tempo repository for two reasons:

The dashboard provides a stack of metrics for other operators to consider monitoring while running Tempo.
We want the dashboard in our internal infrastructure and we vendor the tempo-mixin to do this.

Tempo Backend Work dashboard

This is available as tempo-backendwork.json.

The Backend Work dashboard monitors blocklist maintenance, compaction jobs, and backend component resources. It tracks blocklist length and poll duration, active and completed compaction jobs, failure and retry rates, and objects written and combined during compaction. The dashboard also shows CPU and memory usage for the backend-scheduler and backend-worker components.

Use this dashboard to monitor compaction health, detect stalled or failing jobs, and right-size backend-scheduler and backend-worker resources.

Tempo Block Builder dashboard

This is available as tempo-block-builder.json.

The Block Builder dashboard monitors the Kafka-based ingest path introduced. It tracks Kafka fetch rates and read throughput, flushed blocks per second, and per-partition lag in both records and seconds. Heatmaps show partition section and cycle durations, and resource panels track CPU and memory usage.

Use this dashboard to monitor block-builder throughput, detect partition lag, and ensure the ingest pipeline keeps up with incoming trace data.

Tempo Rollout Progress dashboard

This is available as tempo-rollout-progress.json.

The Rollout Progress dashboard tracks the health of a Tempo deployment during upgrades and rollouts. It breaks down write and read requests by status code (2xx, 4xx, 5xx), shows 99th percentile latency, and counts unhealthy Pods. A version panel shows the number of pods running each version, and a latency comparison panel shows current latency against the previous 24-hour baseline.

Use this dashboard during upgrades to confirm that new versions aren’t introducing errors or latency regressions.

Tempo Tenants dashboard

This is available as tempo-tenants.json.

The Tenants dashboard provides per-tenant visibility into ingestion, reads, storage, and metrics generation. It shows a limits table alongside distributor bytes and spans per second, live trace counts, query rates for ID lookups and searches, blocklist length, outstanding compactions, and metrics-generator bytes and active series.

Use this dashboard in multitenant deployments to identify tenants with high ingestion rates, query volumes, or storage growth.

Rules and alerts

The Rules and Alerts are available as YAML files in the compiled mixin on the repository.

To set up alerting, download the provided JSON files and configure them for use on your Prometheus monitoring server.

Check the runbook to understand the various steps that can be taken to fix firing alerts.

Manage trace ingestion

Thu, 28 May 2026 17:50:33 +0100

Manage trace ingestion

If you are seeing RATE_LIMITED, LIVE_TRACES_EXCEEDED, or TRACE_TOO_LARGE errors, or if your trace storage costs are rising unexpectedly, this page can help.

Grafana Tempo enforces ingestion limits at multiple points in the write path. The distributor checks rate limits before writing spans to Kafka. Downstream, live-stores enforce per-trace size and live trace count limits asynchronously, and block-builders enforce per-trace size limits. If limits are too low for your workload, spans are discarded and data is lost. If limits are unchecked, ingestion volume can grow beyond what you intended.

This page covers three tasks:

Size ingestion limits when deploying or reviewing your configuration.
Find and fix discarded spans when you are actively losing data.
Identify what is driving ingestion volume when costs are growing.

For an overview of how trace data flows through the write path, refer to Tempo architecture.

Size ingestion limits for your workload

Tempo enforces three ingestion limits. Understanding what the defaults mean for your traffic helps you set appropriate values at deploy time, rather than discovering them through production incidents.

Rate limit

rate_limit_bytes (default: 15,000,000) sets the sustained byte rate each distributor allows per tenant, measured in bytes per second. For a typical span size of around 500 bytes, the default accommodates roughly 30,000 spans per second per distributor.

How this scales depends on your rate_strategy:

local (default): each distributor enforces the limit independently. With three distributors, the effective cluster limit is approximately 90,000 spans per second.
global: the configured rate is shared across all distributors. The total cluster rate equals the configured value regardless of how many distributors you run.

burst_size_bytes (default: 20,000,000) allows temporary spikes above the sustained rate, for example during application deployments. The burst allowance is always applied locally, regardless of rate strategy.

Live trace limit

max_traces_per_user (default: 10,000) caps the number of concurrently active traces per tenant on each live-store. This limit is enforced asynchronously in the live-store, not at ingestion time in the distributor. Block-builders do not enforce this limit. If your services produce many short-lived traces in parallel, you may need to raise this.

Note
The max_global_traces_per_user setting, which provides a cluster-wide cap for the ingester write path, has been moved to ingestion.max_global_traces_per_user in Tempo 3.0.

Per-trace size limit

max_bytes_per_trace (default: 5,000,000) caps the total size of a single trace. This limit is enforced asynchronously in live-stores and block-builders. Traces that exceed this limit are partially dropped. Unusually large traces often indicate a retry loop or misconfigured instrumentation rather than normal application behavior.

Example configuration

To estimate the rate limit you need, multiply your average span size by your peak spans-per-second across all services for a given tenant.

The following example raises the defaults for a high-throughput workload:

overrides:
  defaults:
    ingestion:
      rate_strategy: local
      rate_limit_bytes: 30000000
      burst_size_bytes: 40000000
      max_traces_per_user: 50000
    global:
      max_bytes_per_trace: 10000000

If you run a multi-tenant deployment, you can set different limits per tenant using runtime overrides instead of raising the global defaults. Refer to Enable multi-tenancy for per-tenant override examples.

For the full list of available settings, refer to Ingestion limits in the configuration reference. You can also manage per-tenant limits through the API using user-configurable overrides.

Find and fix discarded spans

When a span exceeds an ingestion limit or fails validation, Tempo discards it and increments the tempo_discarded_spans_total metric. The distributor rejects entire push requests that exceed the rate limit or contain invalid trace or span IDs, before any spans reach Kafka. Live-stores discard spans that exceed per-trace size or live trace count limits after consuming them from Kafka. Block-builders discard spans that exceed per-trace size limits.

Error reference

The following table lists the three error types, what each one means, and how to fix it.

Error	Cause	Fix
`RATE_LIMITED`	The tenant’s byte rate exceeded `rate_limit_bytes`.	Raise `rate_limit_bytes`, or add distributors if using `rate_strategy: local`. If volume is genuinely higher than intended, reduce it upstream with sampling.
`LIVE_TRACES_EXCEEDED`	The number of concurrent active traces on a live-store exceeded `max_traces_per_user`.	Raise `max_traces_per_user`, or add live-store instances to distribute the active trace count across more nodes.
`TRACE_TOO_LARGE`	A single trace exceeded `max_bytes_per_trace` (default 5 MB).	Raise `max_bytes_per_trace` in the `global` overrides. Also investigate why the trace is so large. Common causes include retry loops and misconfigured instrumentation.

Check why spans are being discarded

Query the tempo_discarded_spans_total metric. The reason label identifies why Tempo discarded each span:

sum by (reason) (rate(tempo_discarded_spans_total[5m]))

The following table lists the possible reason values:

Reason	Meaning	Component
`rate_limited`	Tenant byte rate exceeded `rate_limit_bytes`.	Distributor
`trace_too_large`	Single trace exceeded `max_bytes_per_trace`.	Live-store, block-builder
`live_traces_exceeded`	Active trace count exceeded `max_traces_per_user`.	Live-store
`invalid_trace_id`	Batch contained a trace ID that isn’t 128 bits.	Distributor
`invalid_span_id`	Batch contained a span ID that isn’t 64 bits or was all zeros.	Distributor
`trace_too_large_to_compact`	Trace too large for the backend-worker to compact.	Backend-worker
`unknown_error`	Unexpected error during span processing.	Live-store

Log discarded spans for debugging

To log spans discarded by the distributor (rate-limited spans) with their trace IDs, enable log_discarded_spans in the distributor configuration:

distributor:
  log_discarded_spans:
    enabled: true

Set include_all_attributes: true for more verbose output that includes span attributes.

Spans discarded by live-stores for LIVE_TRACES_EXCEEDED or TRACE_TOO_LARGE are logged at debug level by the live-store. To see these entries, set the live-store log level to debug.

Refer to Distributor refusing spans for additional troubleshooting steps.

Traces missing from queries without errors

If the distributor is not refusing spans but traces are missing from query results, the issue may be downstream. Live-stores consume trace data from Kafka and serve recent queries. If a live-store falls behind its Kafka partition, query results may be incomplete.

Monitor the tempo_live_store_lagged_requests_total metric to detect when this happens. This counter increments every time a search or metrics query hits a live-store whose Kafka lag overlaps the requested time range, meaning results may be incomplete. The metric is labeled by route (/tempopb.Querier/SearchRecent or /tempopb.Metrics/QueryRange).

The fail_on_high_lag setting (default false) controls how the live-store responds when lag is detected:

When false, the live-store returns whatever data it has, which may be incomplete. The metric still increments.
When true, the live-store returns an error when it cannot guarantee completeness.

Refer to Unable to find traces for query-side troubleshooting.

Identify what is driving ingestion volume

When your cluster is healthy but ingestion is growing, the first step is finding which services are responsible for the most volume.

Set up cost attribution

The usage tracker breaks down ingested bytes by configurable attributes, giving you a per-service view of who is consuming capacity.

Enable cost attribution in the distributor and configure which attributes to track in overrides:

distributor:
  usage:
    cost_attribution:
      enabled: true

overrides:
  defaults:
    cost_attribution:
      dimensions:
        resource.service.name: "service"

Find the top contributors

After enabling cost attribution, the distributor exposes the tempo_usage_tracker_bytes_received_total metric on the /usage_metrics endpoint, labeled by the dimensions you configured.

You can query this endpoint directly:

curl http://<distributor-host>:3200/usage_metrics

If you scrape this endpoint with Prometheus, you can use the following query to find which services are sending the most data:

topk(10,
  sum by (service) (
    rate(tempo_usage_tracker_bytes_received_total[1h])
  )
)

For the full set of configuration options, including scoping dimensions by resource or span and customizing label names, refer to Usage tracker.

Reduce volume from noisy services

After you know which services are driving volume, the most effective way to reduce it is sampling at the collector layer. Tail sampling lets you keep traces with errors or high latency while dropping routine ones, reducing volume without losing visibility into the problems that matter.

Dedicated attribute columns

Thu, 28 May 2026 17:50:33 +0100

Dedicated attribute columns

Dedicated attribute columns improve query performance by storing the most frequently used attributes in their own columns, rather than in the generic attribute key-value list.

Dedicated attribute columns are available when using vParquet4 or later block formats.

vParquet5 expands on the feature with doubled string column limits, integer dedicated columns, event-scoped attributes, array-valued attributes, and blob attributes using the options field.

Configuration

You can configure dedicated attribute columns in the storage block or via overrides.

# Storage configuration for traces
storage:
  trace:
    block:
      # Default dedicated columns for all blocks
      parquet_dedicated_columns:
        - name: <string> # name of the attribute
          type: <string> # type of the attribute. options: string (vParquet3+), int (vParquet5 only)
          scope: <string> # scope of the attribute. options: resource, span (vParquet3+), event (vParquet5 only)
          options: [<string>] # optional, vParquet5 only. options: array, blob

overrides:
  # Global overrides for dedicated columns configuration
  parquet_dedicated_columns:
    - name: <string> # name of the attribute
      type: <string> # type of the attribute. options: string, int
      scope: <string> # scope of the attribute. options: resource, span, event
      options: [<string>] # optional, vParquet5 only. options: array, blob

  per_tenant_override_config: /conf/overrides.yaml
---
# /conf/overrides.yaml
# Tenant-specific overrides configuration
overrides:
  "<tenant id>":
    parquet_dedicated_columns:
      - name: <string> # name of the attribute
        type: <string> # type of the attribute. options: string, int
        scope: <string> # scope of the attribute. options: resource, span, event
        options: [<string>] # optional, vParquet5 only. options: array, blob

  # A "wildcard" override can be used that will apply to all tenants if a match is not found.
  "*":
    parquet_dedicated_columns:
      - name: <string> # name of the attribute
        type: <string> # type of the attribute. options: string, int
        scope: <string> # scope of the attribute. options: resource, span, event
        options: [<string>] # optional, vParquet5 only. options: array, blob

Priority is given to the most specific configuration, so tenant-specific overrides will take precedence over global overrides. Similarly, default overrides take precedence over storage block configuration.

Usage

The number of dedicated attribute columns depends on the block format version:

vParquet4: Up to 10 string attributes per scope (span and resource). Integer dedicated columns and event scope are not supported.
vParquet5: Up to 20 string attributes and 5 integer attributes per scope (span, resource, and event).

As a rule of thumb, good candidates for dedicated attribute columns are attributes that contribute the most to the block size, even if they aren’t frequently queried. Reducing the generic attribute key-value list size significantly improves query performance.

Integer attribute selection

Integer dedicated columns are most effective when the attribute is present on at least 5% of rows in a scope. The tempo-cli analyse block command shows the percentage of rows for each integer attribute and marks candidates that meet this threshold with a recommendation. Sparse integer attributes (below 5% prevalence) typically perform better in the generic attribute storage.

Array-valued attributes

Dedicated attribute columns support array-valued attributes. This is useful for attributes that can have multiple values, such as HTTP headers or custom tags.

To enable array support, add options: ["array"] to the dedicated attribute column configuration. The options field is only available in vParquet5 and later. Earlier versions ignore this field and only support single-valued attributes.

When options: ["array"] is set, the dedicated column stores multiple values per attribute. Without this option (default), only single values are supported.

storage:
  trace:
    block:
      version: vParquet5
      parquet_dedicated_columns:
        # Single-valued string attribute (default behavior)
        - name: http.method
          type: string
          scope: span

        # Array-valued string attribute
        - name: http.request.header.accept
          type: string
          scope: span
          options: ["array"]

        # Array-valued integer attribute
        - name: custom.retry.counts
          type: int
          scope: span
          options: ["array"]

Blob attributes

Blob attributes are designed for high-cardinality or high-length string values where dictionary encoding becomes inefficient. Examples include UUIDs, stack traces, request bodies, or any attribute with many unique values.

When an attribute’s dictionary size per row group exceeds a threshold (default 4MiB), the column should be marked as a blob. The tempo-cli analyse block command can help identify blob candidates by showing the estimated dictionary size per row group.

To enable blob mode, add options: ["blob"] to the dedicated attribute column configuration. Blob columns use zstd compression instead of dictionary encoding, which is more efficient for high-cardinality data.

storage:
  trace:
    block:
      version: vParquet5
      parquet_dedicated_columns:
        # Standard dedicated column (uses dictionary encoding)
        - name: http.method
          type: string
          scope: span

        # Blob column for high-cardinality data
        - name: db.statement
          type: string
          scope: span
          options: ["blob"]

        # Blob column for stack traces
        - name: exception.stacktrace
          type: string
          scope: event
          options: ["blob"]

Use the tempo-cli analyse block command with the --blob-threshold option to identify attributes that should be configured as blobs. Attributes exceeding the threshold are marked as “(blob)” in the output.

Event-scoped attributes

With vParquet5, dedicated columns support event-scoped attributes in addition to span and resource scopes. Event-scoped columns are useful for frequently queried event attributes such as exception details or custom event data.

storage:
  trace:
    block:
      version: vParquet5
      parquet_dedicated_columns:
        # Event-scoped string attribute
        - name: exception.message
          type: string
          scope: event

        # Event-scoped attribute with blob option
        - name: exception.stacktrace
          type: string
          scope: event
          options: ["blob"]

Tempo-cli

You can use the tempo-cli tool to find good candidates for dedicated attribute columns. The tempo-cli provides the commands analyse block <tenant-id> <block-id> and analyse blocks <tenant-id> that will output the top N attributes by size for a given block or all blocks in a tenant.

Example:

tempo-cli analyse blocks --backend=local --bucket=./cmd/tempo-cli/test-data/ single-tenant

Refer to the tempo-cli documentation for more information.

Tune search performance

Thu, 28 May 2026 17:50:33 +0100

Tune search performance

Regardless of whether or not you are using TraceQL or the original search API, Tempo searches all of the blocks in the specified time range. Depending on your volume, this may result in slow queries.

This document explains how read path in Tempo works, what controls you have to tune the read path, details of the core configuration options, and how to tune these configuration options to get the most out of your Tempo cluster.

The general advice is to scale your workers and queriers. Additional queriers can more effectively run jobs in parallel while additional workers more aggressively reduce the length of your blocklist and copies of data (if using RF=3).

Note
All forms of search (TraceQL and tags based) require vParquet block formats. The v2 block format has been removed in Tempo 3.0.

Dedicated attribute columns are another effective method to improve search performance.

Before you begin

You should understand the basic Tempo architecture.

For more information, refer to the Tempo architecture to learn about how Tempo works.

Glossary

query: A search query, issued by end user. For example, { traceDuration > 1s } is a TraceQL query.
job: A shard of a search query, the lowest unit of work. A query is broken down into multiple jobs and processed.
batch: A group of jobs is called a batch.
frontend: Another name for query-frontend
querier: Responsible for executing (processing) the jobs, and sending the results back to query-frontend.

Tempo query path

Tempo has a query path made up of a query-frontend, querier, live-store, and backend.

You can think about each component in a query path as a generic producer and worker model:

The query-frontend is a producer and it has multiple workers connected to it. The query-frontend takes a single query and shards it into multiple jobs (units of work).
The queriers are workers. They enqueue work from a queue, process it, and send the results back to the producer (query-frontend).
The querier either reads data from backend and processes the query, or it delegates the query to live-stores based on the type of job, time range, and query type.

Lifecycle of a query

A search request comes from the user, and it is split into multiple jobs in the query-frontend.

These jobs are then added to a queue and queriers pick up jobs in batches to process.

A single search query can create hundreds of jobs based on the time range and other variables. These jobs are batched and sent to queriers.

The querier receives the batch and processes the jobs in the batch, builds results, and sends the results of the whole batch back to the query-frontend.

The query-frontend merges and deduplicates the results, and sends them back to the client.

When querier starts, it opens multiple connections to each query-frontend.

Job batches are sent over these connections, and return the results. This process is synchronized so a single job batch can block a connection.

The number of connections control the number of batches a querier processes concurrently. The number of connections is controlled by querier.max_concurrent_queries OR frontend_worker.parallelism.

General guidelines

Tuning the search pipeline can be difficult as it requires balancing a number of different configuration parameters. The below tips can you get your head around the general problem, but the specifics require experimentation.

Review the query-frontend logs for lines like the following to get a feeling for how many jobs your queries are creating:

level=info ts=2023-07-19T19:38:01.354220385Z caller=searchsharding.go:236 msg="sharded search query request stats and SearchMetrics" ...

For a single TraceQL query the maximum number of parallel jobs is constrained by:
- query_frontend.search.concurrent_jobs: This is the maximum number of jobs the frontend dispatches for one TraceQL query.
- # queriers * querier.max_concurrent_queries * query_frontend.max_batch_size: This is the maximum job capacity of your Tempo cluster. If a given TraceQL query produces less jobs then these two values it should be executed entirely in parallel on the queriers.
Increasing querier.max_concurrent_queries is a great way to get more out of your queriers. However, if queriers are OOMing or saturating other resources then this should be lowered. Lowering query_frontend.max_batch_size will also reduce the total work attempted by one querier.

Querier and query-frontend configuration

Queriers and query-frontends have additional configuration related to search of the backend datastore.

Querier

querier:
  # Control the amount of work each querier will attempt. The total number of
  # jobs a querier will attempt this is this value * query_frontend.max_batch_size
  max_concurrent_queries: 20

Query-frontend

Query frontend lists all configuration options.

These suggestions help deal with scaling issues.

server:
  # At larger scales, searching starts to feel more like a batch job.
  # Increase the server timeout intervals.
  http_server_read_timeout: 2m
  http_server_write_timeout: 2m

query_frontend:
  # When increasing concurrent_jobs, also increase the queue size per tenant,
  # or search requests will be cause 429 errors. This is the total number of jobs
  # per tenant allowed in the queue.
  max_outstanding_per_tenant: 2000

  # The number of jobs the query-frontend will batch together when passing jobs to the queriers. This value
  # This value * querier.max_concurrent_queries is your the max number of jobs a given querier will try at once.
  max_batch_size: 3

  search:
    # At larger scales, increase the number of jobs attempted simultaneously,
    # per search query.
    concurrent_jobs: 2000

    # The query-frontend will attempt to divide jobs up by an estimate of job size. The smallest possible
    # job size is a single parquet row group. Increasing this value will create fewer, larger jobs. Decreasing
    # it will create more, smaller jobs.
    target_bytes_per_job: 50_000_000

Settings that are safe to increase without major impact.

Scaling up queriers is a safe way to add more query capacity. At Grafana Labs, we prefer to scale queries horizontally by adding more replicas. If you see out of memory (OOM) errors, it might be worth scaling the queriers vertically.

We recommend running at least two replicates of query-frontend. These replicas should be scaled vertically instead of horizontally. If you need to scale, scale query-frontends vertically by adding more CPU and RAM. Currently, query-frontends aren’t scaled horizontally, but this might change in the future.

The reason we keep only two replicas is because each query-frontend has its own request queue and it also impacts the amount of jobs sent to each querier. If you add more query-frontends, you need to tune other configuration parameters to account for the change. We decided to keep query-frontend to two replicas for now.

In a dedicated cluster, you can increase query_frontend.max_outstanding_per_tenant because the cluster is dedicated to a single customer. In a shared cluster, you need to use more caution when increasing querier.max_outstanding_per_tenant

Parameter interactions

The query_frontend configuration options control how Tempo shards a query into multiple jobs. These options also control the size of a single job, number of jobs, and the number of jobs added to the work queue at once (concurrently) for queriers to be picked up. The max_batch_size option controls how many jobs are sent to queriers at once in a single batch.

Querier configuration options control processing of these jobs, queries operate at a job level. They’re workers: they pick a job from the queue and execute the job, and return the results back to the query-frontend. Querier options control how many jobs it processes at once by opening concurrent connections to query-frontends, and each connection processes a single.

Guidelines on key configuration parameters

The following sections provide recommendations for key configuration parameters.

`query_frontend.max_outstanding_per_tenant` parameter

The query_frontend.max_outstanding_per_tenant parameter decides the maximum number of outstanding requests per tenant per frontend; requests beyond this error with HTTP 429.

This configuration controls how much load a single tenant can put on a cluster. A return of 429 indicates users to slow down.

This configuration can be used to maintain quality of service (QoS) for tenants by asking tenants to slow down by sending 429 to the tenant that’s overloading the system.

Guidelines

In a dedicated cluster with one big tenant, it’s okay to increase the number for query_frontend.max_outstanding_per_tenant.
In a shared cluster with a lot of small tenants, keep the number small.
If a single tenant is overwhelming the whole cluster, you should decrease this parameter. It reduces the amount of work the tenant can enqueue at once. It starts returning 429 to the tenant.
If your tenants are complaining about getting 429, you might want to increase this parameter and scale out queries to handle the query load.
If you increase other configurations that increase the query throughput or scale queriers, you might want to reduce this parameter to control the amount of work that can be enqueued.

`query_frontend.max_batch_size` parameter

The number of jobs to batch together in one HTTP request to the querier. Large search queries over a longer time range produce thousands of jobs. Instead of sending jobs one by one, batch the jobs and send them at once. The querier picks up and process them.

Batching pushes jobs faster to the queriers, and reduce the time spent waiting around for jobs to be picked up by the querier.

Guidelines

Default value of max_batch_size is set to 7.
We DO NOT recommend changing the batch size from the default value of 7. Based on testing at Grafana Labs, 7 is a good default.
If the batch size is big, you push more jobs at once but it takes longer for the querier to process the batch and return the results back.
Big batch size will increase the latency of the querier requests, and they might start hitting timeouts of 5xx, which will increase the rate of retries.
Bigger max_batch_size results in pushing too many jobs to the queriers. The jobs then have to be canceled if a query is exited early.

`query_frontend.search.concurrent_jobs` parameter

The number of concurrent jobs to execute when searching the backend. This controls how much work is produced concurrently.

To put it another way, if a search job is sharded into 5000 jobs, and concurrent_jobs is set to 1000, then Tempo only executes 1000 jobs concurrently. The other 4000 jobs are processed one by one as the first 1000 jobs return the responses.

If Tempo manages to answer the search query without executing all 5000 jobs, Tempo exits early and cancels the jobs that weren’t started.

Guidelines

This params controls the numbers of concurrent jobs to the number of jobs to put in the queue at once for processing a query.
If this number is low, big queries that create lots of jobs processed slowly.
If this number is high, the frontend pushes work faster to queries, and the frontend needs to cancel those jobs if the search exits early because the query is fulfilled.
If you see high load, spikes, or OOMs in the queriers, you can lower this configuration for a fair QoS and stability.
Having this number to a high value will allow a single tenant to push lots of work at once and overwhelm the querier. So keeping it to a lower number in a shared cluster allows for fair scheduling.
This parameter controls how fast jobs are pushed into the queue for queries to be picked up by queries
Adjusting this can impact your batch size so that metric should be watched while making changes to this configuration, and make sure that actual batch size is around the max_batch_size.
We recommend keeping it high in dedicated clusters with a single big tenant.
We recommend keeping it low in shared clusters because it ensures that a single tenant can’t overwhelm the read path.

`query_frontend.max_retries` parameter

This option controls the number of times to retry a request sent to a querier. We only retry on 5xx from a queriers (translated from the equivalent gRPC error).

Guidelines

This option controls the retries from query-frontend to queriers.
High max_retries amplifies the slow queriers and result in a lot more load then required.
Low max_retries can lead to users having bad experience due failed queriers, because a job failed and wasn’t retried.
If there are too many retries in a cluster, it can be a sign that queriers are struggling to finish work and needs to be scaled up.
Big jobs (high target_bytes_per_job or max_batch_size) can result in increased retries. Make the jobs smaller by reducing the target_bytes_per_job or max_batch_size to reduce the retries. If not, scale up the queriers.
Don’t set query_frontend.max_retries to a high number. The default of 2 is a good default.
Higher retries impact the backend when you have a big query that’s generating lots of jobs.
High max_retries can overwhelm the queriers when they’re under heavy load and failing to process the jobs. Retries can snowball, and degrade the query performance.

`query_frontend.search.target_bytes_per_job` parameter

The target number of bytes for each job to handle when performing a backend search.

This parameter controls the size of a single job, and size of a job dictates how much data a single job reads. You can tune this to make big or small jobs.

This option controls the upper limit on the size of a job, and can be used as a proxy for how much data a single search job scans or the amount of work a querier needs to do to process a job.

Guidelines

Setting this to a small value produces too many jobs, and results in more overhead, setting it too high produces big jobs. Queriers might struggle to finish those jobs and it can lead to high latency.
In testing at Grafana Labs, 100MB to 200MB is a good range for this configuration, and works across different sizes of clusters.
We recommend keeping this fixed within the recommended range.

`querier.frontend_worker.parallelism` parameter

Number of simultaneous queries to process per query-frontend or query-scheduler. This configuration controls the number of concurrent requests per query-frontend a querier process.

If parallelism is set to 5 and there are two query-frontends running, then a querier process opens 5 connections to each query-frontend, and in total of 10 connections.

One batch (or job, if batching is disabled) is processed per connection, so this controls the maximum number of concurrent jobs processed.

A single batch is processed in sync and size of the batch is controlled by max_batch_size, a connection is blocked until it’s process and returns the result of a batch.

You can also disable it and use max_concurrent_connections but we use this to make sure we queriers are picking up work from both query-frontends and jobs are being scheduled fairly.

If you want to make sure you always have connections defined in this config, you should set match_max_concurrent: false in your configuration to ensure you are not limited by max_concurrent_connections.

Guidelines

Maximum number of jobs a querier processes when using parallelism is equal to parallelism * query_frontend replicas * max_batch_size.
As you add more queriers, connections to an individual frontend increase. Query-frontend has a shared lock on these connections so if you see issues when you scale out queriers, lower the parallelism to reduce contention.
It’s recommended to set match_max_concurrent: false and not set max_concurrent_queries when using parallelism.

`querier.max_concurrent_queries` parameter

This controls the maximum number of jobs a querier processes concurrently. It doesn’t distinguish between the types of queries. These jobs can be from two different queries, or different tenants as well.

Querier processes these in sync, blocks the connection, and once it returns the results, it picks up a new job from the query-frontend queue.

Guidelines

The max_concurrent_queries parameter controls the number of queries that are processed by querier at once (concurrently).
if queriers are under-utilized, increase this value to process more jobs concurrently and process the queries faster.
If queriers are over utilized, seeing spikes, or OOMs, you can reduce this config to do less work per queriers and scale out queriers to add more capacity to the cluster.
This setting depends on the resources, in small queriers, it should be set to a lower number.
If you have large queriers, this configuration should be set to a higher value to fully utilize the capacity of the querier.

Querier memory sizing

If you are seeing that your queries are using more memory then you prefer, reduce the amount of work a querier is doing concurrently. That reduces resource usage.

If you want your queries to use more memory then they’re currently using, increase the amount of work a querier is doing concurrently and it increases the resource usage.

You can tune query_frontend.search.target_bytes_per_job, querier.frontend_worker.parallelism, and querier.max_concurrent_queries to tune the amount of work a querier is doing concurrently.

Queriers memory request roughly translates to job size times the concurrent work and some buffer.

Scaling cache

Tempo can be configured to use multiple caches for different types of data. When configured, Tempo uses these caches to improve the performance of queries.

Here are few general heuristics to know when to scale a cache:

Look at cache latency. If cache latency is hitting cache_timeout, that means that cache is under scaled and it’s taking too long to read or write, scale the cache.
Scale the cache if a cache has a high eviction rate. The cache might be under provisioned.
Lower level cache like bloom cache, parquet-page cache, parquet-footer cache sees higher hit rates (usually around 90% of above). If you have a consistent query traffic and these lower level caches have low hit rate, they’re undervalued and needs to be scaled up.
Higher level cache like frontend-search cache has a low hit rate and is only useful when the same query is being repeated. Size these according to the amount of data you want you want to cache
Cache sizes are also dictated by how much data you want to cache at each tier, it’s better to cache more at lower level caches because they have higher hit rate and is useful across queries.

Improve performance with caching

Thu, 28 May 2026 17:50:33 +0100

Improve performance with caching

Tempo uses an external cache to improve query performance. Tempo supports Memcached and Redis (experimental).

Cache roles

Tempo caches different types of data, each assigned a role. You configure one or more cache instances under the top-level cache: block and assign roles to each instance. This lets you size and tune each cache independently based on the workload it handles.

Role	What it caches	Volume
`bloom`	Bloom filters used for trace ID lookup.	Moderate
`trace-id-index`	Trace ID index used to locate traces within blocks.	Moderate
`parquet-footer`	Parquet file footer metadata. Useful for both search and trace-by-ID queries.	Low
`parquet-column-idx`	Parquet column index sections.	Low
`parquet-offset-idx`	Parquet offset index sections.	Low
`parquet-page`	Parquet data pages. Caches most Parquet reads.	High
`frontend-search`	Query-frontend search job results.	Varies

You can assign multiple roles to a single cache instance, or split high-volume roles (like parquet-page) onto a dedicated instance. For example, you might use a large Memcached pool for parquet-page and a smaller one for bloom and parquet-footer.

For configuration parameters and an example, refer to the Cache section of the Tempo configuration reference.

For information about search performance, refer to Tune search performance.

Memcached

Memcached is used by default in the Tanka and Helm examples. Refer to Deploy Tempo.

Connection limit

As a cluster grows in size, the number of instances of Tempo connecting to the cache servers also increases. By default, Memcached has a connection limit of 1024. Memcached refuses connections when this limit is surpassed. You can resolve this issue by increasing the connection limit of Memcached.

You can use the tempo_memcache_request_duration_seconds_count metric to observe these errors. For example, by using the following query:

sum by (status_code) (
  rate(tempo_memcache_request_duration_seconds_count{}[$__rate_interval])
)

This metric is also shown in the monitoring dashboards (the left panel):

Note that the already open connections continue to function. New connections are refused.

Additionally, Memcached logs the following errors when it can’t accept any new requests:

accept4(): No file descriptors available
Too many open connections
accept4(): No file descriptors available
Too many open connections

When using the memcached_exporter, you can observe the number of open connections at memcached_current_connections.

Cache size control

Tempo querier accesses bloom filters of all blocks while searching for a trace. This essentially mandates the size of cache to be at-least the total size of the bloom filters (the working set). However, in larger deployments, the working set might be larger than the desired size of cache. When that happens, eviction rates on the cache grow high, and hit rate drop.

Tempo provides two configuration parameters to filter down on the items stored in cache.

        # Min compaction level of block to qualify for caching bloom filter
        # Example: "cache_min_compaction_level: 2"
        [cache_min_compaction_level: <int>]

        # Max block age to qualify for caching bloom filter
        # Example: "cache_max_block_age: 48h"
        [cache_max_block_age: <duration>]

Using a combination of these configuration options, you can narrow down on which bloom filters are cached, thereby reducing the cache eviction rate, and increasing the cache hit rate.

In order to decide the values of these configuration parameters, you can use a cache summary command in the tempo-cli that prints a summary of bloom filter shards per day and per compaction level. The result looks something like this:

This image shows the bloom filter shards over 14 days and 6 compaction levels. This can be used to decide the configuration parameters.

Manage advanced systems

Thu, 28 May 2026 17:50:33 +0100

Manage advanced systems

This section provides resources for managing and tuning advanced Tempo configurations. For an overview of Tempo components and how they interact, refer to the Architecture topic.

Tempo CLI

Thu, 28 May 2026 17:50:33 +0100

Tempo CLI

Tempo CLI is a separate executable that contains utility functions related to the Tempo software. Although it’s not required for a working installation, Tempo CLI can be helpful for deeper analysis or for troubleshooting.

Tempo CLI command syntax

The general syntax for commands in Tempo CLI is:

tempo-cli command [subcommand] [options] [arguments...]

--help or -h displays the help for a command or subcommand.

Example:

tempo-cli -h
tempo-cli command [subcommand] -h

Run Tempo CLI

Tempo CLI is available as source code and as a Docker image.

Run with Docker (published image)

docker run --rm grafana/tempo-cli [arguments...]

Run directly with `go run`

go run ./cmd/tempo-cli [arguments...]

Build and run a local binary

To build a local binary, you need a working Go installation and build environment.

make tempo-cli
./bin/$(go env GOOS)/tempo-cli-$(go env GOARCH) [arguments...]

Build and run a local Docker image

make docker-tempo-cli
docker run --rm tempo-cli [arguments...]

Backend options

Tempo CLI connects directly to the storage backend for some commands, meaning that it requires the ability to read from S3, GCS, Azure or file-system storage. You can configure the backend using the following options:

Load an existing tempo configuration file using the --config-file (-c) option. This is the recommended option for frequent usage. Refer to Configuration documentation for more information.
Specify individual settings:
- --backend <value> The storage backend type, one of s3, gcs, azure, and local.
- --bucket <value> The bucket name. The meaning of this value is backend-specific. Refer to Configuration documentation for more information.
- --s3-endpoint <value> The S3 API endpoint (i.e. s3.dualstack.us-east-2.amazonaws.com).
- --s3-user <value>, --s3-pass <value> The S3 user name and password (or access key and secret key). Optional, as Tempo CLI supports the same authentication mechanisms as Tempo. Refer to S3 permissions documentation for more information.
- --insecure-skip-verify skip TLS verification, only applies to S3 and GCS.

Each option applies only to the command in which it’s used. For example, --backend <value> doesn’t permanently change where Tempo stores data. It only changes it for command in which you apply the option.

Query API command

Trace ID

Call the Tempo API and retrieve a trace by ID.

tempo-cli query api trace-id <api-endpoint> <trace-id>

Arguments:

api-endpoint URL for the Tempo API.
trace-id Trace ID as a hexadecimal string.

Options:

--org-id <value> Organization ID (for use in multi-tenant setup).
--header <key=value> Extra HTTP header to send with the request. Can be specified multiple times.
--v1 use v1 API (use /api/traces endpoint to fetch traces, default: /api/v2/traces).

Example:

tempo-cli query api trace-id http://tempo:3200 f1cfe82a8eef933b

Example with a custom authentication header:

tempo-cli query api trace-id http://tempo:3200 f1cfe82a8eef933b --header "X-TOKEN=<API_TOKEN>"

Replace <API_TOKEN> with your authentication token.

Search

Call the Tempo API and search using TraceQL.

tempo-cli query api search <host-port> <trace-ql> [<start> <end>]

Arguments:

host-port A host/port combination for Tempo. The scheme is inferred from the options.
trace-ql TraceQL query.
start Start of the time range to search in RFC3339 format (e.g. 2024-01-01T00:00:00Z) or relative (e.g. now-1h)
end End of the time range to search in RFC3339 format (e.g. 2024-01-01T01:00:00Z) or relative (e.g. now)

Options:

--org-id <value> Organization ID (for use in multi-tenant setup).
--header <key=value> Extra header to send with the request (as gRPC metadata when using --use-grpc). Can be specified multiple times.
--use-grpc Use GRPC streaming
--spss <value> Number of spans to return for each spanset
--limit <value> Number of results to return
--path-prefix <value> String to prefix search paths with
--secure Use HTTPS or gRPC with TLS

Note
Set the stream_over_http_enabled flag to true in the Tempo configuration to enable streaming over HTTP. For more information, refer to Tempo GRPC API documentation.

Example searching for error spans using relative time:

tempo-cli query api search localhost:3200 '{status = error}' now-1h now

Example searching for error spans using absolute time:

tempo-cli query api search localhost:3200 '{status = error}' 2024-01-01T00:00:00Z 2024-01-01T01:00:00Z

Example using GRPC streaming with organization ID:

tempo-cli query api search --use-grpc --org-id my-org localhost:3200 '{span.http.status_code >= 400}' now-1h now

Example with a custom authentication header over GRPC:

tempo-cli query api search --use-grpc --header "X-TOKEN=<API_TOKEN>" localhost:9095 '{status = error}' now-1h now

Search tags

Call the Tempo API and search attribute names.

tempo-cli query api search-tags <host-port> [<start> <end>]

Arguments:

host-port A host/port combination for Tempo. The scheme will be inferred based on the options provided.
start Start of the time range to search in RFC3339 format (e.g. 2024-01-01T00:00:00Z) or relative (e.g. now-1h)
end End of the time range to search in RFC3339 format (e.g. 2024-01-01T01:00:00Z) or relative (e.g. now)

Options:

--org-id <value> Organization ID (for use in multi-tenant setup).
--header <key=value> Extra header to send with the request (as gRPC metadata when using --use-grpc). Can be specified multiple times.
--use-grpc Use GRPC streaming
--path-prefix <value> String to prefix search paths with
--secure Use HTTPS or gRPC with TLS

Note
Set the stream_over_http_enabled flag to true in the Tempo configuration to enable streaming over HTTP. For more information, refer to Tempo GRPC API documentation.

Example:

tempo-cli query api search-tags localhost:3200

Example with relative time range:

tempo-cli query api search-tags localhost:3200 now-1h now

Example with absolute time range:

tempo-cli query api search-tags localhost:3200 2024-01-01T00:00:00Z 2024-01-02T00:00:00Z

Search tag values

Call the Tempo API and search attribute values.

tempo-cli query api search-tag-values <host-port> <tag> [<start> <end>]

Arguments:

host-port A host/port combination for Tempo. The scheme is inferred from the options.
tag The fully qualified TraceQL tag to search for. For example, resource.service.name.
start Start of the time range to search in RFC3339 format (e.g. 2024-01-01T00:00:00Z) or relative (e.g. now-1h)
end End of the time range to search in RFC3339 format (e.g. 2024-01-01T01:00:00Z) or relative (e.g. now)

Options:

--org-id <value> Organization ID (for use in multi-tenant setup).
--header <key=value> Extra header to send with the request (as gRPC metadata when using --use-grpc). Can be specified multiple times.
--query <value> TraceQL query to filter attribute results by.
--use-grpc Use GRPC streaming
--path-prefix <value> String to prefix search paths with
--secure Use HTTP or gRPC with TLS

Note
Set the stream_over_http_enabled flag to true in the Tempo configuration to enable streaming over HTTP. For more information, refer to Tempo GRPC API documentation.

Example to find all service names:

tempo-cli query api search-tag-values localhost:3200 resource.service.name

Example with query filter to find service names that have errors:

tempo-cli query api search-tag-values --query '{status = error}' localhost:3200 resource.service.name

Metrics

Call the Tempo API and generate metrics from traces using TraceQL.

tempo-cli query api metrics <host-port> <trace-ql metrics query> [<start> <end>]

Arguments:

host-port A host/port combination for Tempo. The scheme will be inferred based on the options provided.
trace-ql metrics query TraceQL metrics query.
start Start of the time range to search in RFC3339 format (e.g. 2024-01-01T00:00:00Z) or relative (e.g. now-1h)
end End of the time range to search in RFC3339 format (e.g. 2024-01-01T01:00:00Z) or relative (e.g. now)

Options:

--org-id <value> Organization ID (for use in multi-tenant setup).
--header <key=value> Extra header to send with the request (as gRPC metadata when using --use-grpc). Can be specified multiple times.
--use-grpc Use GRPC streaming
--instant Perform an instant query instead of a range query.
--path-prefix <value> String to prefix search paths with
--secure Use HTTPS or gRPC with TLS

Note
Set the stream_over_http_enabled flag to true in the Tempo configuration to enable streaming over HTTP. For more information, refer to Tempo GRPC API documentation.

Example range query for request rates by service:

tempo-cli query api metrics localhost:3200 '{} | rate() by (resource.service.name)' now-1h now

Example instant query for current error rates:

tempo-cli query api metrics --instant localhost:3200 '{status = error} | rate()' now-1h now

Query trace-id command

Iterate over all backend blocks and dump all data found for a given trace ID.

tempo-cli query trace-id <trace-id> <tenant-id>

Note
This can be intense as it downloads every bloom filter and some percentage of indexes/trace data.

Arguments:

trace-id Trace ID as a hexadecimal string.
tenant-id Tenant to search.

Options:

Backend options
--percentage <value> Percentage of blocks to scan (for example, 0.1 for 10%). Useful for sampling large datasets.

Example:

tempo-cli query trace-id f1cfe82a8eef933b single-tenant

Example scanning only 10% of blocks:

tempo-cli query trace-id --percentage 0.1 f1cfe82a8eef933b single-tenant

Query trace summary command

Iterate over all backend blocks and dump a summary for a given trace ID.

The summary includes:

number of blocks the trace is found in
span count
trace size
trace duration
root service name
root span info
top frequent service names

tempo-cli query trace-summary <trace-id> <tenant-id>

Note
This can be intense as it downloads every bloom filter and some percentage of indexes/trace data.

Arguments:

trace-id Trace ID as a hexadecimal string.
tenant-id Tenant to search.

Options:

Backend options
--percentage <value> Percentage of blocks to scan (for example, 0.1 for 10%). Useful for sampling large datasets.

Example:

tempo-cli query trace-summary f1cfe82a8eef933b single-tenant

List blocks

Lists information about all blocks for the given tenant, and optionally perform integrity checks on indexes for duplicate records.

tempo-cli list blocks <tenant-id>

Arguments:

tenant-id The tenant ID. Use single-tenant for single tenant setups.

Options:

--include-compacted Include blocks that have been compacted. Default behavior is to display only active blocks.

Output: Explanation of output:

ID Block ID.
Lvl Compaction level of the block.
Objects Number of objects stored in the block.
Size Data size of the block after any compression.
Vers Block version.
Window The window of time that was considered for compaction purposes.
Start The earliest timestamp stored in the block.
End The latest timestamp stored in the block.
Duration Duration between the start and end time.
Age The age of the block.
Cmp Whether the block has been compacted (present when –include-compacted is specified).

Example:

tempo-cli list blocks -c ./tempo.yaml single-tenant

List compaction summary

Summarizes information about all blocks for the given tenant based on compaction level. This command is useful to analyze or troubleshoot compaction behavior.

tempo-cli list compaction-summary <tenant-id>

Arguments:

tenant-id The tenant ID. Use single-tenant for single tenant setups.

Example:

tempo-cli list compaction-summary -c ./tempo.yaml single-tenant

List cache summary

Prints information about the number of bloom filter shards per day per compaction level. This command is useful to estimate and fine-tune cache storage. Read the caching topic for more information.

tempo-cli list cache-summary <tenant-id>

Arguments:

tenant-id The tenant ID. Use single-tenant for single tenant setups.

Example:

tempo-cli list cache-summary -c ./tempo.yaml single-tenant

List column

Lists values in a given column of a block. Useful for inspecting parquet block data directly.

tempo-cli list column <tenant-id> <block-id> [column-name]

Arguments:

tenant-id The tenant ID. Use single-tenant for single tenant setups.
block-id The block ID as UUID string.
column-name Column name to list values of (default: TraceID).

Options:

Backend options

Example:

tempo-cli list column -c ./tempo.yaml single-tenant ca314fba-efec-4852-ba3f-8d2b0bbf69f1 TraceID

View schema

View block metadata, parquet schema structure, and column sizes for a given block.

tempo-cli view schema <tenant-id> <block-id>

Arguments:

tenant-id The tenant ID. Use single-tenant for single tenant setups.
block-id The block ID as UUID string.

Options:

Backend options

The output includes:

Block metadata
Parquet schema structure
Column sizes in KB

Example:

tempo-cli view schema -c ./tempo.yaml single-tenant ca314fba-efec-4852-ba3f-8d2b0bbf69f1

Query search command

Search blocks in a given time range for a specific key/value pair.

tempo-cli query search <name> <value> <start> <end> <tenant-id>

Note
This can be intense as it downloads all relevant blocks and iterates through them.

Arguments:

name Name of the attribute to search for, for example, http.method.
value Value of the attribute to search for, for example, GET.
start Start of the time range to search in RFC3339 format (e.g. 2024-01-01T00:00:00Z) or relative (e.g. now-1h)
end End of the time range to search in RFC3339 format (e.g. 2024-01-01T01:00:00Z) or relative (e.g. now)
tenant-id Tenant to search.

Options:

Backend options

Example using relative time:

tempo-cli query search http.method GET now-1h now single-tenant --backend=gcs --bucket=tempo-trace-data

Example using absolute time:

tempo-cli query search http.method GET 2024-01-01T00:00:00Z 2024-01-01T00:05:00Z single-tenant --backend=gcs --bucket=tempo-trace-data

Parquet convert A to B command

Converts a vParquet file (actual data.parquet) of format A to a block of newer format B with an optional list of dedicated attribute columns. Actual supported versions for A and B vary by Tempo release. This utility command is useful when testing the impact of different combinations of dedicated columns.

Convert vParquet3 to vParquet4

Warning
vParquet3 is deprecated. Convert any remaining vParquet3 blocks to vParquet4 or later before upgrading to Tempo 3.0.

tempo-cli parquet convert-3to4 <in file> [<out path>] [<list of dedicated columns>]

Arguments:

in file Path to an existing vParquet3 block directory.
out path Path to write the vParquet4 block to. The default is ./out.
list of dedicated columns Optional list of columns to make dedicated. Columns use TraceQL syntax with scope. For example, span.db.statement, resource.namespace.

Example:

tempo-cli parquet convert-3to4 ./block-in ./out span.db.statement span.db.name

Convert vParquet4 to vParquet5

Converts a vParquet4 block to vParquet5 format with an optional list of dedicated attribute columns.

tempo-cli parquet convert-4to5 <in file> [<out path>] [<list of dedicated columns>]

Arguments:

in file Path to an existing vParquet4 block directory.
out path Path to write the vParquet5 block to. The default is ./out.
list of dedicated columns Optional list of columns to make dedicated. Columns use TraceQL syntax with scope. For example, span.http.method, resource.namespace, event.exception.message.

Column prefixes:

int/ marks the column as an integer type. For example, int/span.http.status_code.
blob/ marks the column for blob encoding. For example, blob/span.db.statement.

Example:

tempo-cli parquet convert-4to5 ./block-in ./block-out "span.http.method" "int/span.http.status_code" "blob/span.db.statement"

Migrate tenant command

Copy blocks from one backend and tenant to another. Blocks can be copied within the same backend or between two different backends. The data format isn’t converted but the tenant ID in meta.json is rewritten.

tempo-cli migrate tenant <source tenant> <dest tenant>

Arguments:

source tenant Tenant to copy blocks from
dest tenant Tenant to copy blocks into

Options:

--source-config-file <value> (required) Configuration file for the source backend.
--config-file <value> Configuration file for the destination backend.

Example:

tempo-cli migrate tenant --source-config-file source.yaml --config-file dest.yaml my-tenant my-other-tenant

Migrate overrides config command

Migrate the overrides section of a full Tempo config file from the legacy flat format to the new scoped format. The command reads the full config, converts any legacy overrides, and outputs only user-set values.

tempo-cli migrate overrides-config <config-file>

Arguments:

config-file Path to the full Tempo config file.

Options:

-d, --config-dest <path> Path to write the migrated overrides section. If not specified, output is printed to stdout.

Example:

tempo-cli migrate overrides-config config.yaml -d migrated-overrides.yaml

Warning

Fields set to Go zero values (false, 0, "") may be silently dropped due to omitempty tags. Compare against your original config to ensure nothing is lost.

Secret values (for example, remote_write_headers) are masked as <secret> in the output. You must manually restore the original values.

Some struct fields without omitempty may appear with zero values (for example, exclude: null) that were not in your original config.

Migrate overrides per-tenant command

Migrate a per-tenant overrides file from the legacy flat format to the new scoped format. The command handles both legacy and new format entries, and outputs only tenant-specific values.

tempo-cli migrate overrides-per-tenant <overrides-file>

Arguments:

overrides-file Path to the per-tenant overrides file.

Options:

-d, --output-dest <path> Path to write the migrated per-tenant overrides. If not specified, output is printed to stdout.

Example:

tempo-cli migrate overrides-per-tenant overrides.yaml -d migrated-overrides.yaml

Warning

Fields set to Go zero values (false, 0, "") may be silently dropped due to omitempty tags. Compare against your original config to ensure nothing is lost.

Secret values (for example, remote_write_headers) are masked as <secret> in the output. You must manually restore the original values.

Some struct fields without omitempty may appear with zero values (for example, exclude: null) that were not in your original config.

Migrate config command

Migrate a Tempo 2.x configuration file to a valid 3.0 configuration. The command removes obsolete configuration sections (such as ingester, ingester_client, and compactor), adds Kafka ingest configuration for microservices mode, disables compaction in overrides for parallel operation during migration, and strips the removed local-blocks metrics-generator processor.

The tool works at the YAML map level rather than rewriting the file from fully decoded Tempo structs, so environment variable references like ${VAR} are preserved. Top-level sections that are not recognized by the Tempo 3.0 configuration are dropped from the output. Unknown nested keys normally cause validation to fail, but validation is best-effort when the configuration contains ${VAR} placeholders where non-string types are expected, which may let unknown nested keys through.

tempo-cli migrate config [options] <config-file>

Arguments:

config-file Path to the 2.x Tempo configuration file.

Options:

--kafka-address <address> Kafka broker address. Required when running in microservices mode.
--kafka-topic <topic> Kafka topic name. Defaults to tempo.
--mode <monolithic|microservices> Override automatic deployment mode detection. By default, the mode is detected from the target field (all or absent means monolithic, any other value means microservices).

The migrated configuration is printed to stdout. Warnings are printed to stderr.

Examples

Monolithic mode (no Kafka flags needed):

tempo-cli migrate config old-config.yaml > new-config.yaml

Microservices mode:

tempo-cli migrate config --kafka-address=kafka:9092 --kafka-topic=tempo-traces old-config.yaml > new-config.yaml

Warning

The output is a starting point for your 3.0 configuration. Always review it before deploying.

If your configuration uses legacy (flat) overrides, you must run tempo-cli migrate overrides-config first.

If your configuration references an external per-tenant overrides file (per_tenant_override_config), you must manually add compaction_disabled: true for each tenant in that file.

YAML comments and key ordering from the original file are not preserved.

Remove compaction_disabled: true from overrides after fully decommissioning your 2.x deployment.

Analyse block

Analyses a block and outputs a summary of the block’s generic attributes.

It’s of particular use when trying to determine candidates for dedicated attribute columns in vParquet3+. The output includes span, resource, and event attributes with cardinality and size information.

Arguments:

tenant-id The tenant ID. Use single-tenant for single tenant setups.
block-id The block ID as UUID string.

Options:

Backend options
--num-attr <value> Number of attributes to output (default: 20)
--num-int-attr <value> Number of integer attributes to display. If set to 0, uses the --num-attr value (default: 5)
--blob-threshold <value> Mark attributes as blob candidates when their dictionary size per row group exceeds this value. Set to 0 to disable. (default: 4MiB)
--include-well-known Include well-known attributes in the analysis. Enable when generating dedicated columns for vParquet5 or higher. (default: false)
--generate-jsonnet Generate Jsonnet overrides for dedicated columns
--generate-cli-args Generate command-line arguments for the parquet conversion command
--int-percent-threshold <value> Threshold for integer attributes in dedicated columns (default: 0.05)
--str-percent-threshold <value> Threshold for string attributes in dedicated columns (default: 0.03)
--simple-summary Print only a single line of top attributes (default: false)
--print-full-summary Print full summary of the analysed block (default: true)

Example:

tempo-cli analyse block --backend=local --bucket=./cmd/tempo-cli/test-data/ single-tenant b18beca6-4d7f-4464-9f72-f343e688a4a0

Example with blob detection:

tempo-cli analyse block --blob-threshold=4MiB --generate-jsonnet --backend=local --bucket=./cmd/tempo-cli/test-data/ single-tenant b18beca6-4d7f-4464-9f72-f343e688a4a0

Analyse blocks

Analyses all blocks in a given time range and outputs a summary of the blocks’ generic attributes.

Arguments:

tenant-id The tenant ID. Use single-tenant for single-tenant setups.

Options:

Backend options
--num-attr <value> Number of attributes to output (default: 20)
--num-int-attr <value> Number of integer attributes to display. If set to 0, uses the --num-attr value (default: 5)
--min-compaction-level <value> Minimum compaction level to include in the analysis (default: 3)
--max-blocks <value> Maximum number of blocks to analyze (default: 10)
--max-start-time <value> Oldest start time for a block to be processed. RFC3339 format (default: disabled)
--min-start-time <value> Newest start time for a block to be processed. RFC3339 format (default: disabled)
--blob-threshold <value> Mark attributes as blob candidates when their dictionary size per row group exceeds this value. Set to 0 to disable. (default: 4MiB)
--include-well-known Include well-known attributes in the analysis. (default: false)
--jsonnet Generate Jsonnet overrides for dedicated columns
--cli Generate command-line arguments for the parquet conversion command
--int-percent-threshold <value> Threshold for integer attributes in dedicated columns (default: 0.05)
--str-percent-threshold <value> Threshold for string attributes in dedicated columns (default: 0.03)
--simple-summary Print only a single line of top attributes (default: false)
--print-full-summary Print full summary of the analysed block (default: true)

Example:

tempo-cli analyse blocks --backend=local --bucket=./cmd/tempo-cli/test-data/ single-tenant

Example with blob detection and Jsonnet output:

tempo-cli analyse blocks --blob-threshold=4MiB --jsonnet --backend=local --bucket=./cmd/tempo-cli/test-data/ single-tenant

Suggest columns

Suggests dedicated columns for a tenant based on analysis of blocks. This command analyzes block data and outputs configuration recommendations in YAML or Jsonnet format that can be used to configure dedicated attribute columns.

tempo-cli suggest columns <tenant-id>

Arguments:

tenant-id The tenant ID. Use single-tenant for single tenant setups.

Options:

Backend options
--block-id <value> Specific block ID to analyse. If not provided, analyzes multiple blocks.
--min-compaction-level <value> Minimum compaction level to analyse (default: 3)
--max-blocks <value> Maximum number of blocks to analyse (default: 10)
--num-attr <value> Number of attributes to display (default: 20)
--num-int-attr <value> Number of integer attributes to display. If set to 0, uses the --num-attr value (default: 5)
--int-percent-threshold <value> Threshold for integer attributes put in dedicated columns (default: 0.05)
--str-percent-threshold <value> Threshold for string attributes in dedicated columns (default: 0.03)
--include-well-known Include well-known attributes in the analysis (default: true)
--blob-threshold <value> Convert column to blob when dictionary size reaches this value (default: 4MiB)
--max-start-time <value> Oldest start time for a block to be processed. RFC3339 format.
--min-start-time <value> Newest start time for a block to be processed. RFC3339 format.
-o, --out <value> File to write output to. If not specified, output is printed to stdout.
-f, --format <jsonnet|yaml> Output format (default: yaml)

Example outputting YAML to a file:

tempo-cli suggest columns -c ./tempo.yaml --format yaml --out suggestions.yaml single-tenant

Example outputting Jsonnet for use in overrides:

tempo-cli suggest columns -c ./tempo.yaml --format jsonnet single-tenant

Generate attribute index

Warning
This command is EXPERIMENTAL and meant to facilitate experimentation with different kinds of indexes.

Generate an attribute index for a parquet block. This creates an index file that can be used for faster attribute lookups.

tempo-cli gen attr-index <input-path>

Arguments:

input-path Path to the input parquet block directory.

Options:

--add-intrinsics Add intrinsic attributes to the index such as name, kind, status, and others.
--index-types <rows|codes|rows,codes> Type of index to generate (default: rows,codes)

Example:

tempo-cli gen attr-index ./path/to/block

Example with intrinsic attributes:

tempo-cli gen attr-index --add-intrinsics ./path/to/block

Drop traces by ID

Rewrites all blocks for a tenant that contain specific trace IDs. The traces are dropped from the new blocks and the rewritten blocks are marked compacted so they will be cleaned up.

tempo-cli rewrite-blocks drop-traces <tenant-id> <trace-ids>

Arguments:

tenant-id The tenant ID. Use single-tenant for single tenant setups.
trace-ids The comma-separated trace IDs to drop (also supports single trace ID).

Options:

Backend options
--drop-trace By default, this command runs in dry run mode. Supplying this argument causes it to actually rewrite blocks with the traces dropped.
--background Run in background mode (default: false). Suppresses progress dots for use in automated scripts.

Examples

Dry run (default) to see which blocks would be affected:

tempo-cli rewrite-blocks drop-traces --backend=local --bucket=./cmd/tempo-cli/test-data/ single-tenant 04d5f549746c96e4f3daed6202571db2

Drop one trace (actually perform the operation):

tempo-cli rewrite-blocks drop-traces --drop-trace --backend=local --bucket=./cmd/tempo-cli/test-data/ single-tenant 04d5f549746c96e4f3daed6202571db2

Drop multiple traces:

tempo-cli rewrite-blocks drop-traces --drop-trace --backend=local --bucket=./cmd/tempo-cli/test-data/ single-tenant 04d5f549746c96e4f3daed6202571db2,111fa1850042aea83c17cd7e674210b8

Redact traces

Remove traces containing personally identifiable information or other sensitive data from object storage without waiting for retention to expire.

The redact command submits a redaction request to the backend scheduler. The scheduler creates jobs that rewrite affected blocks in object storage to remove the specified traces. Unlike drop-traces, which operates directly on object storage from the CLI, redact delegates the work to the backend scheduler over gRPC.

tempo-cli redact --tenant=<TENANT_ID> --trace-id=<TRACE_ID> [--trace-id=<TRACE_ID> ...] <scheduler-address>

Arguments:

scheduler-address The backend scheduler gRPC address (host:port).

Options:

--tenant <value> (required) Tenant ID.
--trace-id <value> (required) Trace ID to redact, in hex format. Specify multiple times to redact several traces in one request.
--tls Use TLS for the gRPC connection (default: false).
--tls-server-name <value> Override the TLS server name (SNI).
--tls-ca <value> Path to a PEM-encoded CA certificate file.

On success, the command prints the batch ID and the number of jobs created:

batch_id:     <BATCH_ID>
jobs_created: <COUNT>

Monitor job progress through the /status/backendscheduler endpoint.

Examples

Redact a single trace:

tempo-cli redact --tenant=my-tenant --trace-id=931281e2a09876de16e15f45ff86283d localhost:9095

Redact multiple traces in one request:

tempo-cli redact --tenant=my-tenant --trace-id=931281e2a09876de16e15f45ff86283d --trace-id=00000000000000000000000000000001 localhost:9095

With TLS and a custom CA:

tempo-cli redact --tenant=my-tenant --trace-id=931281e2a09876de16e15f45ff86283d --tls --tls-ca=/path/to/ca.pem scheduler.example.com:9095

Tempo Vulture

Thu, 28 May 2026 17:50:33 +0100

Tempo Vulture

Tempo Vulture is a testing tool that validates the end-to-end functionality of Tempo by continuously writing traces and verifying they can be read back correctly. It is useful for monitoring data integrity in production environments and for post-deployment validation testing.

Overview

Tempo Vulture performs the following operations:

Write traces: Pushes test traces to Tempo using OTLP over gRPC
Read traces by ID: Queries Tempo to retrieve traces by their trace ID
Search traces: Uses TraceQL to search for traces by attributes
Metrics queries: Validates TraceQL metrics queries (enabled by default)

Tempo Vulture can run in two modes:

Continuous mode (default): Runs indefinitely, continuously writing and reading traces while exposing Prometheus metrics
Validation mode: Executes a fixed number of write/read cycles and exits with a status code indicating success or failure

Run Tempo Vulture

Tempo Vulture is available as a Docker image:

docker run grafana/tempo-vulture:latest [arguments...]

You can also build it from source:

make tempo-vulture
./tempo-vulture [arguments...]

Configuration flags

Configure Tempo Vulture using command-line flags.

Required flags

Flag	Description	Example
`--tempo-query-url`	The URL (scheme://hostname) at which to query Tempo	`http://tempo:3200`
`--tempo-push-url`	The URL (scheme://hostname:port) at which to push traces to Tempo	`http://tempo:4317`

Optional flags

Flag	Default	Description
`--tempo-org-id`	`""`	The org ID to use when querying Tempo (for multi-tenant deployments)
`--tempo-push-tls`	`false`	Whether to use TLS when pushing spans to Tempo
`--tempo-write-backoff-duration`	`15s`	Time to pause between write operations
`--tempo-long-write-backoff-duration`	`1m`	Time to pause between long write operations
`--tempo-read-backoff-duration`	`30s`	Time to pause between read operations
`--tempo-search-backoff-duration`	`1m`	Time to pause between search operations. Set to `0s` to disable search validation
`--tempo-metrics-backoff-duration`	`10s`	Time to pause between TraceQL metrics operations. Set to `0s` to disable
`--tempo-retention-duration`	`336h`	The block retention that Tempo is using
`--tempo-recent-traces-backoff-duration`	`14m`	Cutoff between recent and old traces query checks
`--tempo-query-livestore`	`false`	Whether to query live stores
`--prometheus-listen-address`	`:80`	Address to listen on for Prometheus scrapes
`--prometheus-path`	`/metrics`	Path to publish Prometheus metrics

Validation mode flags

Flag	Default	Description
`--validation-mode`	`false`	Run in validation mode: execute a fixed number of cycles and exit
`--validation-cycles`	`3`	Number of write/read cycles to perform in validation mode
`--validation-timeout`	`5m`	Maximum time to run validation mode before timing out

Continuous mode

In continuous mode, Tempo Vulture runs indefinitely and exposes Prometheus metrics that can be used for monitoring and alerting.

Example

docker run grafana/tempo-vulture:latest \
  --tempo-query-url=http://tempo:3200 \
  --tempo-push-url=http://tempo:4317 \
  --tempo-org-id=my-tenant \
  --tempo-write-backoff-duration=15s \
  --tempo-read-backoff-duration=30s \
  --tempo-search-backoff-duration=60s

Prometheus metrics

Tempo Vulture exposes the following key metrics:

Metric	Type	Description
`tempo_vulture_trace_total`	Counter	Total number of trace operations performed
`tempo_vulture_trace_error_total`	Counter	Total number of trace errors by error type

Error types include:

incorrectresult: Retrieved trace doesn’t match the expected trace
incorrect_metrics_result: Metrics query returned unexpected results
missingspans: Trace has missing spans
notfound_byid: Trace not found by ID
notfound_search: Trace not found via search
notfound_traceql: Trace not found via TraceQL
notfound_metrics: Trace not found via metrics query
notfound_search_attribute: No searchable attribute found in trace
inaccurate_metrics: Metrics count doesn’t match actual span count
requestfailed: Request to Tempo failed

Alerting

You can configure alerts based on Tempo Vulture metrics. For example, the Tempo mixin includes a TempoVultureHighErrorRate alert that fires when the error rate exceeds a configurable threshold.

Validation mode

Note
Validation mode was added in Tempo 2.10.

Validation mode runs Tempo Vulture as a one-shot test that can be integrated into CI/CD pipelines for post-deployment validation. Instead of running continuously, it executes a fixed number of write/read cycles and exits with a status code indicating success or failure.

Warning
Running Vulture as a Deployment while in validation mode causes problems in Kubernetes. Because Deployments expect long-running processes, a container that finishes its task and exits triggers a CrashLoopBackOff (even with exit code 0). Run validation mode as a Job instead.

Exit codes

Exit code	Meaning
`0`	All validations passed
`1`	One or more validations failed, or configuration error

Environment variables

Validation mode requires authentication:

Variable	Description
`TEMPO_ACCESS_POLICY_TOKEN`	Access policy token for authenticating with Tempo (required in validation mode)

Example

docker run \
  -e TEMPO_ACCESS_POLICY_TOKEN=your-token \
  grafana/tempo-vulture:latest \
  --tempo-query-url=https://tempo.example.com \
  --tempo-push-url=https://tempo.example.com:4317 \
  --tempo-org-id=my-tenant \
  --validation-mode \
  --validation-cycles=5 \
  --validation-timeout=10m

CI/CD pipeline integration

Validation mode is useful for:

Post-deployment testing: Verify Tempo is functioning correctly after a deployment
Smoke testing: Quick validation that write and read paths are working
Integration testing: Ensure end-to-end trace pipeline is operational

Example CI/CD usage in a GitHub Actions workflow:

- name: Validate Tempo deployment
  run: |
    docker run \
      -e TEMPO_ACCESS_POLICY_TOKEN=${{ secrets.TEMPO_TOKEN }} \
      grafana/tempo-vulture:latest \
      --tempo-query-url=${{ vars.TEMPO_URL }} \
      --tempo-push-url=${{ vars.TEMPO_PUSH_URL }} \
      --tempo-org-id=${{ vars.TEMPO_ORG_ID }} \
      --validation-mode \
      --validation-cycles=3 \
      --validation-timeout=5m

Validation process

In validation mode, Tempo Vulture performs these steps:

For each cycle (default: 3):
- Write a test trace to Tempo
- Wait briefly for ingestion
- Read the trace back by ID and verify it matches
After all cycles complete:
- Wait for traces to be searchable (if search is enabled)
- Search for each trace using a random attribute
- Verify all traces are found
Exit with code 0 if all validations pass, or 1 if any fail

Docker Compose example

The Tempo repository includes Docker Compose examples that demonstrate how to run Tempo Vulture. Refer to the example configurations for complete setups.

Example from a Docker Compose file:

vulture:
  image: grafana/tempo-vulture:latest
  command:
    - "-tempo-push-url=http://distributor:4317"
    - "-tempo-query-url=http://query-frontend:3200"
    - "-tempo-org-id="

Apache Parquet schema

Thu, 28 May 2026 17:50:33 +0100

Apache Parquet schema

Apache Parquet is the only supported block format in Tempo 3.0. Refer to the Parquet configuration options for more information.

This document describes the schema used with the Parquet block format.

Version applicability

Tempo defaults to the vParquet4 schema. vParquet5 is production-ready and differs in some schema details. Unless otherwise noted, the sections below describe vParquet4.

The following sections apply to both vParquet4 and vParquet5:

Fully nested versus span-oriented schema
Static vs dynamic columns (see vParquet5 differences for changes to dedicated columns)
Compression and encoding
Bloom filters

Fully nested versus span-oriented schema

There are two overall approaches to a columnar schema: fully nested or span-oriented. Span-oriented means a flattened schema where traces are destructured into rows of spans. A fully nested schema means the current trace structures such as Resource/Scope/Spans/Events are preserved (nested data is natively supported in Parquet). In both cases, individual leaf values such as span name and duration are individual columns.

We chose the nested schema for several reasons:

The block size is much smaller for the nested schema. This is due to the high data duplication incurred when flattening resource-level attributes such as service.name to each individual span.
A flat schema is not truly “flat” because each span still contains nested data such as attributes and events.
Nested schema is much faster to search for resource-level attributes because the resource-level columns are very small (1 row for each batch).
Translation to and from the OpenTelemetry Protocol Specification (OTLP) is straightforward.
Easily add computed columns (for example, trace duration) at multiple levels such as per-trace, per-batch, etc.

Static vs dynamic columns

Dynamic vs static columns add another layer to the schema. A dynamic schema stores each attribute such as service.name and http.status_code as its own column and the columns in each parquet file can be different. A static schema is unresponsive to the shape of the data, and all attributes are stored in generic key/value containers.

The dynamic schema is the ultimate dream for a columnar format but it is too complex for a first release. However, the benefits of that approach are also too good to pass up, so we propose a hybrid approach. It is primarily a static schema but with some dynamic columns extracted from trace data based on some heuristics of frequently queried attributes. We plan to continue investing in this direction to implement a fully dynamic schema where trace attributes are blown out into independent Parquet columns at runtime.

For more information, refer to the Parquet design document.

Schema details

The adopted Parquet schema is mostly a direct translation of OTLP but with some key differences.

The table below uses these abbreviations:

rs - resource spans
ss - scope spans


Name	Type	Description
TraceID	byte array	The trace ID in 16-byte binary form.
TraceIDText	string	The trace ID in hexadecimal text form.
StartTimeUnixNano	int64	Start time of the first span in the trace, in nanoseconds since unix epoch.
EndTimeUnixNano	int64	End time of the last span in the trace, in nanoseconds since unix epoch.
DurationNano	int64	Total trace duration in nanoseconds, computed as difference between EndTimeUnixNano and StartTimeUnixNano.
RootServiceName	string	The resource-level `service.name` attribute (rs.Resource.ServiceName) from the root span of the trace if one exists, else empty string.
RootSpanName	string	The name (rs.ss.Spans.Name) of the root span if one exists, else empty string.
ServiceStats	map	Per-service counts keyed by service name. Values include span count and error count.
rs		Short-hand for ResourceSpans
rs.Resource.ServiceName	string	A dedicated column for the resource-level `service.name` attribute if present. https://opentelemetry.io/docs/reference/specification/resource/semantic_conventions/#service
rs.Resource.Cluster	string	A dedicated column for the resource-level `cluster` attribute if present and of string type. Values of other types will be stored in the generic attribute columns.
rs.Resource.Namespace	string	A dedicated column for the resource-level `namespace` attribute if present and of string type. Values of other types will be stored in the generic attribute columns.
rs.Resource.Pod	string	A dedicated column for the resource-level `pod` attribute if present and of string type. Values of other types will be stored in the generic attribute columns.
rs.Resource.Container	string	A dedicated column for the resource-level `container` attribute if present and of string type. Values of other types will be stored in the generic attribute columns.
rs.Resource.K8sClusterName	string	A dedicated column for the resource-level `k8s.cluster.name` attribute if present and of string type. Values of other types will be stored in the generic attribute columns. https://opentelemetry.io/docs/reference/specification/resource/semantic_conventions/k8s/#cluster
rs.Resource.K8sNamespaceName	string	A dedicated column for the resource-level `k8s.namespace.name` attribute if present and of string type. Values of other types will be stored in the generic attribute columns. https://opentelemetry.io/docs/reference/specification/resource/semantic_conventions/k8s/#namespace
rs.Resource.K8sPodName	string	A dedicated column for the resource-level `k8s.pod.name` attribute if present and of string type. Values of other types will be stored in the generic attribute columns. https://opentelemetry.io/docs/reference/specification/resource/semantic_conventions/k8s/#pod
rs.Resource.K8sContainerName	string	A dedicated column for the resource-level `k8s.container.name` attribute if present and of string type. Values of other types will be stored in the generic attribute columns. https://opentelemetry.io/docs/reference/specification/resource/semantic_conventions/k8s/#container
rs.Resource.DroppedAttributesCount	int	Number of resource attributes that were dropped.
rs.Resource.Attrs.Key	string	All resource attributes that do not have a dedicated column are stored as a key value pair in these columns. The Key column stores the name, and then one of the Value columns is populated according to the attribute’s data type. The other value columns will contain null.
rs.Resource.Attrs.IsArray	bool	Indicates if the attribute is stored as an array.
rs.Resource.Attrs.Value	string	The attribute value if string type (or array of strings), else null.
rs.Resource.Attrs.ValueInt	int	The attribute value if integer type (or array of integers), else null.
rs.Resource.Attrs.ValueDouble	float	The attribute value if float type (or array of floats), else null.
rs.Resource.Attrs.ValueBool	bool	The attribute value if boolean type (or array of booleans), else null.
rs.Resource.Attrs.ValueUnsupported	string	JSON-encoded AnyValue for unsupported or mixed-type values.
rs.Resource.DedicatedAttributes		Group containing spares for dedicated attribute columns with resource scope.
rs.Resource.DedicatedAttributes.String01 … String10	string	10 spare string columns for dedicated attributes (vParquet4). vParquet5 expands to String01 … String20 and adds Int01 … Int05.
rs.ss		Shorthand for ResourceSpans.ScopeSpans
rs.ss.Scope		Shorthand for ResourceSpans.ScopeSpans.Scope
rs.ss.Scope.Name	string	Scope name if present, else empty string. https://opentelemetry.io/docs/specs/otel/glossary/#instrumentation-scope
rs.ss.Scope.Version	string	The Scope version if present, else empty string. https://opentelemetry.io/docs/specs/otel/glossary/#instrumentation-scope
rs.ss.Scope.Attrs		Scope attributes, using the same columns as rs.Resource.Attrs.*
rs.ss.Scope.DroppedAttributesCount	int	Number of scope attributes that were dropped.
rs.ss.Spans.SpanID	byte array	Span unique ID.
rs.ss.Spans.ParentSpanID	byte array	The unique ID of the span’s parent. For root spans without a parent this is null.
rs.ss.Spans.ParentID	int32	Trace local numeric parent ID.
rs.ss.Spans.NestedSetLeft	int32	Left bound of the nested set model. Also used as a trace local numeric span ID.
rs.ss.Spans.NestedSetRight	int32	Right bound of the nested set model.
rs.ss.Spans.Name	string	Span name.
rs.ss.Spans.StartTimeUnixNano	int64	Start time the span in nanoseconds since unix epoch.
rs.ss.Spans.DurationNano	int64	Span duration in nanoseconds.
rs.ss.Spans.Kind	int	The span’s kind. Defined values: 0. Unset; 1. Internal; 2. Server; 3. Client; 4. Producer; 5. Consumer; https://opentelemetry.io/docs/reference/specification/trace/api/#spankind
rs.ss.Spans.StatusCode	int	The span status. Defined values: 0: Unset; 1: OK; 2: Error. https://opentelemetry.io/docs/reference/specification/trace/api/#set-status
rs.ss.Spans.StatusMessage	string	Optional message to accompany Error status.
rs.ss.Spans.HttpMethod	string	A dedicated column for the span-level `http.method` attribute if present and of string type, else null. Values of other types will be stored in the generic attribute columns. https://opentelemetry.io/docs/reference/specification/trace/semantic_conventions/http/#common-attributes
rs.ss.Spans.HttpStatusCode	int	A dedicated column for the span-level `http.status_code` attribute if present and of integer type, else null. Values of other types will be stored in the generic attribute columns. https://opentelemetry.io/docs/reference/specification/trace/semantic_conventions/http/#common-attributes
rs.ss.Spans.HttpUrl	string	A dedicated column for the span-level `http.url` attribute if present and of string type, else null. Values of other types will be stored in the generic attribute columns. https://opentelemetry.io/docs/reference/specification/trace/semantic_conventions/http/#http-client
rs.ss.Spans.DroppedAttributesCount	int	Number of attributes that were dropped
rs.ss.Spans.Attrs		Span attributes, using the same columns as rs.Resource.Attrs.*
rs.ss.Spans.DedicatedAttributes		Group containing spares for dedicated attribute columns with span scope
rs.ss.Spans.DedicatedAttributes.String01 … String10	string	10 spare string columns for dedicated attributes (vParquet4). vParquet5 expands to String01 … String20 and adds Int01 … Int05.
rs.ss.Spans.DroppedEventsCount	int	The number of events that were dropped
rs.ss.Spans.Events.TimeSinceStartNano	int64	The event timestamp in nanoseconds, relative to the span start time.
rs.ss.Spans.Events.Name	string	The event name or message.
rs.ss.Spans.Events.DroppedAttributesCount	int	The number of event attributes that were dropped.
rs.ss.Spans.Events.Attrs		Event attributes, using the same columns as rs.Resource.Attrs.*
rs.ss.Spans.DroppedLinksCount	int	The number of links that were dropped.
rs.ss.Spans.Links		Repeated link records with TraceID, SpanID, TraceState, Attrs, and DroppedAttributesCount.
rs.ss.Spans.TraceState	string	The span’s TraceState value if present, else empty string. https://opentelemetry.io/docs/reference/specification/trace/api/#tracestate

To increase the readability the table omits the groups list.element that are added for nested list types in Parquet. For maps (for example, ServiceStats), Parquet also inserts map-specific group levels that are omitted here.

message Trace {
  required binary TraceID;
  required binary TraceIDText (STRING);
  required int64 StartTimeUnixNano (INTEGER(64,false));
  required int64 EndTimeUnixNano (INTEGER(64,false));
  required int64 DurationNano (INTEGER(64,false));
  required binary RootServiceName (STRING);
  required binary RootSpanName (STRING);
  optional group ServiceStats (MAP) {
    repeated group key_value {
      required binary key (STRING);
      required group value {
        required int32 SpanCount (INTEGER(32,false));
        required int32 ErrorCount (INTEGER(32,false));
      }
    }
  }
  required group rs (LIST) {
    repeated group list {
      required group element {
        required group Resource {
          required group Attrs (LIST) {
            repeated group list {
              required group element {
                required binary Key (STRING);
                required boolean IsArray;
                required group Value (LIST) {
                  repeated group list {
                    required binary element (STRING);
                  }
                }
                required group ValueInt (LIST) {
                  repeated group list {
                    required int64 element (INTEGER(64,true));
                  }
                }
                required group ValueDouble (LIST) {
                  repeated group list {
                    required double element;
                  }
                }
                required group ValueBool (LIST) {
                  repeated group list {
                    required boolean element;
                  }
                }
                optional binary ValueUnsupported (STRING);
              }
            }
          }
          required int32 DroppedAttributesCount (INTEGER(32,true));
          required binary ServiceName (STRING);
          optional binary Cluster (STRING);
          optional binary Namespace (STRING);
          optional binary Pod (STRING);
          optional binary Container (STRING);
          optional binary K8sClusterName (STRING);
          optional binary K8sNamespaceName (STRING);
          optional binary K8sPodName (STRING);
          optional binary K8sContainerName (STRING);
          required group DedicatedAttributes {
            optional binary String01 (STRING);
            optional binary String02 (STRING);
            optional binary String03 (STRING);
            optional binary String04 (STRING);
            optional binary String05 (STRING);
            optional binary String06 (STRING);
            optional binary String07 (STRING);
            optional binary String08 (STRING);
            optional binary String09 (STRING);
            optional binary String10 (STRING);
          }
        }
        required group ss (LIST) {
          repeated group list {
            required group element {
              required group Scope {
                required binary Name (STRING);
                required binary Version (STRING);
                required group Attrs (LIST) {
                  repeated group list {
                    required group element {
                      required binary Key (STRING);
                      required boolean IsArray;
                      required group Value (LIST) {
                        repeated group list {
                          required binary element (STRING);
                        }
                      }
                      required group ValueInt (LIST) {
                        repeated group list {
                          required int64 element (INTEGER(64,true));
                        }
                      }
                      required group ValueDouble (LIST) {
                        repeated group list {
                          required double element;
                        }
                      }
                      required group ValueBool (LIST) {
                        repeated group list {
                          required boolean element;
                        }
                      }
                      optional binary ValueUnsupported (STRING);
                    }
                  }
                }
                required int32 DroppedAttributesCount (INTEGER(32,true));
              }
              required group Spans (LIST) {
                repeated group list {
                  required group element {
                    required binary SpanID;
                    required binary ParentSpanID;
                    required int32 ParentID (INTEGER(32,true));
                    required int32 NestedSetLeft (INTEGER(32,true));
                    required int32 NestedSetRight (INTEGER(32,true));
                    required binary Name (STRING);
                    required int64 Kind (INTEGER(64,true));
                    required binary TraceState (STRING);
                    required int64 StartTimeUnixNano (INTEGER(64,false));
                    required int64 DurationNano (INTEGER(64,false));
                    required int64 StatusCode (INTEGER(64,true));
                    required binary StatusMessage (STRING);
                    required group Attrs (LIST) {
                      repeated group list {
                        required group element {
                          required binary Key (STRING);
                          required boolean IsArray;
                          required group Value (LIST) {
                            repeated group list {
                              required binary element (STRING);
                            }
                          }
                          required group ValueInt (LIST) {
                            repeated group list {
                              required int64 element (INTEGER(64,true));
                            }
                          }
                          required group ValueDouble (LIST) {
                            repeated group list {
                              required double element;
                            }
                          }
                          required group ValueBool (LIST) {
                            repeated group list {
                              required boolean element;
                            }
                          }
                          optional binary ValueUnsupported (STRING);
                        }
                      }
                    }
                    required int32 DroppedAttributesCount (INTEGER(32,true));
                    required group Events (LIST) {
                      repeated group list {
                        required group element {
                          required int64 TimeSinceStartNano (INTEGER(64,false));
                          required binary Name (STRING);
                          required group Attrs (LIST) {
                            repeated group list {
                              required group element {
                                required binary Key (STRING);
                                required boolean IsArray;
                                required group Value (LIST) {
                                  repeated group list {
                                    required binary element (STRING);
                                  }
                                }
                                required group ValueInt (LIST) {
                                  repeated group list {
                                    required int64 element (INTEGER(64,true));
                                  }
                                }
                                required group ValueDouble (LIST) {
                                  repeated group list {
                                    required double element;
                                  }
                                }
                                required group ValueBool (LIST) {
                                  repeated group list {
                                    required boolean element;
                                  }
                                }
                                optional binary ValueUnsupported (STRING);
                              }
                            }
                          }
                          required int32 DroppedAttributesCount (INTEGER(32,true));
                        }
                      }
                    }
                    required int32 DroppedEventsCount (INTEGER(32,true));
                    required group Links (LIST) {
                      repeated group list {
                        required group element {
                          required binary TraceID;
                          required binary SpanID;
                          required binary TraceState (STRING);
                          required group Attrs (LIST) {
                            repeated group list {
                              required group element {
                                required binary Key (STRING);
                                required boolean IsArray;
                                required group Value (LIST) {
                                  repeated group list {
                                    required binary element (STRING);
                                  }
                                }
                                required group ValueInt (LIST) {
                                  repeated group list {
                                    required int64 element (INTEGER(64,true));
                                  }
                                }
                                required group ValueDouble (LIST) {
                                  repeated group list {
                                    required double element;
                                  }
                                }
                                required group ValueBool (LIST) {
                                  repeated group list {
                                    required boolean element;
                                  }
                                }
                                optional binary ValueUnsupported (STRING);
                              }
                            }
                          }
                          required int32 DroppedAttributesCount (INTEGER(32,true));
                        }
                      }
                    }
                    required int32 DroppedLinksCount (INTEGER(32,true));
                    optional binary HttpMethod (STRING);
                    optional binary HttpUrl (STRING);
                    optional int64 HttpStatusCode (INTEGER(64,true));
                    required group DedicatedAttributes {
                      optional binary String01 (STRING);
                      optional binary String02 (STRING);
                      optional binary String03 (STRING);
                      optional binary String04 (STRING);
                      optional binary String05 (STRING);
                      optional binary String06 (STRING);
                      optional binary String07 (STRING);
                      optional binary String08 (STRING);
                      optional binary String09 (STRING);
                      optional binary String10 (STRING);
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

For the authoritative schema, refer to tempodb/encoding/vparquet4/schema.go and tempodb/encoding/vparquet5/schema.go.

Summary of vParquet5 differences

Resource-level dedicated columns (Cluster/Namespace/Pod/Container/K8s*) and span HTTP columns are removed; vParquet5 relies on dynamically assigned dedicated columns only.
Dedicated attribute columns are doubled from 10 to 20 string spares per scope and add 5 integer spares per scope. Event scope is also supported. Array attributes and optional blob configuration are available for selected columns.
Additional fields exist for optimization, including span ChildCount, rounded start time buckets, and trace-level ServiceStats as a list with explicit service names.

Trace-level attributes

For speed and ease-of-use, we are projecting several values to columns at the trace-level:

Trace ID - Don’t store on each span.
Root service/span names/StartTimeUnixNano - These are selected properties of the root span in each trace (if there is one). These are used for displaying results in the Grafana UI. These properties are computed at ingest time and stored once for efficiency, so we don’t have to find the root span.
DurationNano - The total trace duration, computed at ingest time. This powers the min/max duration filtering in the current Tempo search and is more efficient than scanning the spans duration column. However, it may go away with TraceQL or we could decide to change it to span-level duration filtering too.
ServiceStats - Per-service span and error counts for each trace.

“Any”-type Attributes

OTLP attributes have variable data types, which is simpler in formats like protocol-buffers, but doesn’t translate directly to Parquet. Each column must have a concrete type. There are several possibilities here but we chose to have optional values for each concrete type. Unsupported or mixed-type values are stored as JSON-encoded AnyValue in ValueUnsupported.

Only scalar values are stored in the dedicated attribute columns. Arrays and unsupported types remain in the generic attribute columns.

repeated group Attrs {
  required binary Key (STRING);
  required boolean IsArray;
  repeated binary Value (STRING);
  repeated int64 ValueInt (INTEGER(64,true));
  repeated double ValueDouble;
  repeated boolean ValueBool;
  optional binary ValueUnsupported (STRING);
}

Compression and encoding

Parquet has robust support for many compression algorithms and data encodings. We’ve found excellent combinations of storage size and performance with the following:

Snappy Compression - Enable on all columns
Dictionary encoding - Enable on all string columns. Most strings are very repetitive so this works well to optimize storage size. However, you can greatly speed up search by inspecting the dictionary first and eliminating pages with no matches.
Time and duration UNIX nanos - Delta encoding
Rarely used columns such as DroppedAttributesCount - These columns are usually all zeroes, RLE works well.

Bloom filters

Parquet has native support for bloom filters. However, Tempo doesn’t use them at this time. Tempo already has sophisticated support for sharding and caching bloom filters.

Manage authentication

Thu, 28 May 2026 17:50:33 +0100

Manage authentication

Grafana Tempo does not come with any included authentication layer. You must run an authenticating reverse proxy in front of your services.

We recommend that in all deployment modes you add a reverse proxy in front of Tempo to direct client API requests to the various components.

A list of open-source reverse proxies you can use:

HAProxy
NGINX using their guide on restricting access with HTTP basic authentication
OAuth2 proxy
Pomerium, which has a guide for securing Grafana

Note
When using Tempo in multi-tenant mode, Tempo requires the HTTP header X-Scope-OrgID to be set to a string identifying the tenant. It’s assumed that clients setting X-Scope-OrgID are trusted clients, and the responsibility of populating this value should be handled by the authenticating reverse proxy. For more information, read the multi-tenancy documentation.

Manage Tempo on Grafana Labs

Compaction

Compaction

Block states

Blocklist and poller

Timing requirements

query_backend_after must be at least 2x blocklist_poll

compacted_block_retention must be at least 2x blocklist_poll

Block selection

How compaction works

Autoscaling

Configuration reference

Polling interval (storage.trace)

Compaction settings (backend_worker.compaction)

Query cutoff (must be >= 2x blocklist_poll)

Scheduler compaction (backend_scheduler.provider.compaction)

See also

Monitor Tempo

Monitor Tempo

Instrumentation

Metrics

Query frontend inspected bytes

Logs

Traces

Polling

Dashboards

Tempo Reads dashboard

Tempo Writes dashboard

Tempo Resources dashboard

Tempo Operational dashboard

Tempo Backend Work dashboard

Tempo Block Builder dashboard

Tempo Rollout Progress dashboard

Tempo Tenants dashboard

Rules and alerts

Manage trace ingestion

Manage trace ingestion

Size ingestion limits for your workload

Rate limit

Live trace limit

Per-trace size limit

Example configuration

Find and fix discarded spans

Error reference

Check why spans are being discarded

Log discarded spans for debugging

Traces missing from queries without errors

Identify what is driving ingestion volume

Set up cost attribution

Find the top contributors

Reduce volume from noisy services

Dedicated attribute columns

Dedicated attribute columns

Configuration

Usage

Integer attribute selection

Array-valued attributes

Blob attributes

Event-scoped attributes

Tempo-cli

Tune search performance

Tune search performance

Before you begin

Glossary

Tempo query path

Lifecycle of a query

General guidelines

Querier and query-frontend configuration

Querier

Query-frontend

Settings that are safe to increase without major impact.

Parameter interactions

Guidelines on key configuration parameters

query_frontend.max_outstanding_per_tenant parameter

Guidelines

query_frontend.max_batch_size parameter

Guidelines

query_frontend.search.concurrent_jobs parameter

Guidelines

query_frontend.max_retries parameter

Polling interval (`storage.trace`)

Compaction settings (`backend_worker.compaction`)

Scheduler compaction (`backend_scheduler.provider.compaction`)

`query_frontend.max_outstanding_per_tenant` parameter

`query_frontend.max_batch_size` parameter

`query_frontend.search.concurrent_jobs` parameter

`query_frontend.max_retries` parameter

`query_frontend.search.target_bytes_per_job` parameter

`querier.frontend_worker.parallelism` parameter

`querier.max_concurrent_queries` parameter

Run directly with `go run`