Configure Tempo on Grafana Labs

Tenant IDs

Thu, 28 May 2026 17:50:33 +0100

Tenant IDs

Within a Grafana Tempo cluster, the tenant ID is the unique identifier of a tenant. Tenant IDs enable multi-tenancy in Tempo, allowing multiple organizations or tenants to share the same Tempo cluster while maintaining data isolation.

Tempo uses the X-Scope-OrgID HTTP header to identify and enforce tenant boundaries. This header is set to the tenant ID value and is used for:

Scoped writes (ingest): Each span is stored under its specified tenant, ensuring data isolation at the storage level
Scoped reads (queries): Queries return only data belonging to the specified tenant

For more information about setting up multi-tenancy, refer to Enable multi-tenancy.

Characters and length restrictions

Tenant IDs must be less-than or equal-to 150 bytes in length. The length is measured in bytes, not characters, which is important for multi-byte characters.

Tenant IDs can only include the following supported characters:

Alphanumeric characters (0-9, a-z, A-Z)
Special characters (!, -, _, ., *, ', (, ))

All other characters, including slashes (/), backslashes (\), and whitespace, aren’t supported.

Tenant IDs are case-sensitive. For example, Tenant-123 and tenant-123 are treated as different tenant IDs.

The tenant ID should not be empty.

For security reasons, . and .. aren’t valid tenant IDs. These values are restricted to prevent path traversal attacks.

Cross-tenant queries

Cross-tenant queries are supported for search, search-tags, and trace-by-ID search operations. Refer to Cross-tenant query federation for more information.

Multitenancy

Tenant IDs are transmitted to Tempo via the X-Scope-OrgID HTTP header. This header must be included in all requests to Tempo when multi-tenancy is enabled.

Multi-tenancy on ingestion is supported with both gRPC and HTTP for OTLP (OpenTelemetry Protocol). You can add the header in:

OpenTelemetry Collector configuration
Grafana Alloy configuration
Any HTTP/gRPC client using curl or other relevant tools

Refer to Enable multi-tenancy for more information.

Manifest

Thu, 28 May 2026 17:50:33 +0100

Manifest

This document is a reference for all Tempo options and their defaults. If you are just getting started with Tempo, refer to Tempo examples and other configuration documentation. Most installations will require only setting 10 to 20 of these options.

Complete configuration

target: all
http_api_prefix: ""
memory:
    automemlimit_enabled: false
    automemlimit_ratio: 0.8
server:
    http_listen_network: tcp
    http_listen_address: ""
    http_listen_port: 3200
    http_listen_conn_limit: 0
    grpc_listen_network: tcp
    grpc_listen_address: ""
    grpc_listen_port: 9095
    grpc_listen_conn_limit: 0
    grpc_collect_max_streams_by_conn: true
    proxy_protocol_enabled: false
    tls_cipher_suites: ""
    tls_min_version: ""
    http_tls_config:
        cert: ""
        key: null
        client_ca: ""
        cert_file: ""
        key_file: ""
        client_auth_type: ""
        client_ca_file: ""
    grpc_tls_config:
        cert: ""
        key: null
        client_ca: ""
        cert_file: ""
        key_file: ""
        client_auth_type: ""
        client_ca_file: ""
    register_instrumentation: true
    report_grpc_codes_in_instrumentation_label_enabled: false
    graceful_shutdown_timeout: 30s
    http_server_read_timeout: 30s
    http_server_read_header_timeout: 0s
    http_server_write_timeout: 30s
    http_server_idle_timeout: 2m0s
    http_log_closed_connections_without_response_enabled: false
    grpc_server_max_recv_msg_size: 16777216
    grpc_server_max_send_msg_size: 16777216
    grpc_server_max_concurrent_streams: 100
    grpc_server_max_connection_idle: 2562047h47m16.854775807s
    grpc_server_max_connection_age: 2562047h47m16.854775807s
    grpc_server_max_connection_age_grace: 2562047h47m16.854775807s
    grpc_server_keepalive_time: 2h0m0s
    grpc_server_keepalive_timeout: 20s
    grpc_server_min_time_between_pings: 10s
    grpc_server_ping_without_stream_allowed: true
    grpc_server_num_workers: 0
    grpc_server_stats_tracking_enabled: true
    grpc_server_recv_buffer_pools_enabled: false
    grpc_server_read_buffer_size: 32768
    grpc_server_write_buffer_size: 32768
    log_format: logfmt
    log_level: info
    log_source_ips_enabled: false
    log_source_ips_full: false
    log_source_ips_header: ""
    log_source_ips_regex: ""
    log_request_headers: false
    log_request_at_info_level_enabled: false
    log_request_exclude_headers_list: ""
    trace_request_headers: false
    trace_request_exclude_headers_list: ""
    http_path_prefix: ""
    cluster_validation:
        label: ""
        additional_labels: ""
        grpc:
            enabled: false
            soft_validation: false
        http:
            enabled: false
            soft_validation: false
            excluded_paths: ""
            excluded_user_agents: ""
    create_new_traces: false
internal_server:
    http_listen_network: tcp
    http_listen_address: ""
    http_listen_port: 3101
    http_listen_conn_limit: 0
    grpc_listen_network: ""
    grpc_listen_address: ""
    grpc_listen_port: 0
    grpc_listen_conn_limit: 0
    grpc_collect_max_streams_by_conn: false
    proxy_protocol_enabled: false
    tls_cipher_suites: ""
    tls_min_version: ""
    http_tls_config:
        cert: ""
        key: null
        client_ca: ""
        cert_file: ""
        key_file: ""
        client_auth_type: ""
        client_ca_file: ""
    grpc_tls_config:
        cert: ""
        key: null
        client_ca: ""
        cert_file: ""
        key_file: ""
        client_auth_type: ""
        client_ca_file: ""
    register_instrumentation: false
    report_grpc_codes_in_instrumentation_label_enabled: false
    graceful_shutdown_timeout: 30s
    http_server_read_timeout: 30s
    http_server_read_header_timeout: 0s
    http_server_write_timeout: 30s
    http_server_idle_timeout: 2m0s
    http_log_closed_connections_without_response_enabled: false
    grpc_server_max_recv_msg_size: 0
    grpc_server_max_send_msg_size: 0
    grpc_server_max_concurrent_streams: 0
    grpc_server_max_connection_idle: 0s
    grpc_server_max_connection_age: 0s
    grpc_server_max_connection_age_grace: 0s
    grpc_server_keepalive_time: 0s
    grpc_server_keepalive_timeout: 0s
    grpc_server_min_time_between_pings: 0s
    grpc_server_ping_without_stream_allowed: false
    grpc_server_num_workers: 0
    grpc_server_stats_tracking_enabled: false
    grpc_server_recv_buffer_pools_enabled: false
    grpc_server_read_buffer_size: 0
    grpc_server_write_buffer_size: 0
    log_format: logfmt
    log_level: info
    log_source_ips_enabled: false
    log_source_ips_full: false
    log_source_ips_header: ""
    log_source_ips_regex: ""
    log_request_headers: false
    log_request_at_info_level_enabled: false
    log_request_exclude_headers_list: ""
    trace_request_headers: false
    trace_request_exclude_headers_list: ""
    http_path_prefix: ""
    cluster_validation:
        label: ""
        additional_labels: ""
        grpc:
            enabled: false
            soft_validation: false
        http:
            enabled: false
            soft_validation: false
            excluded_paths: ""
            excluded_user_agents: ""
    create_new_traces: false
    enable: false
distributor:
    ring:
        kvstore:
            store: memberlist
            prefix: collectors/
            consul:
                host: localhost:8500
                acl_token: ""
                http_client_timeout: 20s
                consistent_reads: false
                watch_rate_limit: 1
                watch_burst_size: 1
                cas_retry_delay: 1s
            etcd:
                endpoints: []
                dial_timeout: 10s
                max_retries: 10
                tls_enabled: false
                tls_cert_path: ""
                tls_key_path: ""
                tls_ca_path: ""
                tls_server_name: ""
                tls_insecure_skip_verify: false
                tls_cipher_suites: ""
                tls_min_version: ""
                username: ""
                password: ""
            multi:
                primary: ""
                secondary: ""
                mirror_enabled: false
                mirror_timeout: 2s
        heartbeat_period: 5s
        heartbeat_timeout: 5m0s
        instance_id: hostname
        instance_interface_names:
            - eth0
            - en0
        instance_port: 0
        instance_addr: ""
        enable_inet6: false
    receivers: {}
    override_ring_key: distributor
    forwarders: []
    usage:
        cost_attribution:
            max_cardinality: 10000
            stale_duration: 15m0s
    kafka_config:
        address: ""
        topic: ""
        client_id: ""
        dial_timeout: 0s
        write_timeout: 0s
        sasl_username: ""
        sasl_password: ""
        consumer_group: ""
        consumer_group_offset_commit_interval: 0s
        last_produced_offset_retry_timeout: 0s
        auto_create_topic_enabled: false
        auto_create_topic_default_partitions: 0
        producer_max_record_size_bytes: 0
        producer_max_buffered_bytes: 0
        target_consumer_lag_at_startup: 0s
        max_consumer_lag_at_startup: 0s
        disable_kafka_telemetry: false
        consumer_group_lag_metric_update_interval: 0s
    retry_after_on_resource_exhausted: 5s
    max_attribute_bytes: 2048
live_store_client:
    pool_config:
        checkinterval: 15s
        healthcheckenabled: true
        healthchecktimeout: 1s
        healthcheckgraceperiod: 0s
        maxconcurrenthealthchecks: 0
    remote_timeout: 5s
    grpc_client_config:
        max_recv_msg_size: 104857600
        max_send_msg_size: 104857600
        grpc_compression: snappy
        rate_limit: 0
        rate_limit_burst: 0
        backoff_on_ratelimits: false
        backoff_config:
            min_period: 100ms
            max_period: 10s
            max_retries: 10
        initial_stream_window_size: 63KiB1023B
        initial_connection_window_size: 63KiB1023B
        tls_enabled: false
        tls_cert_path: ""
        tls_key_path: ""
        tls_ca_path: ""
        tls_server_name: ""
        tls_insecure_skip_verify: false
        tls_cipher_suites: ""
        tls_min_version: ""
        connect_timeout: 5s
        connect_backoff_base_delay: 1s
        connect_backoff_max_delay: 5s
        cluster_validation:
            label: ""
querier:
    search:
        query_timeout: 30s
    trace_by_id:
        query_timeout: 10s
        external:
            endpoint: ""
            timeout: 10s
    metrics:
        concurrent_blocks: 2
        time_overlap_cutoff: 0.2
    partition_ring:
        minimize_requests: true
        hedging_delay: 3s
    max_concurrent_queries: 20
    frontend_worker:
        frontend_address: ""
        dns_lookup_duration: 10s
        parallelism: 2
        match_max_concurrent: true
        id: ""
        grpc_client_config:
            max_recv_msg_size: 104857600
            max_send_msg_size: 16777216
            grpc_compression: snappy
            rate_limit: 0
            rate_limit_burst: 0
            backoff_on_ratelimits: false
            backoff_config:
                min_period: 100ms
                max_period: 1s
                max_retries: 5
            initial_stream_window_size: 0B
            initial_connection_window_size: 0B
            tls_enabled: false
            tls_cert_path: ""
            tls_key_path: ""
            tls_ca_path: ""
            tls_server_name: ""
            tls_insecure_skip_verify: false
            tls_cipher_suites: ""
            tls_min_version: ""
            connect_timeout: 0s
            connect_backoff_base_delay: 0s
            connect_backoff_max_delay: 0s
            cluster_validation:
                label: ""
    shuffle_sharding_ingesters_enabled: false
    shuffle_sharding_ingesters_lookback_period: 1h0m0s
    query_relevant_ingesters: false
query_frontend:
    max_outstanding_per_tenant: 2000
    max_batch_size: 7
    log_query_request_headers: ""
    max_retries: 2
    search:
        concurrent_jobs: 1000
        target_bytes_per_job: 104857600
        default_result_limit: 20
        max_result_limit: 262144
        max_duration: 168h0m0s
        query_backend_after: 15m0s
        ingester_shards: 3
        most_recent_shards: 200
        default_spans_per_span_set: 3
        max_spans_per_span_set: 100
    trace_by_id:
        query_shards: 50
    metrics:
        concurrent_jobs: 1000
        target_bytes_per_job: 104857600
        max_duration: 24h0m0s
        query_backend_after: 15m0s
        interval: 5m0s
        max_exemplars: 100
        streaming_shards: 200
        max_intervals: 10000
    multi_tenant_queries_enabled: true
    response_consumers: 10
    weights:
        request_with_weights: true
        retry_with_weights: true
        max_traceql_conditions: 4
        max_regex_conditions: 1
    mcp_server:
        enabled: false
    max_grpc_streaming_packet_size: 2097152
    max_query_expression_size_bytes: 131072
    query_end_cutoff: 30s
metrics_generator:
    ring:
        kvstore:
            store: memberlist
            prefix: collectors/
            consul:
                host: localhost:8500
                acl_token: ""
                http_client_timeout: 20s
                consistent_reads: false
                watch_rate_limit: 1
                watch_burst_size: 1
                cas_retry_delay: 1s
            etcd:
                endpoints: []
                dial_timeout: 10s
                max_retries: 10
                tls_enabled: false
                tls_cert_path: ""
                tls_key_path: ""
                tls_ca_path: ""
                tls_server_name: ""
                tls_insecure_skip_verify: false
                tls_cipher_suites: ""
                tls_min_version: ""
                username: ""
                password: ""
            multi:
                primary: ""
                secondary: ""
                mirror_enabled: false
                mirror_timeout: 2s
        heartbeat_period: 5s
        heartbeat_timeout: 1m0s
        instance_id: hostname
        instance_interface_names:
            - eth0
            - en0
        instance_addr: ""
        instance_port: 0
        enable_inet6: false
        instance_zone: ""
    processor:
        service_graphs:
            wait: 10s
            max_items: 10000
            workers: 10
            histogram_buckets:
                - 0.1
                - 0.2
                - 0.4
                - 0.8
                - 1.6
                - 3.2
                - 6.4
                - 12.8
            dimensions: []
            enable_client_server_prefix: false
            enable_messaging_system_latency_histogram: false
            peer_attributes:
                - peer.service
                - db.name
                - db.system
            span_multiplier_key: ""
            enable_tracestate_span_multiplier: false
            enable_virtual_node_label: false
            database_name_attributes:
                - db.namespace
                - db.name
                - db.system
            filter_policies: []
        span_metrics:
            histogram_buckets:
                - 0.002
                - 0.004
                - 0.008
                - 0.016
                - 0.032
                - 0.064
                - 0.128
                - 0.256
                - 0.512
                - 1.024
                - 2.048
                - 4.096
                - 8.192
                - 16.384
            intrinsic_dimensions:
                service: true
                span_name: true
                span_kind: true
                status_code: true
            dimensions: []
            dimension_mappings: []
            enable_target_info: false
            span_multiplier_key: ""
            enable_tracestate_span_multiplier: false
            subprocessors:
                0: true
                1: true
                2: true
            filter_policies: []
            target_info_excluded_dimensions: []
            enable_instance_label: true
        host_info:
            host_identifiers:
                - k8s.node.name
                - host.id
            metric_name: traces_host_info
    registry:
        collection_interval: 15s
        stale_duration: 15m0s
        max_label_name_length: 1024
        max_label_value_length: 2048
    storage:
        path: ""
        wal:
            wal_segment_size: 134217728
            wal_compression: none
            stripe_size: 16384
            truncate_frequency: 2h0m0s
            min_wal_time: 300000
            max_wal_time: 14400000
            no_lockfile: false
        remote_write_flush_deadline: 1m0s
        remote_write_add_org_id_header: true
    metrics_ingestion_time_range_slack: 30s
    override_ring_key: metrics-generator
    ring_mode: partition
    codec: push-bytes
    limiter_type: series
    ingest_concurrency: 16
    instance_id: hostname
    leave_consumer_group_on_shutdown: false
ingest:
    kafka:
        address: localhost:9092
        topic: ""
        client_id: ""
        dial_timeout: 2s
        write_timeout: 10s
        sasl_username: ""
        sasl_password: ""
        consumer_group: ""
        consumer_group_offset_commit_interval: 1s
        last_produced_offset_retry_timeout: 10s
        auto_create_topic_enabled: true
        auto_create_topic_default_partitions: 1000
        producer_max_record_size_bytes: 15983616
        producer_max_buffered_bytes: 1073741824
        target_consumer_lag_at_startup: 2s
        max_consumer_lag_at_startup: 15s
        disable_kafka_telemetry: false
        consumer_group_lag_metric_update_interval: 1m0s
block_builder:
    instance_id: hostname
    assigned_partitions: {}
    partitions_per_instance: 0
    consume_cycle_duration: 5m0s
    max_consuming_bytes: 5000000000
    block:
        max_block_bytes: 20971520
    wal:
        path: /var/tempo/block-builder/traces
        ingestion_time_range_slack: 2m0s
storage:
    trace:
        pool:
            max_workers: 400
            queue_depth: 20000
        wal:
            path: /var/tempo/wal
            ingestion_time_range_slack: 2m0s
        block:
            bloom_filter_false_positive: 0.01
            bloom_filter_shard_size_bytes: 102400
            version: vParquet4
            parquet_row_group_size_bytes: 100000000
            parquet_dedicated_columns:
                - scope: resource
                  name: k8s.cluster.name
                  type: string
                  options: []
                - scope: resource
                  name: k8s.namespace.name
                  type: string
                  options: []
                - scope: resource
                  name: k8s.pod.name
                  type: string
                  options: []
                - scope: resource
                  name: k8s.container.name
                  type: string
                  options: []
                - scope: span
                  name: http.request.method
                  type: string
                  options: []
                - scope: span
                  name: http.response.status_code
                  type: int
                  options: []
                - scope: span
                  name: url.path
                  type: string
                  options: []
                - scope: span
                  name: url.route
                  type: string
                  options: []
                - scope: span
                  name: server.address
                  type: string
                  options: []
                - scope: span
                  name: server.port
                  type: int
                  options: []
                - scope: span
                  name: http.method
                  type: string
                  options: []
                - scope: span
                  name: http.url
                  type: string
                  options: []
                - scope: span
                  name: http.route
                  type: string
                  options: []
                - scope: span
                  name: http.status_code
                  type: int
                  options: []
        search:
            chunk_size_bytes: 1000000
            prefetch_trace_count: 1000
            read_buffer_count: 32
            read_buffer_size_bytes: 1048576
            cache_control:
                footer: false
                column_index: false
                offset_index: false
        blocklist_poll: 5m0s
        blocklist_poll_concurrency: 50
        blocklist_poll_tenant_concurrency: 0
        blocklist_poll_fallback: true
        blocklist_poll_tenant_index_builders: 2
        blocklist_poll_stale_tenant_index: 0s
        blocklist_poll_jitter_ms: 0
        blocklist_poll_tolerate_consecutive_errors: 1
        blocklist_poll_tolerate_tenant_failures: 1
        empty_tenant_deletion_enabled: false
        empty_tenant_deletion_age: 0s
        backend: ""
        local:
            path: ""
        gcs:
            bucket_name: ""
            prefix: ""
            chunk_buffer_size: 10485760
            endpoint: ""
            hedge_requests_at: 0s
            hedge_requests_up_to: 2
            insecure: false
            object_cache_control: ""
            object_metadata: {}
            list_blocks_concurrency: 3
            max_retries: 3
        s3:
            tls_cert_path: ""
            tls_key_path: ""
            tls_ca_path: ""
            tls_server_name: ""
            tls_insecure_skip_verify: false
            tls_cipher_suites: ""
            tls_min_version: VersionTLS12
            bucket: ""
            prefix: ""
            endpoint: ""
            region: ""
            access_key: ""
            secret_key: ""
            session_token: ""
            insecure: false
            part_size: 0
            hedge_requests_at: 0s
            hedge_requests_up_to: 2
            retry_max_attempts: 10
            retry_backoff_initial: 200ms
            retry_backoff_max: 1s
            signature_v2: false
            forcepathstyle: false
            enable_dual_stack: false
            bucket_lookup_type: 0
            tags: {}
            storage_class: ""
            metadata: {}
            native_aws_auth_enabled: false
            list_blocks_concurrency: 3
            sse:
                type: ""
                kms_key_id: ""
                kms_encryption_context: ""
                encryption_key: ""
        azure:
            storage_account_name: ""
            storage_account_key: ""
            use_managed_identity: false
            use_federated_token: false
            user_assigned_id: ""
            container_name: ""
            prefix: ""
            endpoint_suffix: blob.core.windows.net
            max_buffers: 4
            buffer_size: 3145728
            hedge_requests_at: 0s
            hedge_requests_up_to: 2
        cache: ""
        background_cache:
            writeback_goroutines: 10
            writeback_buffer: 10000
        memcached: null
        redis: null
        cache_min_compaction_level: 0
        cache_max_block_age: 0s
overrides:
    defaults:
        ingestion:
            rate_strategy: local
            rate_limit_bytes: 30000000
            burst_size_bytes: 30000000
            max_traces_per_user: 10000
            retry_info_enabled: true
        read:
            max_bytes_per_tag_values_query: 1000000
            max_condition_groups_per_tag_query: 100
        metrics_generator:
            generate_native_histograms: classic
            native_histogram_bucket_factor: 1.1
            native_histogram_max_bucket_number: 100
            native_histogram_min_reset_duration: 15m0s
        global:
            max_bytes_per_trace: 5000000
    per_tenant_override_config: ""
    per_tenant_override_period: 10s
    user_configurable_overrides:
        enabled: false
        poll_interval: 1m0s
        client:
            backend: ""
            confirm_versioning: true
            local:
                path: ""
            gcs:
                bucket_name: ""
                prefix: ""
                chunk_buffer_size: 10485760
                endpoint: ""
                hedge_requests_at: 0s
                hedge_requests_up_to: 2
                insecure: false
                object_cache_control: ""
                object_metadata: {}
                list_blocks_concurrency: 3
                max_retries: 3
            s3:
                tls_cert_path: ""
                tls_key_path: ""
                tls_ca_path: ""
                tls_server_name: ""
                tls_insecure_skip_verify: false
                tls_cipher_suites: ""
                tls_min_version: VersionTLS12
                bucket: ""
                prefix: ""
                endpoint: ""
                region: ""
                access_key: ""
                secret_key: ""
                session_token: ""
                insecure: false
                part_size: 0
                hedge_requests_at: 0s
                hedge_requests_up_to: 2
                retry_max_attempts: 10
                retry_backoff_initial: 200ms
                retry_backoff_max: 1s
                signature_v2: false
                forcepathstyle: false
                enable_dual_stack: false
                bucket_lookup_type: 0
                tags: {}
                storage_class: ""
                metadata: {}
                native_aws_auth_enabled: false
                list_blocks_concurrency: 3
                sse:
                    type: ""
                    kms_key_id: ""
                    kms_encryption_context: ""
                    encryption_key: ""
            azure:
                storage_account_name: ""
                storage_account_key: ""
                use_managed_identity: false
                use_federated_token: false
                user_assigned_id: ""
                container_name: ""
                prefix: ""
                endpoint_suffix: blob.core.windows.net
                max_buffers: 4
                buffer_size: 3145728
                hedge_requests_at: 0s
                hedge_requests_up_to: 2
        api:
            check_for_conflicting_runtime_overrides: false
    enable_legacy_overrides: false
memberlist:
    node_name: ""
    randomize_node_name: true
    stream_timeout: 2s
    retransmit_factor: 2
    pull_push_interval: 30s
    gossip_interval: 1s
    gossip_nodes: 2
    gossip_to_dead_nodes_time: 30s
    dead_node_reclaim_time: 0s
    compression_enabled: false
    notify_interval: 0s
    received_messages_queue_size: 1024
    advertise_addr: ""
    advertise_port: 7946
    cluster_label: ""
    cluster_label_verification_disabled: false
    join_members: ""
    min_join_backoff: 1s
    max_join_backoff: 1m0s
    max_join_retries: 10
    abort_if_cluster_fast_join_fails: false
    abort_if_cluster_fast_join_fails_min_nodes: 1
    abort_if_cluster_join_fails: false
    rejoin_interval: 0s
    rejoin_seed_nodes: ""
    left_ingesters_timeout: 5m0s
    obsolete_entries_timeout: 30s
    leave_timeout: 20s
    broadcast_timeout_for_local_updates_on_shutdown: 10s
    message_history_buffer_bytes: 0
    watch_prefix_buffer_size: 128
    bind_addr: []
    bind_port: 7946
    packet_dial_timeout: 2s
    packet_write_timeout: 5s
    max_concurrent_writes: 3
    acquire_writer_timeout: 250ms
    tls_enabled: false
    tls_cert_path: ""
    tls_key_path: ""
    tls_ca_path: ""
    tls_server_name: ""
    tls_insecure_skip_verify: false
    tls_cipher_suites: ""
    tls_min_version: ""
    zone_aware_routing:
        enabled: false
        instance_availability_zone: ""
        role: member
    propagation_delay_tracker:
        enabled: false
        beacon_interval: 1m0s
        beacon_lifetime: 10m0s
        log_beacons_latency_longer_than: 0s
usage_report:
    reporting_enabled: true
    backoff:
        min_period: 100ms
        max_period: 10s
        max_retries: 0
cache:
    background:
        writeback_goroutines: 10
        writeback_buffer: 10000
    caches: []
backend_scheduler:
    work:
        prune_age: 1h0m0s
        dead_job_timeout: 24h0m0s
    maintenance_interval: 1m0s
    backend_flush_interval: 1m0s
    provider:
        retention:
            interval: 1h0m0s
        compaction:
            measure_interval: 1m0s
            compaction:
                compaction_window: 1h0m0s
                max_compaction_objects: 6000000
                max_block_bytes: 107374182400
                block_retention: 336h0m0s
                compacted_block_retention: 1h0m0s
                retention_concurrency: 10
                max_time_per_tenant: 5m0s
                compaction_cycle: 30s
            max_jobs_per_tenant: 1000
            min_input_blocks: 2
            max_input_blocks: 4
            max_compaction_level: 0
            min_cycle_interval: 30s
        redaction:
            poll_interval: 2s
            rescan_delay: 5m0s
            max_rescan_generations: 5
    job_timeout: 15s
    local_work_path: /var/tempo
backend_scheduler_client:
    grpc_client_config:
        max_recv_msg_size: 104857600
        max_send_msg_size: 104857600
        grpc_compression: snappy
        rate_limit: 0
        rate_limit_burst: 0
        backoff_on_ratelimits: false
        backoff_config:
            min_period: 100ms
            max_period: 10s
            max_retries: 10
        initial_stream_window_size: 63KiB1023B
        initial_connection_window_size: 63KiB1023B
        tls_enabled: false
        tls_cert_path: ""
        tls_key_path: ""
        tls_ca_path: ""
        tls_server_name: ""
        tls_insecure_skip_verify: false
        tls_cipher_suites: ""
        tls_min_version: ""
        connect_timeout: 5s
        connect_backoff_base_delay: 1s
        connect_backoff_max_delay: 5s
        cluster_validation:
            label: ""
backend_worker:
    backend_scheduler_addr: ""
    backoff:
        min_period: 100ms
        max_period: 1m0s
        max_retries: 0
    compaction:
        compaction_window: 1h0m0s
        max_compaction_objects: 6000000
        max_block_bytes: 107374182400
        block_retention: 336h0m0s
        compacted_block_retention: 1h0m0s
        retention_concurrency: 10
        max_time_per_tenant: 5m0s
        compaction_cycle: 30s
    override_ring_key: backend-worker
    ring:
        kvstore:
            store: ""
            prefix: collectors/
            consul:
                host: localhost:8500
                acl_token: ""
                http_client_timeout: 20s
                consistent_reads: false
                watch_rate_limit: 1
                watch_burst_size: 1
                cas_retry_delay: 1s
            etcd:
                endpoints: []
                dial_timeout: 10s
                max_retries: 10
                tls_enabled: false
                tls_cert_path: ""
                tls_key_path: ""
                tls_ca_path: ""
                tls_server_name: ""
                tls_insecure_skip_verify: false
                tls_cipher_suites: ""
                tls_min_version: ""
                username: ""
                password: ""
            multi:
                primary: ""
                secondary: ""
                mirror_enabled: false
                mirror_timeout: 2s
        heartbeat_period: 5s
        heartbeat_timeout: 1m0s
        wait_stability_min_duration: 1m0s
        wait_stability_max_duration: 5m0s
        instance_id: hostname
        instance_interface_names:
            - eth0
            - en0
        instance_port: 0
        instance_addr: ""
        enable_inet6: false
        wait_active_instance_timeout: 10m0s
    finish_on_shutdown_timeout: 30s
live_store:
    ring:
        kvstore:
            store: memberlist
            prefix: collectors/
            consul:
                host: localhost:8500
                acl_token: ""
                http_client_timeout: 20s
                consistent_reads: false
                watch_rate_limit: 1
                watch_burst_size: 1
                cas_retry_delay: 1s
            etcd:
                endpoints: []
                dial_timeout: 10s
                max_retries: 10
                tls_enabled: false
                tls_cert_path: ""
                tls_key_path: ""
                tls_ca_path: ""
                tls_server_name: ""
                tls_insecure_skip_verify: false
                tls_cipher_suites: ""
                tls_min_version: ""
                username: ""
                password: ""
            multi:
                primary: ""
                secondary: ""
                mirror_enabled: false
                mirror_timeout: 2s
        heartbeat_period: 5s
        heartbeat_timeout: 1m0s
        instance_id: hostname
        instance_interface_names:
            - eth0
        instance_addr: ""
        instance_port: 0
        enable_inet6: false
        instance_zone: ""
    partition_ring:
        kvstore:
            store: memberlist
            prefix: collectors/
            consul:
                host: localhost:8500
                acl_token: ""
                http_client_timeout: 20s
                consistent_reads: false
                watch_rate_limit: 1
                watch_burst_size: 1
                cas_retry_delay: 1s
            etcd:
                endpoints: []
                dial_timeout: 10s
                max_retries: 10
                tls_enabled: false
                tls_cert_path: ""
                tls_key_path: ""
                tls_ca_path: ""
                tls_server_name: ""
                tls_insecure_skip_verify: false
                tls_cipher_suites: ""
                tls_min_version: ""
                username: ""
                password: ""
            multi:
                primary: ""
                secondary: ""
                mirror_enabled: false
                mirror_timeout: 2s
        min_partition_owners_count: 1
        min_partition_owners_duration: 10s
        delete_inactive_partition_after: 13h0m0s
    metrics:
        time_overlap_cutoff: 0.2
    commit_interval: 5s
    wal:
        path: /var/tempo/live-store/traces
        ingestion_time_range_slack: 2m0s
    query_block_concurrency: 10
    complete_block_timeout: 20m0s
    complete_block_concurrency: 2
    block_reclaim_grace: 2m0s
    shutdown_marker_dir: /var/tempo/live-store/shutdown-marker
    flush_check_period: 5s
    flush_op_timeout: 5m0s
    max_trace_live: 30s
    max_trace_idle: 5s
    max_live_traces_bytes: 250000000
    max_block_duration: 30s
    max_block_bytes: 52428800
    readiness_target_lag: 0s
    readiness_max_wait: 30m0s
    fail_on_high_lag: true
    remove_owner_on_shutdown: true

Apache Parquet block format

Thu, 28 May 2026 17:50:33 +0100

Apache Parquet block format

Tempo has a default columnar block format based on Apache Parquet. This format is required for tags-based search as well as TraceQL, the query language for traces. The columnar block format improves search performance and enables an ecosystem of tools, including Tempo CLI, to access the underlying trace data.

Considerations

The Parquet block format has been the default since Tempo 2.0 and is the only supported block format in Tempo 3.0.

If you install using the Tempo Helm charts, then Parquet is enabled by default. No data conversion or upgrade process is necessary. As soon as a block format version is enabled, Tempo starts writing data in that format, leaving existing data as-is.

Block format versions

Warning
The v2 and vParquet3 block formats have been removed in Tempo 3.0. Use vParquet4 (default) or vParquet5.

vParquet4 (default)

vParquet4 is the default block format in Tempo 3.0. It introduces columns that support querying array attributes, events, and links. For more information, refer to Dedicated attribute columns.

vParquet5

vParquet5 is production-ready and available as an opt-in alternative to vParquet4. It builds on vParquet4 with the following improvements:

Expanded dedicated columns: Up to 20 dedicated string columns and 5 dedicated integer columns per scope (span, resource, and event), compared with 10 string columns per scope in vParquet4.
Event-scoped dedicated columns: Dedicated attribute columns can target event-scoped attributes such as exception.message.
Blob column support: High-cardinality or high-length string attributes (for example, stack traces or UUIDs) can use zstd compression instead of dictionary encoding for better efficiency.
Array-valued dedicated columns: Dedicated columns can store multiple values per attribute using the options: ["array"] configuration.

For details on configuring dedicated attribute columns with vParquet5 features, refer to Dedicated attribute columns.

Change the block format version

To change the block format version, set the version option in the Storage section of the configuration file:

storage:
  trace:
    block:
      version: <version>

Replace <version> with vParquet4 or vParquet5.

To restore the default vParquet4 format, remove the version option from the configuration file or set it to vParquet4.

Parquet configuration parameters

Some parameters in the Tempo configuration are specific to Parquet. For more information, refer to the storage configuration documentation.

Trace search parameters

These configuration options impact trace search.

Parameter	Default value	Description
`[read_buffer_size_bytes: <int>]`	10485676	Size of read buffers used when performing search on a vParquet block. This value times the read_buffer_count is the total amount of bytes used for buffering when performing search on a Parquet block.
`[read_buffer_count: <int>]`	32	Number of read buffers used when performing search on a vParquet block. This value times the read_buffer_size_bytes is the total amount of bytes used for buffering when performing search on a Parquet block.

The cache_control section contains the follow parameters for Parquet metadata objects:

Parameter	Default value	Description
`[footer: \| default = false]`	`false`	Specifies if the footer should be cached
`[column_index: <bool> \| default = false]`	`false`	Specifies if the column index should be cached
`[offset_index: <bool> \| default = false]`	`false`	Specifies if the offset index should be cached

Hosted storage

Thu, 28 May 2026 17:50:33 +0100

Hosted storage

Tempo provides additional hosted storage configuration options discussed on the pages below. These options relate to providers such as Google Cloud, AWS S3, and Azure.

For additional details about storage configuration, refer to Storage configuration.

Amazon S3 and S3-compatible storage
Configure Tempo with Amazon S3 or S3-compatible object stores such as MinIO, SeaweedFS, or rclone.
Azure blob storage permissions and management
Azure blob storage permissions and configuration options for Tempo.
Google Cloud Storage
Learn about Google Cloud Storage permissions for Tempo.

Network options

Thu, 28 May 2026 17:50:33 +0100

Network options

Tempo provides additional network configuration options discussed on the pages below.

Configure IPv6
Learn how to configure IPv6 for Tempo.
Configure TLS communication
Configure TLS for Tempo server endpoints, gRPC clients, trace receivers, object storage, and caches to secure communication between components.
Run Tempo distributed with sidecar proxies
Configure Tempo distributed to run with sidecar proxies

Polling

Thu, 28 May 2026 17:50:33 +0100

Polling

The polling cycle is controlled by a number of configuration options detailed here.

storage:
    trace:
        # How often to repoll the backend for new blocks. Default is 5m
        [blocklist_poll: <duration>]

        # Number of blocks to process in parallel during polling. Default is 50.
        [blocklist_poll_concurrency: <int>]

        # By default components will pull the blocklist from the tenant index. If that fails the component can
        # fallback to scanning the entire bucket. Set to false to disable this behavior. Default is true.
        [blocklist_poll_fallback: <bool>]

        # Maximum number of workers that should build the tenant index. All other components will download
        # the index. Default 2.
        [blocklist_poll_tenant_index_builders: <int>]

        # The oldest allowable tenant index. If an index is pulled that is older than this duration,
        # the polling will consider this an error. Note that `blocklist_poll_fallback` applies here.
        # If fallback is true and a tenant index exceeds this duration, it will fall back to listing
        # the bucket contents.
        # Default 0 (disabled).
        [blocklist_poll_stale_tenant_index: <duration>]

Due to the mechanics of the tenant index, the blocklist will be stale by at most twice the configured blocklist_poll duration. This 2x staleness bound is the basis for the minimum values of query_backend_after and compacted_block_retention described below. For a detailed explanation of how compaction interacts with the polling cycle, refer to Compaction.

How blocks reach object storage

Block-builders consume trace data from Kafka and organize spans into blocks based on a configurable time window. Once a block is complete, the block-builder flushes it to object storage. After the block is written, queriers won’t discover it until their next polling cycle completes.

Handling blocklist staleness

Two mechanisms ensure data availability despite blocklist staleness:

Live-stores cover the recent data gap
Compacted block retention prevents premature deletion

Live-stores cover the recent data gap

Live-stores serve recent trace data directly from Kafka, covering the window between when a block-builder flushes a new block and when queriers discover it through polling. The query-frontend uses query_backend_after to control when backend storage is searched:

Time ranges more recent than query_backend_after are searched only in live-stores.
Time ranges older than query_backend_after are searched in backend storage.

query_backend_after must be set to at least twice the blocklist_poll duration. A newly flushed block will not appear in any querier’s blocklist for up to 2 * blocklist_poll after it is written. If query_backend_after is shorter than this window, the query-frontend may route requests to the backend before the block is discoverable, causing data gaps. With the default blocklist_poll of 5 minutes, query_backend_after should be at least 10 minutes.

query_frontend:
    search:
        # Time after which the query-frontend starts searching backend storage. Default is 15m.
        [query_backend_after: <duration>]

Compacted block retention prevents premature deletion

The compacted_block_retention keeps a block in object storage for a period of time after it has been compacted and the data has been merged into a new block. This allows queriers with a stale blocklist to still access these blocks until they complete their polling cycles and have up-to-date blocklists. At a minimum, this should be twice the configured blocklist_poll duration.

backend_worker:
  compaction:
    # How long to leave a block in the backend after it has been compacted successfully. Default is 1h
    [compacted_block_retention: <duration>]

Additionally, the querier blocklist_poll duration needs to be greater than or equal to the worker blocklist_poll duration. Otherwise, a querier may not correctly check all assigned blocks and incorrectly return 404. It is recommended to simply set both components to use the same poll duration.

Usage tracker

Thu, 28 May 2026 17:50:33 +0100

Usage tracker

The usage tracker accurately tracks the amount of ingested traffic using a set of custom labels on a per-tenant basis, providing fine-grained control over your data.

Use the cost_attributes option to configure usage trackers in the distributor, which expose metrics of ingested traffic grouped by configurable attributes exposed on /usage_metrics.

Enable the usage tracker

To use this feature, you need to enable it in the distributor and configure the overrides to handle the data.

Open your configuration file, for example, tempo.yaml.
In the distributor section, locate the cost_attribution line. If it is not present, then copy and paste the example below into your distributor section and update the options.

 usage:
        cost_attribution:
            # Enables the "cost-attribution" usage tracker. Per-tenant attributes are configured in overrides.
            [enabled: <boolean> | default = false]
            # Maximum number of series per tenant.
            [max_cardinality: <int> | default = 10000]
            # Interval after which a series is considered stale and will be deleted from the registry.
            # Once a metrics series is deleted, it won't be emitted anymore, keeping active series low.
            [stale_duration: <duration> | default = 15m0s]

Configure overrides

You also need to configure the dimensions to break down your the usage data in the standard overrides. In the overrides section, you can define attributes to group ingested data by and you can rename and combine attributes.

For more information, refer to the standard overrides section of the Configuration documentation.

In this example, usage data is grouped by service.name.

# Overrides configuration block
overrides:
  # Global ingestion limits configurations
  defaults:
    # Cost attribution usage tracker configuration
    cost_attribution:
      dimensions:
        service.name: ""

You can also configure per tenant in the runtime-overrides or in the user-configurable-overrides.

Review usage metrics

Once everything is configured, the usage metrics are exposed in the distributor:

GET /usage_metrics

Example:

curl http://localhost:3200/usage_metrics
# HELP tempo_usage_tracker_bytes_received_total bytes total received with these attributes
# TYPE tempo_usage_tracker_bytes_received_total counter
tempo_usage_tracker_bytes_received_total{service_name="article-service",tenant="single-tenant",tracker="cost-attribution"} 7327
tempo_usage_tracker_bytes_received_total{service_name="auth-service",tenant="single-tenant",tracker="cost-attribution"} 8938
tempo_usage_tracker_bytes_received_total{service_name="billing-service",tenant="single-tenant",tracker="cost-attribution"} 2401
tempo_usage_tracker_bytes_received_total{service_name="cart-service",tenant="single-tenant",tracker="cost-attribution"} 4116
tempo_usage_tracker_bytes_received_total{service_name="postgres",tenant="single-tenant",tracker="cost-attribution"} 3571
tempo_usage_tracker_bytes_received_total{service_name="shop-backend",tenant="single-tenant",tracker="cost-attribution"} 17619

You can configure Prometheus to scrape this endpoint to collect usage metrics.

Example:

scrape_configs:
  - job_name: 'tempo-usage'
    static_configs:
      - targets: ['<DISTRIBUTOR-HOST>:3200']
    metrics_path: '/usage_metrics'

Scope dimensions by resource or span

You can configure dimensions to look for attributes at specific scopes using prefixes:

resource.<attribute> - Look only at resource-level attributes
span.<attribute> - Look only at span-level attributes
<attribute> (no prefix) - Look at both resource and span levels

Example:

overrides:
  defaults:
    cost_attribution:
      dimensions:
        resource.service.name: "service"    # Scoped by resource, only look at `service.name` in resource attributes.
        span.db.system: "database"          # Scoped by span, only look at `db.system` in span attributes.
        k8s.namespace.name: "namespace"     # Unscoped, look at `k8s.namespace.name` in both resource and span attributes

Note
When using unscoped attributes (no prefix), span-level attributes will always overwrite resource-level attributes if both exist.

This can lead to mixed values from both scopes, use scope prefixes (resource. or span.) for predictable behavior when attributes exist at both and you only want values from one of them.

This table provides more information about the behavior:

Configuration	Resource has `service.name="api"`	Span has `service.name="worker"`	Result
`resource.service.name`	Uses `"api"`	Ignores span value	`"api"`
`span.service.name`	Ignores resource value	Uses `"worker"`	`"worker"`
`service.name`	Uses `"api"` initially	Overwrites with `"worker"`	`"worker"`

Configure target labels

You can customize the label names that appear in the /usage_metrics metrics by configuring a custom label name in the dimensions configuration.

dimensions:
  <ATTRIBUTE_NAME>: "<LABEL_NAME>"

Note
The _<LABEL_NAME>_ should be a valid Prometheus label name. If you provide an invalid label name, Tempo sanitizes it.

Example:

overrides:
  defaults:
    cost_attribution:
      dimensions:
        service.name: ""                    # Results in: service_name
        span.service.name: ""               # Results in: service_name
        resource.service.name: ""           # Results in: service_name
        span.db.system: ""                  # Results in: db_system
        span.db.system: "database.type"     # Results in: database_type
        k8s.namespace.name: "namespace"     # Results in: namespace

Use tracing data in Grafana

Thu, 28 May 2026 17:50:33 +0100

Use tracing data in Grafana

You can use Tempo as a data source in Grafana. Grafana Cloud comes pre-configured with a Tempo data source.

If you are using Grafana on-prem, you need to set up the Tempo data source.

Tip
If you want to explore tracing data in Grafana, try the Intro to Metrics, Logs, Traces, and Profiling example.

This video explains how to add data sources, including Loki, Tempo, and Mimir, to Grafana and Grafana Cloud. Tempo data source set up starts at 4:58 in the video.

Configure the data source

For detailed instructions on the Tempo data source in Grafana, refer to Tempo data source.

To configure Tempo with Grafana:

Point the Grafana data source at your Tempo query frontend (or monolithic mode Tempo).
Enter the URL: http://<tempo hostname>:<http port number>. For most of the Tempo examples the following works.

The port of 3200 is a common port used in our examples. Tempo default HTTP port is 3200.

Query the data source

Refer to Tempo in Grafana for an overview about how tracing data can be viewed and queried in Grafana.

For information on querying the Tempo data source, refer to Tempo query editor.

Anonymous usage reporting

Thu, 28 May 2026 17:50:33 +0100

Anonymous usage reporting

By default, Tempo reports anonymous usage data about itself to Grafana Labs. This data is used to understand which features are commonly enabled, as well as which deployment modes, replication factors, and compression levels are most popular, etc.

By providing information on how people use Tempo, usage reporting helps the Tempo team decide where to focus their development and documentation efforts.

The following configuration values are used:

Receivers enabled
Frontend concurrency and version
Storage cache, backend, WAL and block encodings
Ring replication factor, and kvstore
Features toggles enabled

No private information is collected, and all reports are completely anonymous.

Configure anonymous usage reporting

Reporting is controlled by the usage_report configuration option and can be disabled. For instructions, refer to the Configuration documentation.

View usage stats report

Tempo provides a usage stats report that you can view in your browser.

To view the report, go to the following URL on the Tempo instance: http://localhost:3200/status/usage-stats

An example report output looks like this:

{
  "clusterID": "",
  "createdAt": "0001-01-01T00:00:00Z",
  "interval": "0001-01-01T00:00:00Z",
  "intervalPeriod": 14400,
  "target": "all",
  "version": {
    "version": "v2.8.0",
    "revision": "31e2dddb5",
    "branch": "main",
    "buildUser": "",
    "buildDate": "",
    "goVersion": "go1.24.3"
  },
  "os": "linux",
  "arch": "arm64",
  "edition": "oss",
  "metrics": {
    "ring_kv_store": "inmemory",
    "memstats": {
      "heap_inuse": 14106624,
      "stack_inuse": 2490368,
      "pause_total_ns": 872084,
      "num_gc": 3,
      "gc_cpu_fraction": 0.08786506719453682,
      "heap_alloc": 11640400,
      "alloc": 11640400,
      "total_alloc": 16491760,
      "sys": 27874568
    },
    "num_cpu": 8,
    "feature_enabled_multitenancy": 0,
    "receiver_enabled_jaeger": 0,
    "storage_block_encoding": "zstd",
    "storage_block_search_encoding": "snappy",
    "storage_cache": "",
    "feature_enabled_auth_stats": 0,
    "frontend_version": "v1",
    "storage_backend": "local",
    "receiver_enabled_otlp": 0,
    "ring_replication_factor": 1,
    "storage_wal_encoding": "snappy",
    "storage_wal_search_encoding": "none",
    "num_goroutine": 813,
    "cache_memcached": 1,
    "cache_redis": 0,
    "distributor_bytes_received": {
      "total": 0,
      "rate": 0
    },
    "distributor_spans_received": {
      "total": 0,
      "rate": 0
    },
    "receiver_enabled_kafka": 0,
    "receiver_enabled_zipkin": 0
  }
}

Which information is collected?

Tempo collects and reports the following information to Grafana Labs. The report from your Tempo instance may vary from the provided example. Each field provides insight into the Tempo instance, its environment, and configuration. The fields are grouped by their purpose.

This information helps Grafana Labs understand how Tempo is used, which features are enabled, and the typical deployment environments, without collecting any private or user-identifying data.

Note
Tempo maintainers commit to keeping the list of tracked information updated over time, and reporting any change both via the CHANGELOG and the release notes.

Instance identification

clusterID: A unique, randomly generated identifier for the Tempo cluster. This value helps Grafana Labs distinguish between different deployments.
createdAt: The timestamp when anonymous usage reporting was first enabled and the cluster ID was created.
interval: The timestamp marking the start of the current reporting interval.
intervalPeriod: The length of the reporting interval, in seconds.

Deployment and version information

target: The deployment mode or target for the Tempo instance, such as all for monolithic mode.
version: An object containing detailed version information:
- version: The Tempo version, for example v2.8.0.
- revision: The Git commit hash or revision used to build the binary.
- branch: The Git branch used for the build.
- buildUser: The user who built the binary.
- buildDate: The date and time when the binary was built.
- goVersion: The Go language version used for the build.

Environment details

os: The operating system the Tempo instance is running on, such as linux.
arch: The system architecture, such as arm64.
edition: The edition of Tempo, such as oss for open source.

Metrics and configuration

metrics: An object containing runtime metrics and configuration:
- ring_kv_store: The key-value store used for the ring, for example inmemory.
- memstats: Memory usage statistics, including heap and stack usage, garbage collection metrics, and total allocations.
- num_cpu: The number of logical CPU cores available.
- feature_enabled_multitenancy: Indicates if multitenancy is enabled (1) or not (0).
- receiver_enabled_jaeger, receiver_enabled_otlp, receiver_enabled_kafka, receiver_enabled_zipkin: Flags indicating if each trace receiver is enabled (1) or not (0).
- storage_block_encoding, storage_block_search_encoding, storage_wal_encoding, storage_wal_search_encoding: The encoding or compression algorithms used for storage blocks and write-ahead logs.
- storage_cache: The cache backend used for storage, if any.
- feature_enabled_auth_stats: Indicates if authentication statistics are enabled.
- frontend_version: The version of the frontend component.
- storage_backend: The storage backend in use, such as local.
- ring_replication_factor: The replication factor for the ring.
- num_goroutine: The number of active Go routines.
- cache_memcached, cache_redis: Flags indicating if Memcached or Redis caching is enabled.
- distributor_bytes_received, distributor_spans_received: Objects showing the total and rate of bytes and spans received by the distributor.