Configure Tempo on Grafana Labs

Tenant IDs

Thu, 09 Apr 2026 14:59:14 +0000

Tenant IDs

Within a Grafana Tempo cluster, the tenant ID is the unique identifier of a tenant. Tenant IDs enable multi-tenancy in Tempo, allowing multiple organizations or tenants to share the same Tempo cluster while maintaining data isolation.

Tempo uses the X-Scope-OrgID HTTP header to identify and enforce tenant boundaries. This header is set to the tenant ID value and is used for:

Scoped writes (ingest): Each span is stored under its specified tenant, ensuring data isolation at the storage level
Scoped reads (queries): Queries return only data belonging to the specified tenant

For more information about setting up multi-tenancy, refer to Enable multi-tenancy.

Characters and length restrictions

Tenant IDs must be less-than or equal-to 150 bytes in length. The length is measured in bytes, not characters, which is important for multi-byte characters.

Tenant IDs can only include the following supported characters:

Alphanumeric characters (0-9, a-z, A-Z)
Special characters (!, -, _, ., *, ', (, ))

All other characters, including slashes (/), backslashes (\), and whitespace, aren’t supported.

Tenant IDs are case-sensitive. For example, Tenant-123 and tenant-123 are treated as different tenant IDs.

The tenant ID should not be empty.

For security reasons, . and .. aren’t valid tenant IDs. These values are restricted to prevent path traversal attacks.

Cross-tenant queries

Cross-tenant queries are supported for search, search-tags, and trace-by-ID search operations. Refer to Cross-tenant query federation for more information.

Multitenancy

Tenant IDs are transmitted to Tempo via the X-Scope-OrgID HTTP header. This header must be included in all requests to Tempo when multi-tenancy is enabled.

Multi-tenancy on ingestion is supported with both gRPC and HTTP for OTLP (OpenTelemetry Protocol). You can add the header in:

OpenTelemetry Collector configuration
Grafana Alloy configuration
Any HTTP/gRPC client using curl or other relevant tools

Refer to Enable multi-tenancy for more information.

Manifest

Thu, 09 Apr 2026 14:59:14 +0000

Manifest

This document is a reference for all Tempo options and their defaults. If you are just getting started with Tempo, refer to Tempo examples and other configuration documentation. Most installations will require only setting 10 to 20 of these options.

Complete configuration

target: all
http_api_prefix: ""
server:
    http_listen_network: tcp
    http_listen_address: ""
    http_listen_port: 3200
    http_listen_conn_limit: 0
    grpc_listen_network: tcp
    grpc_listen_address: ""
    grpc_listen_port: 9095
    grpc_listen_conn_limit: 0
    grpc_collect_max_streams_by_conn: true
    proxy_protocol_enabled: false
    tls_cipher_suites: ""
    tls_min_version: ""
    http_tls_config:
        cert: ""
        key: null
        client_ca: ""
        cert_file: ""
        key_file: ""
        client_auth_type: ""
        client_ca_file: ""
    grpc_tls_config:
        cert: ""
        key: null
        client_ca: ""
        cert_file: ""
        key_file: ""
        client_auth_type: ""
        client_ca_file: ""
    register_instrumentation: true
    report_grpc_codes_in_instrumentation_label_enabled: false
    graceful_shutdown_timeout: 30s
    http_server_read_timeout: 30s
    http_server_read_header_timeout: 0s
    http_server_write_timeout: 30s
    http_server_idle_timeout: 2m0s
    http_log_closed_connections_without_response_enabled: false
    grpc_server_max_recv_msg_size: 16777216
    grpc_server_max_send_msg_size: 16777216
    grpc_server_max_concurrent_streams: 100
    grpc_server_max_connection_idle: 2562047h47m16.854775807s
    grpc_server_max_connection_age: 2562047h47m16.854775807s
    grpc_server_max_connection_age_grace: 2562047h47m16.854775807s
    grpc_server_keepalive_time: 2h0m0s
    grpc_server_keepalive_timeout: 20s
    grpc_server_min_time_between_pings: 10s
    grpc_server_ping_without_stream_allowed: true
    grpc_server_num_workers: 0
    grpc_server_stats_tracking_enabled: true
    grpc_server_recv_buffer_pools_enabled: false
    log_format: logfmt
    log_level: info
    log_source_ips_enabled: false
    log_source_ips_full: false
    log_source_ips_header: ""
    log_source_ips_regex: ""
    log_request_headers: false
    log_request_at_info_level_enabled: false
    log_request_exclude_headers_list: ""
    trace_request_headers: false
    trace_request_exclude_headers_list: ""
    http_path_prefix: ""
    cluster_validation:
        label: ""
        additional_labels: ""
        grpc:
            enabled: false
            soft_validation: false
        http:
            enabled: false
            soft_validation: false
            excluded_paths: ""
            excluded_user_agents: ""
internal_server:
    http_listen_network: tcp
    http_listen_address: ""
    http_listen_port: 3101
    http_listen_conn_limit: 0
    grpc_listen_network: ""
    grpc_listen_address: ""
    grpc_listen_port: 0
    grpc_listen_conn_limit: 0
    grpc_collect_max_streams_by_conn: false
    proxy_protocol_enabled: false
    tls_cipher_suites: ""
    tls_min_version: ""
    http_tls_config:
        cert: ""
        key: null
        client_ca: ""
        cert_file: ""
        key_file: ""
        client_auth_type: ""
        client_ca_file: ""
    grpc_tls_config:
        cert: ""
        key: null
        client_ca: ""
        cert_file: ""
        key_file: ""
        client_auth_type: ""
        client_ca_file: ""
    register_instrumentation: false
    report_grpc_codes_in_instrumentation_label_enabled: false
    graceful_shutdown_timeout: 30s
    http_server_read_timeout: 30s
    http_server_read_header_timeout: 0s
    http_server_write_timeout: 30s
    http_server_idle_timeout: 2m0s
    http_log_closed_connections_without_response_enabled: false
    grpc_server_max_recv_msg_size: 0
    grpc_server_max_send_msg_size: 0
    grpc_server_max_concurrent_streams: 0
    grpc_server_max_connection_idle: 0s
    grpc_server_max_connection_age: 0s
    grpc_server_max_connection_age_grace: 0s
    grpc_server_keepalive_time: 0s
    grpc_server_keepalive_timeout: 0s
    grpc_server_min_time_between_pings: 0s
    grpc_server_ping_without_stream_allowed: false
    grpc_server_num_workers: 0
    grpc_server_stats_tracking_enabled: false
    grpc_server_recv_buffer_pools_enabled: false
    log_format: logfmt
    log_level: info
    log_source_ips_enabled: false
    log_source_ips_full: false
    log_source_ips_header: ""
    log_source_ips_regex: ""
    log_request_headers: false
    log_request_at_info_level_enabled: false
    log_request_exclude_headers_list: ""
    trace_request_headers: false
    trace_request_exclude_headers_list: ""
    http_path_prefix: ""
    cluster_validation:
        label: ""
        additional_labels: ""
        grpc:
            enabled: false
            soft_validation: false
        http:
            enabled: false
            soft_validation: false
            excluded_paths: ""
            excluded_user_agents: ""
    enable: false
distributor:
    ring:
        kvstore:
            store: memberlist
            prefix: collectors/
            consul:
                host: localhost:8500
                acl_token: ""
                http_client_timeout: 20s
                consistent_reads: false
                watch_rate_limit: 1
                watch_burst_size: 1
                cas_retry_delay: 1s
            etcd:
                endpoints: []
                dial_timeout: 10s
                max_retries: 10
                tls_enabled: false
                tls_cert_path: ""
                tls_key_path: ""
                tls_ca_path: ""
                tls_server_name: ""
                tls_insecure_skip_verify: false
                tls_cipher_suites: ""
                tls_min_version: ""
                username: ""
                password: ""
            multi:
                primary: ""
                secondary: ""
                mirror_enabled: false
                mirror_timeout: 2s
        heartbeat_period: 5s
        heartbeat_timeout: 5m0s
        instance_id: hostname
        instance_interface_names:
            - eth0
            - en0
        instance_port: 0
        instance_addr: ""
        enable_inet6: false
    receivers: {}
    override_ring_key: distributor
    forwarders: []
    usage:
        cost_attribution:
            max_cardinality: 10000
            stale_duration: 15m0s
    ingester_write_path_enabled: true
    kafka_write_path_enabled: false
    kafka_config:
        address: ""
        topic: ""
        client_id: ""
        dial_timeout: 0s
        write_timeout: 0s
        sasl_username: ""
        sasl_password: ""
        consumer_group: ""
        consumer_group_offset_commit_interval: 0s
        last_produced_offset_retry_timeout: 0s
        auto_create_topic_enabled: false
        auto_create_topic_default_partitions: 0
        producer_max_record_size_bytes: 0
        producer_max_buffered_bytes: 0
        target_consumer_lag_at_startup: 0s
        max_consumer_lag_at_startup: 0s
        disable_kafka_telemetry: false
        consumer_group_lag_metric_update_interval: 0s
    extend_writes: true
    retry_after_on_resource_exhausted: 0s
    max_attribute_bytes: 2048
ingester_client:
    pool_config:
        checkinterval: 15s
        healthcheckenabled: true
        healthchecktimeout: 1s
        maxconcurrenthealthchecks: 0
    remote_timeout: 5s
    grpc_client_config:
        max_recv_msg_size: 104857600
        max_send_msg_size: 104857600
        grpc_compression: snappy
        rate_limit: 0
        rate_limit_burst: 0
        backoff_on_ratelimits: false
        backoff_config:
            min_period: 100ms
            max_period: 10s
            max_retries: 10
        initial_stream_window_size: 63KiB1023B
        initial_connection_window_size: 63KiB1023B
        tls_enabled: false
        tls_cert_path: ""
        tls_key_path: ""
        tls_ca_path: ""
        tls_server_name: ""
        tls_insecure_skip_verify: false
        tls_cipher_suites: ""
        tls_min_version: ""
        connect_timeout: 5s
        connect_backoff_base_delay: 1s
        connect_backoff_max_delay: 5s
        cluster_validation:
            label: ""
metrics_generator_client:
    pool_config:
        checkinterval: 15s
        healthcheckenabled: true
        healthchecktimeout: 1s
        maxconcurrenthealthchecks: 0
    remote_timeout: 5s
    grpc_client_config:
        max_recv_msg_size: 104857600
        max_send_msg_size: 104857600
        grpc_compression: snappy
        rate_limit: 0
        rate_limit_burst: 0
        backoff_on_ratelimits: false
        backoff_config:
            min_period: 100ms
            max_period: 10s
            max_retries: 10
        initial_stream_window_size: 63KiB1023B
        initial_connection_window_size: 63KiB1023B
        tls_enabled: false
        tls_cert_path: ""
        tls_key_path: ""
        tls_ca_path: ""
        tls_server_name: ""
        tls_insecure_skip_verify: false
        tls_cipher_suites: ""
        tls_min_version: ""
        connect_timeout: 5s
        connect_backoff_base_delay: 1s
        connect_backoff_max_delay: 5s
        cluster_validation:
            label: ""
live_store_client:
    pool_config:
        checkinterval: 15s
        healthcheckenabled: true
        healthchecktimeout: 1s
        maxconcurrenthealthchecks: 0
    remote_timeout: 5s
    grpc_client_config:
        max_recv_msg_size: 104857600
        max_send_msg_size: 104857600
        grpc_compression: snappy
        rate_limit: 0
        rate_limit_burst: 0
        backoff_on_ratelimits: false
        backoff_config:
            min_period: 100ms
            max_period: 10s
            max_retries: 10
        initial_stream_window_size: 63KiB1023B
        initial_connection_window_size: 63KiB1023B
        tls_enabled: false
        tls_cert_path: ""
        tls_key_path: ""
        tls_ca_path: ""
        tls_server_name: ""
        tls_insecure_skip_verify: false
        tls_cipher_suites: ""
        tls_min_version: ""
        connect_timeout: 5s
        connect_backoff_base_delay: 1s
        connect_backoff_max_delay: 5s
        cluster_validation:
            label: ""
querier:
    search:
        query_timeout: 30s
    trace_by_id:
        query_timeout: 10s
        external:
            endpoint: ""
            timeout: 10s
    metrics:
        concurrent_blocks: 2
        time_overlap_cutoff: 0.2
    partition_ring:
        minimize_requests: true
        hedging_delay: 3s
    max_concurrent_queries: 20
    frontend_worker:
        frontend_address: ""
        dns_lookup_duration: 10s
        parallelism: 2
        match_max_concurrent: true
        id: ""
        grpc_client_config:
            max_recv_msg_size: 104857600
            max_send_msg_size: 16777216
            grpc_compression: snappy
            rate_limit: 0
            rate_limit_burst: 0
            backoff_on_ratelimits: false
            backoff_config:
                min_period: 100ms
                max_period: 1s
                max_retries: 5
            initial_stream_window_size: 0B
            initial_connection_window_size: 0B
            tls_enabled: false
            tls_cert_path: ""
            tls_key_path: ""
            tls_ca_path: ""
            tls_server_name: ""
            tls_insecure_skip_verify: false
            tls_cipher_suites: ""
            tls_min_version: ""
            connect_timeout: 0s
            connect_backoff_base_delay: 0s
            connect_backoff_max_delay: 0s
            cluster_validation:
                label: ""
    shuffle_sharding_ingesters_enabled: false
    shuffle_sharding_ingesters_lookback_period: 1h0m0s
    query_relevant_ingesters: false
query_frontend:
    max_outstanding_per_tenant: 2000
    max_batch_size: 7
    log_query_request_headers: ""
    max_retries: 2
    search:
        concurrent_jobs: 1000
        target_bytes_per_job: 104857600
        default_result_limit: 20
        max_result_limit: 262144
        max_duration: 168h0m0s
        query_backend_after: 15m0s
        query_ingesters_until: 30m0s
        ingester_shards: 3
        most_recent_shards: 200
        default_spans_per_span_set: 3
        max_spans_per_span_set: 100
    trace_by_id:
        query_shards: 50
    metrics:
        concurrent_jobs: 1000
        target_bytes_per_job: 104857600
        max_duration: 3h0m0s
        query_backend_after: 30m0s
        interval: 5m0s
        max_exemplars: 100
        streaming_shards: 200
        max_intervals: 10000
    multi_tenant_queries_enabled: true
    response_consumers: 10
    weights:
        request_with_weights: true
        retry_with_weights: true
        max_traceql_conditions: 4
        max_regex_conditions: 1
    mcp_server:
        enabled: false
    max_query_expression_size_bytes: 131072
    rf1_after: 0001-01-01T00:00:00Z
compactor:
    ring:
        kvstore:
            store: ""
            prefix: collectors/
            consul:
                host: localhost:8500
                acl_token: ""
                http_client_timeout: 20s
                consistent_reads: false
                watch_rate_limit: 1
                watch_burst_size: 1
                cas_retry_delay: 1s
            etcd:
                endpoints: []
                dial_timeout: 10s
                max_retries: 10
                tls_enabled: false
                tls_cert_path: ""
                tls_key_path: ""
                tls_ca_path: ""
                tls_server_name: ""
                tls_insecure_skip_verify: false
                tls_cipher_suites: ""
                tls_min_version: ""
                username: ""
                password: ""
            multi:
                primary: ""
                secondary: ""
                mirror_enabled: false
                mirror_timeout: 2s
        heartbeat_period: 5s
        heartbeat_timeout: 1m0s
        wait_stability_min_duration: 1m0s
        wait_stability_max_duration: 5m0s
        instance_id: hostname
        instance_interface_names:
            - eth0
            - en0
        instance_port: 0
        instance_addr: ""
        enable_inet6: false
        wait_active_instance_timeout: 10m0s
    compaction:
        v2_in_buffer_bytes: 5242880
        v2_out_buffer_bytes: 20971520
        v2_prefetch_traces_count: 1000
        compaction_window: 1h0m0s
        max_compaction_objects: 6000000
        max_block_bytes: 107374182400
        block_retention: 336h0m0s
        compacted_block_retention: 1h0m0s
        retention_concurrency: 10
        max_time_per_tenant: 5m0s
        compaction_cycle: 30s
    override_ring_key: compactor
ingester:
    lifecycler:
        ring:
            kvstore:
                store: memberlist
                prefix: collectors/
                consul:
                    host: localhost:8500
                    acl_token: ""
                    http_client_timeout: 20s
                    consistent_reads: false
                    watch_rate_limit: 1
                    watch_burst_size: 1
                    cas_retry_delay: 1s
                etcd:
                    endpoints: []
                    dial_timeout: 10s
                    max_retries: 10
                    tls_enabled: false
                    tls_cert_path: ""
                    tls_key_path: ""
                    tls_ca_path: ""
                    tls_server_name: ""
                    tls_insecure_skip_verify: false
                    tls_cipher_suites: ""
                    tls_min_version: ""
                    username: ""
                    password: ""
                multi:
                    primary: ""
                    secondary: ""
                    mirror_enabled: false
                    mirror_timeout: 2s
            heartbeat_timeout: 5m0s
            replication_factor: 1
            zone_awareness_enabled: false
            excluded_zones: ""
        num_tokens: 128
        heartbeat_period: 5s
        heartbeat_timeout: 1m0s
        observe_period: 0s
        join_after: 0s
        min_ready_duration: 15s
        interface_names:
            - eth0
        enable_inet6: false
        final_sleep: 0s
        tokens_file_path: ""
        availability_zone: ""
        unregister_on_shutdown: true
        readiness_check_ring_health: true
        address: ""
        port: 0
        id: hostname
    partition_ring:
        kvstore:
            store: memberlist
            prefix: collectors/
            consul:
                host: localhost:8500
                acl_token: ""
                http_client_timeout: 20s
                consistent_reads: false
                watch_rate_limit: 1
                watch_burst_size: 1
                cas_retry_delay: 1s
            etcd:
                endpoints: []
                dial_timeout: 10s
                max_retries: 10
                tls_enabled: false
                tls_cert_path: ""
                tls_key_path: ""
                tls_ca_path: ""
                tls_server_name: ""
                tls_insecure_skip_verify: false
                tls_cipher_suites: ""
                tls_min_version: ""
                username: ""
                password: ""
            multi:
                primary: ""
                secondary: ""
                mirror_enabled: false
                mirror_timeout: 2s
        min_partition_owners_count: 1
        min_partition_owners_duration: 10s
        delete_inactive_partition_after: 13h0m0s
    concurrent_flushes: 4
    flush_check_period: 10s
    flush_op_timeout: 5m0s
    trace_idle_period: 5s
    trace_live_period: 30s
    max_block_duration: 30m0s
    max_block_bytes: 524288000
    complete_block_timeout: 15m0s
    override_ring_key: ring
    flush_all_on_shutdown: false
    flush_object_storage: true
metrics_generator:
    ring:
        kvstore:
            store: memberlist
            prefix: collectors/
            consul:
                host: localhost:8500
                acl_token: ""
                http_client_timeout: 20s
                consistent_reads: false
                watch_rate_limit: 1
                watch_burst_size: 1
                cas_retry_delay: 1s
            etcd:
                endpoints: []
                dial_timeout: 10s
                max_retries: 10
                tls_enabled: false
                tls_cert_path: ""
                tls_key_path: ""
                tls_ca_path: ""
                tls_server_name: ""
                tls_insecure_skip_verify: false
                tls_cipher_suites: ""
                tls_min_version: ""
                username: ""
                password: ""
            multi:
                primary: ""
                secondary: ""
                mirror_enabled: false
                mirror_timeout: 2s
        heartbeat_period: 5s
        heartbeat_timeout: 1m0s
        instance_id: hostname
        instance_interface_names:
            - eth0
            - en0
        instance_addr: ""
        instance_port: 0
        enable_inet6: false
        instance_zone: ""
    processor:
        service_graphs:
            wait: 10s
            max_items: 10000
            workers: 10
            histogram_buckets:
                - 0.1
                - 0.2
                - 0.4
                - 0.8
                - 1.6
                - 3.2
                - 6.4
                - 12.8
            dimensions: []
            enable_client_server_prefix: false
            enable_messaging_system_latency_histogram: false
            peer_attributes:
                - peer.service
                - db.name
                - db.system
            span_multiplier_key: ""
            enable_virtual_node_label: false
            database_name_attributes:
                - db.namespace
                - db.name
                - db.system
        span_metrics:
            histogram_buckets:
                - 0.002
                - 0.004
                - 0.008
                - 0.016
                - 0.032
                - 0.064
                - 0.128
                - 0.256
                - 0.512
                - 1.024
                - 2.048
                - 4.096
                - 8.192
                - 16.384
            intrinsic_dimensions:
                service: true
                span_name: true
                span_kind: true
                status_code: true
            dimensions: []
            dimension_mappings: []
            enable_target_info: false
            span_multiplier_key: ""
            subprocessors:
                0: true
                1: true
                2: true
            filter_policies: []
            target_info_excluded_dimensions: []
            enable_instance_label: true
        local_blocks:
            block:
                bloom_filter_false_positive: 0.01
                bloom_filter_shard_size_bytes: 102400
                version: vParquet4
                search_encoding: snappy
                search_page_size_bytes: 1048576
                v2_index_downsample_bytes: 1048576
                v2_index_page_size_bytes: 256000
                v2_encoding: zstd
                parquet_row_group_size_bytes: 100000000
                parquet_dedicated_columns:
                    - scope: resource
                      name: k8s.cluster.name
                      type: string
                      options: []
                    - scope: resource
                      name: k8s.namespace.name
                      type: string
                      options: []
                    - scope: resource
                      name: k8s.pod.name
                      type: string
                      options: []
                    - scope: resource
                      name: k8s.container.name
                      type: string
                      options: []
                    - scope: span
                      name: http.request.method
                      type: string
                      options: []
                    - scope: span
                      name: http.response.status_code
                      type: int
                      options: []
                    - scope: span
                      name: url.path
                      type: string
                      options: []
                    - scope: span
                      name: url.route
                      type: string
                      options: []
                    - scope: span
                      name: server.address
                      type: string
                      options: []
                    - scope: span
                      name: server.port
                      type: int
                      options: []
                    - scope: span
                      name: http.method
                      type: string
                      options: []
                    - scope: span
                      name: http.url
                      type: string
                      options: []
                    - scope: span
                      name: http.route
                      type: string
                      options: []
                    - scope: span
                      name: http.status_code
                      type: int
                      options: []
            search:
                chunk_size_bytes: 1000000
                prefetch_trace_count: 1000
                read_buffer_count: 32
                read_buffer_size_bytes: 1048576
                cache_control:
                    footer: false
                    column_index: false
                    offset_index: false
            flush_check_period: 10s
            trace_idle_period: 5s
            trace_live_period: 30s
            max_block_duration: 1m0s
            max_block_bytes: 500000000
            concurrency: 4
            complete_block_timeout: 1h0m0s
            max_live_traces: 0
            max_live_traces_bytes: 250000000
            filter_server_spans: true
            flush_to_storage: false
            concurrent_blocks: 10
            time_overlap_cutoff: 0.2
        host_info:
            host_identifiers:
                - k8s.node.name
                - host.id
            metric_name: traces_host_info
    registry:
        collection_interval: 15s
        stale_duration: 15m0s
        max_label_name_length: 1024
        max_label_value_length: 2048
    storage:
        path: ""
        wal:
            wal_segment_size: 134217728
            wal_compression: none
            stripe_size: 16384
            truncate_frequency: 2h0m0s
            min_wal_time: 300000
            max_wal_time: 14400000
            no_lockfile: false
        remote_write_flush_deadline: 1m0s
        remote_write_add_org_id_header: true
    traces_storage:
        path: ""
        v2_encoding: none
        search_encoding: none
        ingestion_time_range_slack: 2m0s
        version: vParquet4
    traces_query_storage:
        path: ""
        v2_encoding: none
        search_encoding: none
        ingestion_time_range_slack: 2m0s
        version: vParquet4
    metrics_ingestion_time_range_slack: 30s
    query_timeout: 30s
    override_ring_key: metrics-generator
    codec: push-bytes
    disable_local_blocks: false
    disable_grpc: false
    limiter_type: series
    ingest_concurrency: 16
    instance_id: hostname
ingest:
    enabled: false
    kafka:
        address: localhost:9092
        topic: ""
        client_id: ""
        dial_timeout: 2s
        write_timeout: 10s
        sasl_username: ""
        sasl_password: ""
        consumer_group: ""
        consumer_group_offset_commit_interval: 1s
        last_produced_offset_retry_timeout: 10s
        auto_create_topic_enabled: true
        auto_create_topic_default_partitions: 1000
        producer_max_record_size_bytes: 15983616
        producer_max_buffered_bytes: 1073741824
        target_consumer_lag_at_startup: 2s
        max_consumer_lag_at_startup: 15s
        disable_kafka_telemetry: false
        consumer_group_lag_metric_update_interval: 1m0s
block_builder:
    instance_id: hostname
    assigned_partitions: {}
    partitions_per_instance: 0
    consume_cycle_duration: 5m0s
    max_consuming_bytes: 5000000000
    block:
        max_block_bytes: 20971520
        bloom_filter_false_positive: 0.01
        bloom_filter_shard_size_bytes: 102400
        version: vParquet4
        search_encoding: snappy
        search_page_size_bytes: 1048576
        v2_index_downsample_bytes: 1048576
        v2_index_page_size_bytes: 256000
        v2_encoding: zstd
        parquet_row_group_size_bytes: 100000000
        parquet_dedicated_columns:
            - scope: resource
              name: k8s.cluster.name
              type: string
              options: []
            - scope: resource
              name: k8s.namespace.name
              type: string
              options: []
            - scope: resource
              name: k8s.pod.name
              type: string
              options: []
            - scope: resource
              name: k8s.container.name
              type: string
              options: []
            - scope: span
              name: http.request.method
              type: string
              options: []
            - scope: span
              name: http.response.status_code
              type: int
              options: []
            - scope: span
              name: url.path
              type: string
              options: []
            - scope: span
              name: url.route
              type: string
              options: []
            - scope: span
              name: server.address
              type: string
              options: []
            - scope: span
              name: server.port
              type: int
              options: []
            - scope: span
              name: http.method
              type: string
              options: []
            - scope: span
              name: http.url
              type: string
              options: []
            - scope: span
              name: http.route
              type: string
              options: []
            - scope: span
              name: http.status_code
              type: int
              options: []
    wal:
        path: /var/tempo/block-builder/traces
        v2_encoding: none
        search_encoding: none
        ingestion_time_range_slack: 2m0s
        version: vParquet4
storage:
    trace:
        pool:
            max_workers: 400
            queue_depth: 20000
        wal:
            path: /var/tempo/wal
            v2_encoding: snappy
            search_encoding: none
            ingestion_time_range_slack: 2m0s
        block:
            bloom_filter_false_positive: 0.01
            bloom_filter_shard_size_bytes: 102400
            version: vParquet4
            search_encoding: snappy
            search_page_size_bytes: 1048576
            v2_index_downsample_bytes: 1048576
            v2_index_page_size_bytes: 256000
            v2_encoding: zstd
            parquet_row_group_size_bytes: 100000000
            parquet_dedicated_columns:
                - scope: resource
                  name: k8s.cluster.name
                  type: string
                  options: []
                - scope: resource
                  name: k8s.namespace.name
                  type: string
                  options: []
                - scope: resource
                  name: k8s.pod.name
                  type: string
                  options: []
                - scope: resource
                  name: k8s.container.name
                  type: string
                  options: []
                - scope: span
                  name: http.request.method
                  type: string
                  options: []
                - scope: span
                  name: http.response.status_code
                  type: int
                  options: []
                - scope: span
                  name: url.path
                  type: string
                  options: []
                - scope: span
                  name: url.route
                  type: string
                  options: []
                - scope: span
                  name: server.address
                  type: string
                  options: []
                - scope: span
                  name: server.port
                  type: int
                  options: []
                - scope: span
                  name: http.method
                  type: string
                  options: []
                - scope: span
                  name: http.url
                  type: string
                  options: []
                - scope: span
                  name: http.route
                  type: string
                  options: []
                - scope: span
                  name: http.status_code
                  type: int
                  options: []
        search:
            chunk_size_bytes: 1000000
            prefetch_trace_count: 1000
            read_buffer_count: 32
            read_buffer_size_bytes: 1048576
            cache_control:
                footer: false
                column_index: false
                offset_index: false
        blocklist_poll: 5m0s
        blocklist_poll_concurrency: 50
        blocklist_poll_tenant_concurrency: 0
        blocklist_poll_fallback: true
        blocklist_poll_tenant_index_builders: 2
        blocklist_poll_stale_tenant_index: 0s
        blocklist_poll_jitter_ms: 0
        blocklist_poll_tolerate_consecutive_errors: 1
        blocklist_poll_tolerate_tenant_failures: 1
        empty_tenant_deletion_enabled: false
        empty_tenant_deletion_age: 0s
        backend: ""
        local:
            path: ""
        gcs:
            bucket_name: ""
            prefix: ""
            chunk_buffer_size: 10485760
            endpoint: ""
            hedge_requests_at: 0s
            hedge_requests_up_to: 2
            insecure: false
            object_cache_control: ""
            object_metadata: {}
            list_blocks_concurrency: 3
            max_retries: 3
        s3:
            tls_cert_path: ""
            tls_key_path: ""
            tls_ca_path: ""
            tls_server_name: ""
            tls_insecure_skip_verify: false
            tls_cipher_suites: ""
            tls_min_version: VersionTLS12
            bucket: ""
            prefix: ""
            endpoint: ""
            region: ""
            access_key: ""
            secret_key: ""
            session_token: ""
            insecure: false
            part_size: 0
            hedge_requests_at: 0s
            hedge_requests_up_to: 2
            signature_v2: false
            forcepathstyle: false
            enable_dual_stack: false
            bucket_lookup_type: 0
            tags: {}
            storage_class: ""
            metadata: {}
            native_aws_auth_enabled: false
            list_blocks_concurrency: 3
            sse:
                type: ""
                kms_key_id: ""
                kms_encryption_context: ""
                encryption_key: ""
        azure:
            storage_account_name: ""
            storage_account_key: ""
            use_managed_identity: false
            use_federated_token: false
            user_assigned_id: ""
            container_name: ""
            prefix: ""
            endpoint_suffix: blob.core.windows.net
            max_buffers: 4
            buffer_size: 3145728
            hedge_requests_at: 0s
            hedge_requests_up_to: 2
        cache: ""
        background_cache:
            writeback_goroutines: 10
            writeback_buffer: 10000
        memcached: null
        redis: null
        cache_min_compaction_level: 0
        cache_max_block_age: 0s
overrides:
    defaults:
        ingestion:
            rate_strategy: local
            rate_limit_bytes: 15000000
            burst_size_bytes: 20000000
            max_traces_per_user: 10000
            retry_info_enabled: true
        read:
            max_bytes_per_tag_values_query: 1000000
        metrics_generator:
            generate_native_histograms: classic
            ingestion_time_range_slack: 0s
            native_histogram_bucket_factor: 1.1
            native_histogram_max_bucket_number: 100
            native_histogram_min_reset_duration: 15m0s
        global:
            max_bytes_per_trace: 5000000
    per_tenant_override_config: ""
    per_tenant_override_period: 10s
    user_configurable_overrides:
        enabled: false
        poll_interval: 1m0s
        client:
            backend: ""
            confirm_versioning: true
            local:
                path: ""
            gcs:
                bucket_name: ""
                prefix: ""
                chunk_buffer_size: 10485760
                endpoint: ""
                hedge_requests_at: 0s
                hedge_requests_up_to: 2
                insecure: false
                object_cache_control: ""
                object_metadata: {}
                list_blocks_concurrency: 3
                max_retries: 3
            s3:
                tls_cert_path: ""
                tls_key_path: ""
                tls_ca_path: ""
                tls_server_name: ""
                tls_insecure_skip_verify: false
                tls_cipher_suites: ""
                tls_min_version: VersionTLS12
                bucket: ""
                prefix: ""
                endpoint: ""
                region: ""
                access_key: ""
                secret_key: ""
                session_token: ""
                insecure: false
                part_size: 0
                hedge_requests_at: 0s
                hedge_requests_up_to: 2
                signature_v2: false
                forcepathstyle: false
                enable_dual_stack: false
                bucket_lookup_type: 0
                tags: {}
                storage_class: ""
                metadata: {}
                native_aws_auth_enabled: false
                list_blocks_concurrency: 3
                sse:
                    type: ""
                    kms_key_id: ""
                    kms_encryption_context: ""
                    encryption_key: ""
            azure:
                storage_account_name: ""
                storage_account_key: ""
                use_managed_identity: false
                use_federated_token: false
                user_assigned_id: ""
                container_name: ""
                prefix: ""
                endpoint_suffix: blob.core.windows.net
                max_buffers: 4
                buffer_size: 3145728
                hedge_requests_at: 0s
                hedge_requests_up_to: 2
        api:
            check_for_conflicting_runtime_overrides: false
memberlist:
    node_name: ""
    randomize_node_name: true
    stream_timeout: 2s
    retransmit_factor: 2
    pull_push_interval: 30s
    gossip_interval: 1s
    gossip_nodes: 2
    gossip_to_dead_nodes_time: 30s
    dead_node_reclaim_time: 0s
    compression_enabled: false
    notify_interval: 0s
    advertise_addr: ""
    advertise_port: 7946
    cluster_label: ""
    cluster_label_verification_disabled: false
    join_members: []
    min_join_backoff: 1s
    max_join_backoff: 1m0s
    max_join_retries: 10
    abort_if_cluster_fast_join_fails: false
    abort_if_cluster_join_fails: false
    rejoin_interval: 0s
    left_ingesters_timeout: 5m0s
    obsolete_entries_timeout: 30s
    leave_timeout: 20s
    broadcast_timeout_for_local_updates_on_shutdown: 10s
    message_history_buffer_bytes: 0
    watch_prefix_buffer_size: 128
    bind_addr: []
    bind_port: 7946
    packet_dial_timeout: 2s
    packet_write_timeout: 5s
    max_concurrent_writes: 3
    acquire_writer_timeout: 250ms
    tls_enabled: false
    tls_cert_path: ""
    tls_key_path: ""
    tls_ca_path: ""
    tls_server_name: ""
    tls_insecure_skip_verify: false
    tls_cipher_suites: ""
    tls_min_version: ""
    zone_aware_routing:
        enabled: false
        instance_availability_zone: ""
        role: member
usage_report:
    reporting_enabled: true
    backoff:
        min_period: 100ms
        max_period: 10s
        max_retries: 0
cache:
    background:
        writeback_goroutines: 10
        writeback_buffer: 10000
    caches: []
backend_scheduler:
    work:
        prune_age: 1h0m0s
        dead_job_timeout: 24h0m0s
    maintenance_interval: 1m0s
    backend_flush_interval: 1m0s
    provider:
        retention:
            interval: 1h0m0s
        compaction:
            measure_interval: 1m0s
            compaction:
                v2_in_buffer_bytes: 5242880
                v2_out_buffer_bytes: 20971520
                v2_prefetch_traces_count: 1000
                compaction_window: 1h0m0s
                max_compaction_objects: 6000000
                max_block_bytes: 107374182400
                block_retention: 336h0m0s
                compacted_block_retention: 1h0m0s
                retention_concurrency: 10
                max_time_per_tenant: 5m0s
                compaction_cycle: 30s
            max_jobs_per_tenant: 1000
            min_input_blocks: 2
            max_input_blocks: 4
            max_compaction_level: 0
            min_cycle_interval: 30s
    job_timeout: 15s
    local_work_path: /var/tempo
backend_scheduler_client:
    grpc_client_config:
        max_recv_msg_size: 104857600
        max_send_msg_size: 104857600
        grpc_compression: snappy
        rate_limit: 0
        rate_limit_burst: 0
        backoff_on_ratelimits: false
        backoff_config:
            min_period: 100ms
            max_period: 10s
            max_retries: 10
        initial_stream_window_size: 63KiB1023B
        initial_connection_window_size: 63KiB1023B
        tls_enabled: false
        tls_cert_path: ""
        tls_key_path: ""
        tls_ca_path: ""
        tls_server_name: ""
        tls_insecure_skip_verify: false
        tls_cipher_suites: ""
        tls_min_version: ""
        connect_timeout: 5s
        connect_backoff_base_delay: 1s
        connect_backoff_max_delay: 5s
        cluster_validation:
            label: ""
backend_worker:
    backend_scheduler_addr: ""
    backoff:
        min_period: 100ms
        max_period: 1m0s
        max_retries: 0
    compaction:
        v2_in_buffer_bytes: 5242880
        v2_out_buffer_bytes: 20971520
        v2_prefetch_traces_count: 1000
        compaction_window: 1h0m0s
        max_compaction_objects: 6000000
        max_block_bytes: 107374182400
        block_retention: 336h0m0s
        compacted_block_retention: 1h0m0s
        retention_concurrency: 10
        max_time_per_tenant: 5m0s
        compaction_cycle: 30s
    override_ring_key: backend-worker
    ring:
        kvstore:
            store: ""
            prefix: collectors/
            consul:
                host: localhost:8500
                acl_token: ""
                http_client_timeout: 20s
                consistent_reads: false
                watch_rate_limit: 1
                watch_burst_size: 1
                cas_retry_delay: 1s
            etcd:
                endpoints: []
                dial_timeout: 10s
                max_retries: 10
                tls_enabled: false
                tls_cert_path: ""
                tls_key_path: ""
                tls_ca_path: ""
                tls_server_name: ""
                tls_insecure_skip_verify: false
                tls_cipher_suites: ""
                tls_min_version: ""
                username: ""
                password: ""
            multi:
                primary: ""
                secondary: ""
                mirror_enabled: false
                mirror_timeout: 2s
        heartbeat_period: 5s
        heartbeat_timeout: 1m0s
        wait_stability_min_duration: 1m0s
        wait_stability_max_duration: 5m0s
        instance_id: hostname
        instance_interface_names:
            - eth0
            - en0
        instance_port: 0
        instance_addr: ""
        enable_inet6: false
        wait_active_instance_timeout: 10m0s
    finish_on_shutdown_timeout: 30s
live_store:
    ring:
        kvstore:
            store: memberlist
            prefix: collectors/
            consul:
                host: localhost:8500
                acl_token: ""
                http_client_timeout: 20s
                consistent_reads: false
                watch_rate_limit: 1
                watch_burst_size: 1
                cas_retry_delay: 1s
            etcd:
                endpoints: []
                dial_timeout: 10s
                max_retries: 10
                tls_enabled: false
                tls_cert_path: ""
                tls_key_path: ""
                tls_ca_path: ""
                tls_server_name: ""
                tls_insecure_skip_verify: false
                tls_cipher_suites: ""
                tls_min_version: ""
                username: ""
                password: ""
            multi:
                primary: ""
                secondary: ""
                mirror_enabled: false
                mirror_timeout: 2s
        heartbeat_period: 5s
        heartbeat_timeout: 1m0s
        instance_id: hostname
        instance_interface_names:
            - eth0
        instance_addr: ""
        instance_port: 0
        enable_inet6: false
        instance_zone: ""
    partition_ring:
        kvstore:
            store: memberlist
            prefix: collectors/
            consul:
                host: localhost:8500
                acl_token: ""
                http_client_timeout: 20s
                consistent_reads: false
                watch_rate_limit: 1
                watch_burst_size: 1
                cas_retry_delay: 1s
            etcd:
                endpoints: []
                dial_timeout: 10s
                max_retries: 10
                tls_enabled: false
                tls_cert_path: ""
                tls_key_path: ""
                tls_ca_path: ""
                tls_server_name: ""
                tls_insecure_skip_verify: false
                tls_cipher_suites: ""
                tls_min_version: ""
                username: ""
                password: ""
            multi:
                primary: ""
                secondary: ""
                mirror_enabled: false
                mirror_timeout: 2s
        min_partition_owners_count: 1
        min_partition_owners_duration: 10s
        delete_inactive_partition_after: 13h0m0s
    metrics:
        time_overlap_cutoff: 0.2
    commit_interval: 5s
    wal:
        path: /var/tempo/live-store/traces
        v2_encoding: none
        search_encoding: none
        ingestion_time_range_slack: 2m0s
        version: vParquet4
    query_block_concurrency: 10
    complete_block_timeout: 1h0m0s
    complete_block_concurrency: 2
    shutdown_marker_dir: /var/tempo/live-store/shutdown-marker
    flush_check_period: 10s
    flush_op_timeout: 5m0s
    max_trace_live: 30s
    max_trace_idle: 5s
    max_live_traces_bytes: 250000000
    max_block_duration: 30m0s
    max_block_bytes: 104857600
    block_config:
        bloom_filter_false_positive: 0.01
        bloom_filter_shard_size_bytes: 102400
        version: ""
        search_encoding: snappy
        search_page_size_bytes: 1048576
        v2_index_downsample_bytes: 1048576
        v2_index_page_size_bytes: 256000
        v2_encoding: zstd
        parquet_row_group_size_bytes: 100000000
        parquet_dedicated_columns:
            - scope: resource
              name: k8s.cluster.name
              type: string
              options: []
            - scope: resource
              name: k8s.namespace.name
              type: string
              options: []
            - scope: resource
              name: k8s.pod.name
              type: string
              options: []
            - scope: resource
              name: k8s.container.name
              type: string
              options: []
            - scope: span
              name: http.request.method
              type: string
              options: []
            - scope: span
              name: http.response.status_code
              type: int
              options: []
            - scope: span
              name: url.path
              type: string
              options: []
            - scope: span
              name: url.route
              type: string
              options: []
            - scope: span
              name: server.address
              type: string
              options: []
            - scope: span
              name: server.port
              type: int
              options: []
            - scope: span
              name: http.method
              type: string
              options: []
            - scope: span
              name: http.url
              type: string
              options: []
            - scope: span
              name: http.route
              type: string
              options: []
            - scope: span
              name: http.status_code
              type: int
              options: []

Compression and encoding

Thu, 09 Apr 2026 14:59:14 +0000

Compression and encoding

Tempo can compress traces that it pushes to backend storage. This requires extra memory and CPU, but it reduces the quantity of stored data. Anecdotal tests suggest that zstd will cut your storage costs to ~15% of the uncompressed amount. It is highly recommended to use the default zstd. (The compression field is used for the old v2 format. the vParquet* formats compress columns individually.)

Compression is configured under storage like so:

storage:
  trace:
    block:
      v2_encoding: zstd

The following options are supported:

none
gzip
lz4-64k
lz4-256k
lz4-1M
lz4
snappy
zstd
s2

Although all of these compression formats are supported in Tempo, at Grafana we use zstd. It’s possible/probable that the other compression algorithms may have issue at scale. File an issue if you have any problems.

WAL

The WAL also supports compression. By default, this is configured to use snappy. This comes with a small performance penalty but reduces disk I/O and and adds checksums to the WAL. All of the above configuration options are supported but only snappy has been tested at scale.

storage:
  trace:
    wal:
      v2_encoding: snappy

Apache Parquet block format

Thu, 09 Apr 2026 14:59:14 +0000

Apache Parquet block format

Tempo has a default columnar block format based on Apache Parquet. This format is required for tags-based search as well as TraceQL, the query language for traces. The columnar block format improves search performance and enables an ecosystem of tools, including Tempo CLI, to access the underlying trace data.

For more information, refer to Issue 4694.

Considerations

The Parquet block format is enabled by default since Tempo 2.0.

If you install using the Tempo Helm charts, then Parquet is enabled by default. No data conversion or upgrade process is necessary. As soon as a block format is enabled, Tempo starts writing data in that format, leaving existing data as-is.

Block formats based on Parquet require more CPU and memory resources than the previous v2 format but provide search and TraceQL functionality.

Choose a different block format

Warning
vParquet3 and v2 have been deprecated and will be removed in Tempo 3.0. In order to cleanly migrate forward to Tempo 3.0 you will need vParquet4 or higher blocks.

The default block format is vParquet4, which is the latest iteration of the Parquet-based columnar block format in Tempo. vParquet4 introduces columns that enable querying for data in array attributes as well as events and links. For more information, refer to Dedicated attribute columns.

You can still use the previous format vParquet3. To enable it, set the block version option to vParquet3 in the Storage section of the configuration file.

# block format version. options: vParquet4
[version: vParquet3]

In some cases, you may choose to disable Parquet and use the old v2 block format. Using the v2 block format disables all forms of search, but also reduces resource consumption, and may be desired for a high-throughput cluster that doesn’t need these capabilities. To make this change, set the block version option to v2 in the Storage section of the configuration file.

# block format version. options: v2, vParquet3, vParquet4
[version: v2]

To re-enable the default vParquet4 format, remove the block version option from the Storage section of the configuration file or set the option to vParquet4.

Parquet configuration parameters

Some parameters in the Tempo configuration are specific to Parquet. For more information, refer to the storage configuration documentation.

Trace search parameters

These configuration options impact trace search.

Parameter	Default value	Description
`[read_buffer_size_bytes: <int>]`	10485676	Size of read buffers used when performing search on a vParquet block. This value times the read_buffer_count is the total amount of bytes used for buffering when performing search on a Parquet block.
`[read_buffer_count: <int>]`	32	Number of read buffers used when performing search on a vParquet block. This value times the read_buffer_size_bytes is the total amount of bytes used for buffering when performing search on a Parquet block.

The cache_control section contains the follow parameters for Parquet metadata objects:

Parameter	Default value	Description
`[footer: \| default = false]`	`false`	Specifies if the footer should be cached
`[column_index: <bool> \| default = false]`	`false`	Specifies if the column index should be cached
`[offset_index: <bool> \| default = false]`	`false`	Specifies if the offset index should be cached

Hosted storage

Thu, 09 Apr 2026 14:59:14 +0000

Hosted storage

Tempo provides additional hosted storage configuration options discussed on the pages below. These options relate to providers such as Google Cloud, AWS S3, and Azure.

For additional details about storage configuration, refer to Storage configuration.

Amazon S3 permissions
Set Amazon S3 permissions for Tempo.
Azure blob storage permissions and management
Azure blob storage permissions and configuration options for Tempo.
Google Cloud Storage
Learn about Google Cloud Storage permissions for Tempo.

Network options

Thu, 09 Apr 2026 14:59:14 +0000

Network options

Tempo provides additional network configuration options discussed on the pages below.

Configure IPv6
Learn how to configure IPv6 for Tempo.
Configure TLS communication
Configure Tempo components to communicate using TLS.
Run Tempo distributed with sidecar proxies
Configure Tempo distributed to run with sidecar proxies

Polling

Thu, 09 Apr 2026 14:59:14 +0000

Polling

The polling cycle is controlled by a number of configuration options detailed here.

storage:
    trace:
        # How often to repoll the backend for new blocks. Default is 5m
        [blocklist_poll: <duration>]

        # Number of blocks to process in parallel during polling. Default is 50.
        [blocklist_poll_concurrency: <int>]

        # By default components will pull the blocklist from the tenant index. If that fails the component can
        # fallback to scanning the entire bucket. Set to false to disable this behavior. Default is true.
        [blocklist_poll_fallback: <bool>]

        # Maximum number of compactors or workers that should build the tenant index. All other components will download
        # the index. Default 2.
        [blocklist_poll_tenant_index_builders: <int>]

        # The oldest allowable tenant index. If an index is pulled that is older than this duration,
        # the polling will consider this an error. Note that `blocklist_poll_fallback` applies here.
        # If fallback is true and a tenant index exceeds this duration, it will fall back to listing
        # the bucket contents.
        # Default 0 (disabled).
        [blocklist_poll_stale_tenant_index: <duration>]

Due to the mechanics of the tenant index, the blocklist will be stale by at most 2 times the configured blocklist_poll duration. There are two configuration options that need to be balanced against the blockist_poll to handle this:

The ingester complete_block_timeout is used to hold a block in the ingester for a given period of time after it has been flushed. This allows the ingester to return traces to the queriers while they are still unaware of the newly flushed blocks.

ingester:
  # How long to hold a complete block in the ingester after it has been flushed to the backend. Default is 15m
  [complete_block_timeout: <duration>]

The compactor compacted_block_retention is used to keep a block in the backend for a given period of time after it has been compacted and the data is no longer needed. This allows queriers with a stale blocklist to access these blocks successfully until they complete their polling cycles and have up to date blocklists. Like the complete_block_timeout, this should be at a minimum 2x the configured blocklist_poll duration.

compactor:
  compaction:
    # How long to leave a block in the backend after it has been compacted successfully. Default is 1h
    [compacted_block_retention: <duration>]

Additionally, the querier blocklist_poll duration needs to be greater than or equal to the compactor blocklist_poll duration. Otherwise, a querier may not correctly check all assigned blocks and incorrectly return 404. It is recommended to simply set both components to use the same poll duration.

Usage tracker

Thu, 09 Apr 2026 14:59:14 +0000

Usage tracker

The usage tracker accurately tracks the amount of ingested traffic using a set of custom labels on a per-tenant basis, providing fine-grained control over your data.

Use the cost_attributes option to configure usage trackers in the distributor, which expose metrics of ingested traffic grouped by configurable attributes exposed on /usage_metrics.

Enable the usage tracker

To use this feature, you need to enable it in the distributor and configure the overrides to handle the data.

Open your configuration file, for example, tempo.yaml.
In the distributor section, locate the cost_attribution line. If it is not present, then copy and paste the example below into your distributor section and update the options.

 usage:
        cost_attribution:
            # Enables the "cost-attribution" usage tracker. Per-tenant attributes are configured in overrides.
            [enabled: <boolean> | default = false]
            # Maximum number of series per tenant.
            [max_cardinality: <int> | default = 10000]
            # Interval after which a series is considered stale and will be deleted from the registry.
            # Once a metrics series is deleted, it won't be emitted anymore, keeping active series low.
            [stale_duration: <duration> | default = 15m0s]

Configure overrides

You also need to configure the dimensions to break down your the usage data in the standard overrides. In the overrides section, you can define attributes to group ingested data by and you can rename and combine attributes.

For more information, refer to the standard overrides section of the Configuration documentation.

In this example, usage data is grouped by service.name.

# Overrides configuration block
overrides:
  # Global ingestion limits configurations
  defaults:
    # Cost attribution usage tracker configuration
    cost_attribution:
      dimensions:
        service.name: ""

You can also configure per tenant in the runtime-overrides or in the user-configurable-overrides.

Review usage metrics

Once everything is configured, the usage metrics are exposed in the distributor:

GET /usage_metrics

Example:

curl http://localhost:3200/usage_metrics
# HELP tempo_usage_tracker_bytes_received_total bytes total received with these attributes
# TYPE tempo_usage_tracker_bytes_received_total counter
tempo_usage_tracker_bytes_received_total{service_name="article-service",tenant="single-tenant",tracker="cost-attribution"} 7327
tempo_usage_tracker_bytes_received_total{service_name="auth-service",tenant="single-tenant",tracker="cost-attribution"} 8938
tempo_usage_tracker_bytes_received_total{service_name="billing-service",tenant="single-tenant",tracker="cost-attribution"} 2401
tempo_usage_tracker_bytes_received_total{service_name="cart-service",tenant="single-tenant",tracker="cost-attribution"} 4116
tempo_usage_tracker_bytes_received_total{service_name="postgres",tenant="single-tenant",tracker="cost-attribution"} 3571
tempo_usage_tracker_bytes_received_total{service_name="shop-backend",tenant="single-tenant",tracker="cost-attribution"} 17619

You can configure Prometheus to scrape this endpoint to collect usage metrics.

Example:

scrape_configs:
  - job_name: 'tempo-usage'
    static_configs:
      - targets: ['<DISTRIBUTOR-HOST>:3200']
    metrics_path: '/usage_metrics'

Scope dimensions by resource or span

You can configure dimensions to look for attributes at specific scopes using prefixes:

resource.<attribute> - Look only at resource-level attributes
span.<attribute> - Look only at span-level attributes
<attribute> (no prefix) - Look at both resource and span levels

Example:

overrides:
  defaults:
    cost_attribution:
      dimensions:
        resource.service.name: "service"    # Scoped by resource, only look at `service.name` in resource attributes.
        span.db.system: "database"          # Scoped by span, only look at `db.system` in span attributes.
        k8s.namespace.name: "namespace"     # Unscoped, look at `k8s.namespace.name` in both resource and span attributes

Note
When using unscoped attributes (no prefix), span-level attributes will always overwrite resource-level attributes if both exist.

This can lead to mixed values from both scopes, use scope prefixes (resource. or span.) for predictable behavior when attributes exist at both and you only want values from one of them.

This table provides more information about the behavior:

Configuration	Resource has `service.name="api"`	Span has `service.name="worker"`	Result
`resource.service.name`	Uses `"api"`	Ignores span value	`"api"`
`span.service.name`	Ignores resource value	Uses `"worker"`	`"worker"`
`service.name`	Uses `"api"` initially	Overwrites with `"worker"`	`"worker"`

Configure target labels

You can customize the label names that appear in the /usage_metrics metrics by configuring a custom label name in the dimensions configuration.

dimensions:
  <ATTRIBUTE_NAME>: "<LABEL_NAME>"

Note
The _<LABEL_NAME>_ should be a valid Prometheus label name. If you provide an invalid label name, Tempo sanitizes it.

Example:

overrides:
  defaults:
    cost_attribution:
      dimensions:
        service.name: ""                    # Results in: service_name
        span.service.name: ""               # Results in: service_name
        resource.service.name: ""           # Results in: service_name
        span.db.system: ""                  # Results in: db_system
        span.db.system: "database.type"     # Results in: database_type
        k8s.namespace.name: "namespace"     # Results in: namespace

Use tracing data in Grafana

Thu, 09 Apr 2026 14:59:14 +0000

Use tracing data in Grafana

You can use Tempo as a data source in Grafana. Grafana Cloud comes pre-configured with a Tempo data source.

If you are using Grafana on-prem, you need to set up the Tempo data source.

Tip
If you want to explore tracing data in Grafana, try the Intro to Metrics, Logs, Traces, and Profiling example.

This video explains how to add data sources, including Loki, Tempo, and Mimir, to Grafana and Grafana Cloud. Tempo data source set up starts at 4:58 in the video.

Configure the data source

For detailed instructions on the Tempo data source in Grafana, refer to Tempo data source.

To configure Tempo with Grafana:

Point the Grafana data source at your Tempo query frontend (or monolithic mode Tempo).
Enter the URL: http://<tempo hostname>:<http port number>. For most of the Tempo examples the following works.

The port of 3200 is a common port used in our examples. Tempo default HTTP port is 3200.

Query the data source

Refer to Tempo in Grafana for an overview about how tracing data can be viewed and queried in Grafana.

For information on querying the Tempo data source, refer to Tempo query editor.

Anonymous usage reporting

Thu, 09 Apr 2026 14:59:14 +0000

Anonymous usage reporting

By default, Tempo reports anonymous usage data about itself to Grafana Labs. This data is used to understand which features are commonly enabled, as well as which deployment modes, replication factors, and compression levels are most popular, etc.

By providing information on how people use Tempo, usage reporting helps the Tempo team decide where to focus their development and documentation efforts.

The following configuration values are used:

Receivers enabled
Frontend concurrency and version
Storage cache, backend, WAL and block encodings
Ring replication factor, and kvstore
Features toggles enabled

No private information is collected, and all reports are completely anonymous.

Configure anonymous usage reporting

Reporting is controlled by the usage_report configuration option and can be disabled. For instructions, refer to the Configuration documentation.

View usage stats report

Tempo provides a usage stats report that you can view in your browser.

To view the report, go to the following URL on the Tempo instance: http://localhost:3200/status/usage-stats

An example report output looks like this:

{
  "clusterID": "",
  "createdAt": "0001-01-01T00:00:00Z",
  "interval": "0001-01-01T00:00:00Z",
  "intervalPeriod": 14400,
  "target": "all",
  "version": {
    "version": "v2.8.0",
    "revision": "31e2dddb5",
    "branch": "main",
    "buildUser": "",
    "buildDate": "",
    "goVersion": "go1.24.3"
  },
  "os": "linux",
  "arch": "arm64",
  "edition": "oss",
  "metrics": {
    "ring_kv_store": "inmemory",
    "memstats": {
      "heap_inuse": 14106624,
      "stack_inuse": 2490368,
      "pause_total_ns": 872084,
      "num_gc": 3,
      "gc_cpu_fraction": 0.08786506719453682,
      "heap_alloc": 11640400,
      "alloc": 11640400,
      "total_alloc": 16491760,
      "sys": 27874568
    },
    "num_cpu": 8,
    "feature_enabled_multitenancy": 0,
    "receiver_enabled_jaeger": 0,
    "storage_block_encoding": "zstd",
    "storage_block_search_encoding": "snappy",
    "storage_cache": "",
    "receiver_enabled_opencensus": 0,
    "feature_enabled_auth_stats": 0,
    "frontend_version": "v1",
    "storage_backend": "local",
    "receiver_enabled_otlp": 0,
    "ring_replication_factor": 1,
    "storage_wal_encoding": "snappy",
    "storage_wal_search_encoding": "none",
    "num_goroutine": 813,
    "cache_memcached": 1,
    "cache_redis": 0,
    "distributor_bytes_received": {
      "total": 0,
      "rate": 0
    },
    "distributor_spans_received": {
      "total": 0,
      "rate": 0
    },
    "receiver_enabled_kafka": 0,
    "receiver_enabled_zipkin": 0
  }
}

Which information is collected?

Tempo collects and reports the following information to Grafana Labs. The report from your Tempo instance may vary from the provided example. Each field provides insight into the Tempo instance, its environment, and configuration. The fields are grouped by their purpose.

This information helps Grafana Labs understand how Tempo is used, which features are enabled, and the typical deployment environments, without collecting any private or user-identifying data.

Note
Tempo maintainers commit to keeping the list of tracked information updated over time, and reporting any change both via the CHANGELOG and the release notes.

Instance identification

clusterID: A unique, randomly generated identifier for the Tempo cluster. This value helps Grafana Labs distinguish between different deployments.
createdAt: The timestamp when anonymous usage reporting was first enabled and the cluster ID was created.
interval: The timestamp marking the start of the current reporting interval.
intervalPeriod: The length of the reporting interval, in seconds.

Deployment and version information

target: The deployment mode or target for the Tempo instance, such as all for monolithic mode.
version: An object containing detailed version information:
- version: The Tempo version, for example v2.8.0.
- revision: The Git commit hash or revision used to build the binary.
- branch: The Git branch used for the build.
- buildUser: The user who built the binary.
- buildDate: The date and time when the binary was built.
- goVersion: The Go language version used for the build.

Environment details

os: The operating system the Tempo instance is running on, such as linux.
arch: The system architecture, such as arm64.
edition: The edition of Tempo, such as oss for open source.

Metrics and configuration

metrics: An object containing runtime metrics and configuration:
- ring_kv_store: The key-value store used for the ring, for example inmemory.
- memstats: Memory usage statistics, including heap and stack usage, garbage collection metrics, and total allocations.
- num_cpu: The number of logical CPU cores available.
- feature_enabled_multitenancy: Indicates if multitenancy is enabled (1) or not (0).
- receiver_enabled_jaeger, receiver_enabled_opencensus, receiver_enabled_otlp, receiver_enabled_kafka, receiver_enabled_zipkin: Flags indicating if each trace receiver is enabled (1) or not (0).
- storage_block_encoding, storage_block_search_encoding, storage_wal_encoding, storage_wal_search_encoding: The encoding or compression algorithms used for storage blocks and write-ahead logs.
- storage_cache: The cache backend used for storage, if any.
- feature_enabled_auth_stats: Indicates if authentication statistics are enabled.
- frontend_version: The version of the frontend component.
- storage_backend: The storage backend in use, such as local.
- ring_replication_factor: The replication factor for the ring.
- num_goroutine: The number of active Go routines.
- cache_memcached, cache_redis: Flags indicating if Memcached or Redis caching is enabled.
- distributor_bytes_received, distributor_spans_received: Objects showing the total and rate of bytes and spans received by the distributor.