Manage advanced systems on Grafana Labs

Enable multi-tenancy

Thu, 28 May 2026 17:50:33 +0100

Enable multi-tenancy

Tempo is a multi-tenant distributed tracing backend. Tempo uses the X-Scope-OrgID header to enforce multi-tenancy in Tempo and Grafana Enterprise Traces. It is set to the tenant (or “organization”) name. It is used for scoped writes (ingest) so that each span is stored under its specified tenant, and scoped reads so that queries return only that tenant’s data.

If you’re interested in setting up multi-tenancy, consult the multi-tenant example in the repository. This example uses the following settings to achieve multi-tenancy in Tempo.

Tenant IDs

Tenant IDs are transmitted to Tempo via the X-Scope-OrgID HTTP header. This header must be included in all requests to Tempo when multi-tenancy is enabled.

Multi-tenancy on ingestion is supported with both gRPC and HTTP for OTLP (OpenTelemetry Protocol). You can add the header in:

OpenTelemetry Collector configuration
Grafana Alloy configuration
Any HTTP/gRPC client using curl or other relevant tools

Example Alloy configuration

The following example shows how to configure Grafana Alloy to send traces with a tenant ID.

otelcol.exporter.otlphttp "tempo" {
    // Define the client for exporting.
    client {
        // Send the X-Scope-OrgID header to the Tempo instance for multi-tenancy (tenant 1234).
        headers = {
            "X-Scope-OrgID" = "1234",
        }

        // Send to the locally running Tempo instance, on port 4318 (OTLP HTTP).
        endpoint = "http://tempo:4318"

        // Configure TLS settings for communicating with the endpoint.
        tls {
            // The connection is insecure.
            insecure = true
            // Do not verify TLS certificates when connecting.
            insecure_skip_verify = true
        }
    }
}

Configure multi-tenancy

Configure the OpenTelemetry Collector to attach the X-Scope-OrgID header on push:
```
exporters:
  otlp:
    headers:
      x-scope-orgid: foo-bar-baz
```

Configure the Tempo data source in Grafana to pass the tenant with the same header:

- name: Tempo-Multitenant
  jsonData:
    httpHeaderName1: "X-Scope-OrgID"
  secureJsonData:
    httpHeaderValue1: "foo-bar-baz"

Enable multi-tenancy on the Tempo backend by setting the following configuration value on all Tempo components:
YAML
```
multitenancy_enabled: true
```
or from the command line:
shell
```
tempo --multitenancy.enabled=true
```
This option forces all Tempo components to require the X-Scope-OrgID header.

Configure per-tenant settings

After you enable multi-tenancy, you can customize settings on a per-tenant basis using runtime overrides. This is useful when different tenants have different requirements, for example varying block retention periods or ingestion limits.

This configuration works for both Tempo 2.x and Tempo 3.x architectures. If you are upgrading from Tempo 2.x, refer to the Tempo 3.0 migration guide.

To configure per-tenant overrides, create a separate overrides file and reference it from your main Tempo configuration:

# tempo.yaml
overrides:
  per_tenant_override_config: /conf/overrides.yaml

Define tenant-specific settings in the overrides.yaml file. The following example sets different block retention periods and ingestion limits for two tenants:

# overrides.yaml
overrides:
  "tenant-a":
    compaction:
      block_retention: 24h
    ingestion:
      burst_size_bytes: 20000000
      rate_limit_bytes: 4000000
      max_traces_per_user: 10000
    global:
      max_bytes_per_trace: 30000000
    metrics_generator:
      collection_interval: 15s
      ring_size: 2
      max_active_series: 5000
      processors:
        - span-metrics
        - service-graphs
      forwarder:
        queue_size: 5000
        workers: 3
  "tenant-b":
    compaction:
      block_retention: 48h
    ingestion:
      burst_size_bytes: 20000000
      rate_limit_bytes: 4000000
      max_traces_per_user: 10000
    global:
      max_bytes_per_trace: 30000000
    metrics_generator:
      ring_size: 2
      max_active_series: 100000
      processors: []
      forwarder:
        queue_size: 5000
        workers: 3

Warning
When using per-tenant overrides, you must set every field you want to preserve. Tempo replaces unset fields with zero values, not with the defaults from your main configuration. Omitting a field can lead to unexpected behavior, such as a 0s retention effectively disabling block retention for that tenant.

The overrides file is dynamically loaded, so you can update it at runtime without restarting Tempo.

For the full list of available override fields, refer to Overrides.

Cross-tenant query federation

Thu, 28 May 2026 17:50:33 +0100

Cross-tenant query federation

Cross-tenant query federation lets you run a single query across multiple tenants at the same time, instead of repeating the same query for each tenant individually. This is useful when you operate many teams or environments and need to troubleshoot, compare behavior, or enforce governance consistently across them.

Supported operations

Tempo supports multi-tenant queries for search, search-tags, and trace-by-ID search operations. TraceQL metrics queries (for example, query range and instant endpoints that use TraceQLMetrics) also support cross-tenant queries.

These operations can be federated across multiple tenants when you specify more than one tenant ID in the X-Scope-OrgID header.

To perform multi-tenant queries, send tenant IDs separated by a | character in the X-Scope-OrgID header, for example, foo|bar.

Enable cross-tenant query

By default, cross-tenant query is enabled and can be controlled using multi_tenant_queries_enabled configuration setting.

This setting is a no-op unless multitenancy_enabled: true is enabled in the cluster configuration.

query_frontend:
  multi_tenant_queries_enabled: true

For more information on configuration options, refer to Enable multi-tenancy.

TraceQL queries

Queries performed using the cross-tenant configured data source, in either Explore or inside of dashboards, are performed across all the tenants that you specified in the X-Scope-OrgID header.

TraceQL queries that compare multiple spansets may not correctly return all traces in a cross-tenant query. For instance,

{ span.attr1 = "bar" } && { span.attr2 = "foo" }

TraceQL evaluates a contiguously stored trace. If these two conditions are satisfied in separate tenants, then Tempo doesn’t correctly return the trace.

Tenant IDs in cross-tenant queries

Tempo uses the same X-Scope-OrgID header for cross-tenant queries as for single-tenant queries. To query multiple tenants, specify tenant IDs separated by a | character in the header.

When multiple tenant IDs are specified, Tempo normalizes them by sorting and removing duplicate tenant IDs. For example, bar|foo|bar is normalized to bar|foo.

The following example shows an HTTP request that performs a cross-tenant query across the foo and bar tenants:

curl -H "X-Scope-OrgID: foo|bar" \
  "http://tempo-query:3200/api/traces/search?limit=20&query={ service.name = \"checkout\" }"

For tenant ID format, allowed characters, and reserved values, refer to Tenant IDs.

Security considerations

The X-Scope-OrgID header should be set by a trusted reverse proxy, not by client applications. Tempo assumes that clients setting X-Scope-OrgID are trusted, and the responsibility of populating this value should be handled by your authenticating reverse proxy.

Allowing clients to set the tenant ID directly can lead to unauthorized access to other tenants’ data. A malicious client could change the tenant ID in the header to access traces from other tenants.

For more information about setting up authentication and reverse proxies, refer to Manage authentication.

User-configurable overrides

Thu, 28 May 2026 17:50:33 +0100

User-configurable overrides

User-configurable overrides in Tempo let you change overrides for your tenant using an API. Instead of modifying a file or Kubernetes configmap, you (and other services relying on Tempo) can use this API to modify the overrides directly.

Architecture

User-configurable overrides are stored in an object store bucket managed by Tempo.

Note
We recommend using a different bucket for overrides and traces storage, but they can share a bucket if needed. When sharing a bucket, make sure any lifecycle rules are scoped correctly to not remove data of user-configurable overrides module.

Overrides of every tenant are stored at /{tenant name}/overrides.json:

overrides/
├── 1/
│   └── overrides.json
└── 2/
    └── overrides.json

Tempo regularly polls this bucket and keeps a copy of the limits in-memory. When requesting the overrides for a tenant, the overrides module:

Checks this override is set in the user-configurable overrides, if so return that value.
Checks if this override is set in the runtime configuration (configmap), if so return that value.
Returns the default value.

Supported fields

User-configurable overrides are designed to be a subset of the runtime overrides. Refer to Overrides for information about all overrides. When you set a field in both the user-configurable overrides and the runtime overrides, the value from the user-configurable overrides takes priority.

Note
In Tempo 2.x, metrics_generator.processors user-configurable override field is OR-merged with the runtime overrides list.

In Tempo 3.0, if processors is set in user-configurable overrides, that will override the runtime metrics_generator.processors config. Setting processors: [] in user-configurable overrides disables all processors for the tenant.

Warning
The local-blocks processor was removed in Tempo 3.0. TraceQL metrics queries on recent data are now served by the live-store instead. If your overrides reference local-blocks, remove it before upgrading. For details, refer to Metrics generator.

[forwarders: <list of strings>]

cost_attribution:
  [dimensions: <map string to string>]

metrics_generator:

  [processors: <list of strings>]
  [collection_interval: <duration>]
  [trace_id_label_name: <string>]
  [ingestion_time_range_slack: <duration>]
  [disable_collection: <bool> | default = false]
  [generate_native_histograms: <classic|native|both> | default = classic]
  [native_histogram_max_bucket_number: <int> | default = 100]
  [native_histogram_bucket_factor: <float> | default = 1.1]
  [native_histogram_min_reset_duration: <duration> | default = 15m]
  [span_name_sanitization: <string>]

  processor:

    service_graphs:
      [histogram_buckets: <list of float>]
      [dimensions: <list of string>]
      [peer_attributes: <list of string>]
      [enable_client_server_prefix: <bool>]
      [enable_messaging_system_latency_histogram: <bool>]
      [enable_virtual_node_label: <bool>]
      [span_multiplier_key: <string>]
      [enable_tracestate_span_multiplier: <bool>]
      [filter_policies: [
        [
          include/include_any/exclude:
            match_type: <string> # options: strict, regex
            attributes:
              - key: <string>
                value: <any>
        ]
      ]]
    span_metrics:
      [histogram_buckets: <list of float>]
      [dimensions: <list of string>]
      [dimension_mappings: <list of mappings {name: <string>, source_labels: <list of string>, join: <string>}>]
      [intrinsic_dimensions: <map string to bool>]
      [filter_policies: [
        [
          include/include_any/exclude:
            match_type: <string> # options: strict, regex
            attributes:
              - key: <string>
                value: <any>
        ]
      ]]
      [enable_target_info: <bool>]
      [target_info_excluded_dimensions: <list of string>]
      [enable_instance_label: <bool>]
      [span_multiplier_key: <string>]
      [enable_tracestate_span_multiplier: <bool>]

    host_info:
      [host_identifiers: <list of string>]
      [metric_name: <string>]

Filter policy validation

Filter policies submitted through this API are fully validated. Tempo checks that attribute keys are valid TraceQL identifiers, only resource and span scopes are supported, regular expression patterns compile, and intrinsic values (kind, status) use recognized values. Invalid policies are rejected with a descriptive error.

For the complete list of validation rules and supported values, refer to Filtering.

API

All API requests are handled on the /api/overrides endpoint. The module supports GET, POST, PATCH, and DELETE requests. If you set http_api_prefix in your Tempo configuration, prepend it to the path (for example, /tempo/api/overrides).

This endpoint is tenant-specific. If Tempo is run in multitenant mode, all requests should have an appropriate X-Scope-OrgID header.

If the tenant is run in distributed mode, only the query-frontend will accept API requests.

Operations

GET /api/overrides

Returns the current overrides and its version.

Query-parameters:

scope: whether to return overrides from the API only api or merge it with the runtime overrides merged. Defaults to api.

Example:

curl -X GET -v -H "X-Scope-OrgID: 3" http://localhost:3100/api/overrides\?scope=merged

POST /api/overrides

Update the overrides with the given payload. Note this overwrites any existing overrides.

Example:

curl -X POST -v -H "X-Scope-OrgID: 3" -H "If-Match: 1697726795401423" http://localhost:3100/api/overrides --data "{}"

PATCH /api/overrides

Update the existing overrides by patching it with the payload. It follows the JSON merge patch protocol (RFC 7386).

Example:

curl -X PATCH -v -H "X-Scope-OrgID: 3" http://localhost:3100/api/overrides --data "{\"forwarders\":null}"

DELETE /api/overrides

Delete the existing overrides.

Example:

curl -X DELETE -H "X-Scope-OrgID: 3" -H "If-Match: 1697726795401423" http://localhost:3100/api/overrides

Versioning

To handle concurrent read and write operations, the backend stores the overrides with a version. Whenever the overrides are returned, the response has an Etag header with the current version.

$ curl -v http://localhost:3100/api/overrides
...
< HTTP/1.1 200 OK
< Content-Type: application/json
< Etag: 1697726795401423
< Date: Wed, 07 Feb 2024 17:49:04 GMT
< Content-Length: 118
...

Requests that modify or delete overrides need to pass the current version using the If-Match header:

curl -X POST -H "If-Match: 1697726795401423" http://localhost:3100/api/overrides --data "..."

This example uses overrides in the overrides.json file with the location in pwd:

curl -X POST -H "X-Scope-OrgID: 3" -H "If-Match: 1697726795401423" http://localhost:3100/api/overrides --data @overrides.json

If the version doesn’t match the version in the backend, the request is rejected with HTTP error 412.

Conflicting runtime overrides check

Overrides set through the user-configurable overrides take priority over runtime overrides. This can lead to misleading scenarios because a value set in the runtime overrides is not actively being used.

To warn users about preexisting runtime overrides, there is an optional check for conflicting runtime overrides. If enabled requests are rejected if:

There are no user-configurable overrides yet for this tenant.
There are runtime overrides set that contain overrides present in the user-configurable overrides.

The check can be enabled in the configuration:

overrides:
  user_configurable_overrides:
    api:
      check_for_conflicting_runtime_overrides: true

You can bypass this check by setting the query parameter skip-conflicting-overrides-check=true:

curl -X POST -H "If-Match: 1697726795401423" http://localhost:3100/api/overrides?skip-conflicting-overrides-check=true --data "..."

Generic forwarding

Thu, 28 May 2026 17:50:33 +0100

Generic forwarding

Generic forwarding allows asynchronous replication of ingested traces. The distributor writes received spans to both Kafka and defined endpoints, if enabled. This feature works in a best-effort manner, meaning that no retries happen if an error occurs during replication.

Warning
Generic forwarding doesn’t work retroactively. After it’s enabled, the distributor only replicates freshly ingested spans.

Configure generic forwarding

Enabling generic forwarding requires the configuration of the distributor and overrides.

Define a list of forwarders in the distributor section. Each forwarder must specify a unique name, supported backend, and backend-specific configuration.
Reference these forwarders in the overrides section. This allows for fine-grained control over forwarding and makes it possible to enable this feature globally or on a per-tenant basis.

For a detailed view of all the configuration options for the generic forwarding feature, refer to distributor and overrides configuration pages.

Tune the consistent hash rings

Thu, 28 May 2026 17:50:33 +0100

Tune the consistent hash rings

Tempo uses the Consistent Hash Ring implementation from Cortex. By default, the ring is gossiped between all Tempo components. However, it can be configured to use Consul or Etcd, if desired.

Tempo uses several consistent hash rings. Each hash ring exists for a distinct reason.

Distributor

Participants: Distributors

Used by: Distributors

Unless you are running with limits, this ring does not impact Tempo operation.

This ring is only used when global rate limits are used. The distributors use it to count the other active distributors. Incoming traffic is assumed to be evenly spread across all distributors and (global_rate_limit / # of distributors) is used to rate limit locally.

Live-store (partition ring)

Participants: Live-stores

Used by: Distributors, Queriers, Block-builders

The partition ring tracks which Tempo partitions are active and which live-stores own them. Distributors use this ring to determine which partitions to write to when sending data to Kafka. Queriers use it to find the live-stores for querying recent traces. Block-builders use it to determine which partitions to consume from.

Backend-worker

Participants: Backend-workers

Used by: Backend-workers

The backend-worker ring shards tenant index writing across backend-workers. This ring is only active when the backend-worker is configured with an external KV store for ring membership.

Metrics-generator (partition ring)

Participants: Metrics-generators

Used by: Metrics-generators

In microservices mode, the metrics-generator partition ring tracks which generator instances own which partitions. This ring is only active when metrics_generator.ring_mode is set to generator.

Interacting with the rings

Web pages are available at the following endpoints. They show every ring member, their tokens, and include the ability to “Forget” a ring member. “Forgetting” is useful when a ring member leaves the ring without properly shutting down, and therefore leaves its tokens in the ring.

To access a ring page, send a GET request to the Tempo HTTP API. By default, Tempo listens on port 3200 (configured with server.http_listen_port). For example:

http://<tempo-host>:3200/live-store/ring

In single-binary mode, any enabled ring endpoints are available on the same host. In microservices mode, each endpoint is available on the component listed in the Available on field.

Distributor

Available on: Distributors

Path: /distributor/ring

Note
This endpoint is only available when Tempo is configured with the global ingestion rate strategy.

Unhealthy distributors have little impact but should be forgotten to reduce the cost of maintaining the ring.

Live-store (partition ring)

Available on: Distributors, Queriers, Live-stores

Path: /partition-ring

The partition ring shows partition ownership across live-stores. Unhealthy live-stores may cause recent data queries to degrade.

Live-store

Available on: Distributors, Queriers, Live-stores

Path: /live-store/ring

Backend-worker

Available on: Backend-workers

Path: /backend-worker/ring

Note
This endpoint is only available when backend_worker.ring.kvstore.store is set to a non-empty value other than inmemory (for example, memberlist, consul, or etcd).

The backend-worker ring page shows how tenant index writing is distributed across workers. Forget unhealthy workers so that sharding redistributes correctly.

Metrics-generator (partition ring)

Available on: Metrics-generators

Path: /partition/ring

Note
This endpoint is only available in microservices mode when metrics_generator.ring_mode is set to generator.

The metrics-generator partition ring shows partition ownership across generator instances.

Configuring the rings

Ring/Lifecycler configuration control how a component interacts with the ring. Refer to the configuration topic for more details.

Zone-aware replication for live-stores

Thu, 28 May 2026 17:50:33 +0100

Zone-aware replication for live-stores

Zone awareness is a feature that ensures data is replicated across failure domains (which we refer to as “zones”) to provide greater reliability. A failure domain is whatever you define it to be, but commonly may be an availability zone, data center, or server rack.

When zone awareness is set up for live-stores, each Tempo partition is owned by one live-store per zone. This means that if a live-store in one zone becomes unavailable, the live-store in the other zone can continue serving queries for that partition. While data is replicated across zones (RF2), the read quorum is 1. The queriers only need a response from one live-store per partition. This provides high availability without requiring data deduplication on the read path.