Manage advanced systems on Grafana Labs

Enable multi-tenancy

Wed, 08 Apr 2026 02:52:41 +0000

Enable multi-tenancy

Tempo is a multi-tenant distributed tracing backend. Tempo uses the X-Scope-OrgID header to enforce multi-tenancy in Tempo and Grafana Enterprise Traces. It is set to the tenant (or “organization”) name. It is used for scoped writes (ingest) so that each span is stored under its specified tenant, and scoped reads so that queries return only that tenant’s data.

If you’re interested in setting up multi-tenancy, consult the multi-tenant example in the repository. This example uses the following settings to achieve multi-tenancy in Tempo.

Multi-tenancy on ingestion is supported with GPRC and HTTP for OTLP. You can add headers both in the OpenTelemetry Collector and Grafana Alloy or using curl or any other relevant HTTP/gRPC protocol tool. Here is an example configuration for Alloy.

otelcol.exporter.otlphttp "tempo" {
    // Define the client for exporting.
    client {
        // Send the X-Scope-OrgID header to the Tempo instance for multi-tenancy (tenant 1234).
        headers = {
            "X-Scope-OrgID" = "1234",
        }

        // Send to the locally running Tempo instance, on port 4317 (OTLP gRPC).
        endpoint = "http://tempo:4318"

        // Configure TLS settings for communicating with the endpoint.
        tls {
            // The connection is insecure.
            insecure = true
            // Do not verify TLS certificates when connecting.
            insecure_skip_verify = true
        }
    }
}

Configure multi-tenancy

Configure the OTEL Collector to attach the X-Scope-OrgID header on push:
```
exporters:
  otlp:
    headers:
      x-scope-orgid: foo-bar-baz
```

Configure the Tempo data source in Grafana to pass the tenant with the same header:

- name: Tempo-Multitenant
  jsonData:
    httpHeaderName1: 'X-Scope-OrgID'
  secureJsonData:
    httpHeaderValue1: 'foo-bar-baz'

Enable multi-tenancy on the Tempo backend by setting the following configuration value on all Tempo components:
YAML
```
multitenancy_enabled: true
```
or from the command line:
YAML
```
--multitenancy.enabled=true
```
This option forces all Tempo components to require the X-Scope-OrgID header.

Cross-tenant query federation

Wed, 08 Apr 2026 02:52:41 +0000

Cross-tenant query federation

Note
You need to enable multitenancy_enabled: true in the cluster for multi-tenant querying to work. Refer to Enable multi-tenancy for more details and implications of multitenancy_enabled: true.

Tempo supports multi-tenant queries for search, search-tags, and trace-by-ID search operations.

To perform multi-tenant queries, send tenant IDs separated by a | character in the X-Scope-OrgID header, for example, foo|bar.

By default, cross-tenant query is enabled and can be controlled using multi_tenant_queries_enabled configuration setting.

query_frontend:
   multi_tenant_queries_enabled: true

For more information on configuration options, refer to Enable multitenancy.

TraceQL queries

Queries performed using the cross-tenant configured data source, in either Explore or inside of dashboards, are performed across all the tenants that you specified in the X-Scope-OrgID header.

TraceQL queries that compare multiple spansets may not correctly return all traces in a cross-tenant query. For instance,

{ span.attr1 = "bar" } && { span.attr2 = "foo" }

TraceQL evaluates a contiguously stored trace. If these two conditions are satisfied in separate tenants, then Tempo doesn’t correctly return the trace.

User-configurable overrides

Wed, 08 Apr 2026 02:52:41 +0000

User-configurable overrides

User-configurable overrides in Tempo let you change overrides for your tenant using an API. Instead of modifying a file or Kubernetes configmap, you (and other services relying on Tempo) can use this API to modify the overrides directly.

Architecture

User-configurable overrides are stored in an object store bucket managed by Tempo.

Note
We recommend using a different bucket for overrides and traces storage, but they can share a bucket if needed. When sharing a bucket, make sure any lifecycle rules are scoped correctly to not remove data of user-configurable overrides module.

Overrides of every tenant are stored at /{tenant name}/overrides.json:

overrides/
├── 1/
│   └── overrides.json
└── 2/
    └── overrides.json

Tempo regularly polls this bucket and keeps a copy of the limits in-memory. When requesting the overrides for a tenant, the overrides module:

Checks this override is set in the user-configurable overrides, if so return that value.
Checks if this override is set in the runtime configuration (configmap), if so return that value.
Returns the default value.

Supported fields

User-configurable overrides are designed to be a subset of the runtime overrides. Refer to Overrides for information about all overrides. When you set a field in both the user-configurable overrides and the runtime overrides, the value from the user-configurable overrides takes priority.

Note
Note that processors is an exception. Tempo merges values from both user-configurable overrides and runtime overrides into a single list.

[forwarders: <list of strings>]

metrics_generator:

  [processors: <list of strings>]
  [collection_interval: <duration>]
  [disable_collection: <bool> | default = false]

  processor:

    service_graphs:
      [histogram_buckets: <list of float>]
      [dimensions: <list of string>]
      [peer_attributes: <list of string>]
      [enable_client_server_prefix: <bool>]
      [enable_messaging_system_latency_histogram: <bool>]

    span_metrics:
      [histogram_buckets: <list of float>]
      [dimensions: <list of string>]
      [intrinsic_dimensions: <map string to bool>]
      [filter_policies: [
        [
          include/exclude:
            match_type: <string> # options: strict, regexp
            attributes:
              - key: <string>
                value: <any>
        ]
      ]
      [enable_target_info: <bool>]
      [target_info_excluded_dimensions: <list of string>]
      [enable_instance_label: <bool>]
      [enable_virtual_node_label: <string>]

    host_info:
      [host_identifiers: <list of string>]
      [metric_name: <string>]

API

All API requests are handled on the /api/overrides endpoint. The module supports GET, POST, PATCH, and DELETE requests.

This endpoint is tenant-specific. If Tempo is run in multitenant mode, all requests should have an appropriate X-Scope-OrgID header.

If the tenant is run in distributed mode, only the query-frontend will accept API requests.

Operations

GET /api/overrides

Returns the current overrides and it’s version.

Query-parameters:

scope: whether to return overrides from the API only api or merge it with the runtime overrides merged. Defaults to api.

Example:

curl -X GET -v -H "X-Scope-OrgID: 3" http://localhost:3100/tempo/api/overrides\?scope=merged

POST /api/overrides

Update the overrides with the given payload. Note this overwrites any existing overrides.

Example:

curl -X POST -v -H "X-Scope-OrgID: 3" -H "If-Match: 1697726795401423" http://localhost:3100/api/overrides --data "{}"

PATCH /api/overrides

Update the existing overrides by patching it with the payload. It follows the JSON merge patch protocol (RFC 7386).

Example:

curl -X PATCH -v -H "X-Scope-OrgID: 3" http://localhost:3100/api/overrides --data "{\"forwarders\":null}"

DELETE /api/overrides

Delete the existing overrides.

Example:

curl -X DELETE -H "X-Scope-OrgID: 3" -H "If-Match: 1697726795401423" http://localhost:3100/api/overrides

Versioning

To handle concurrent read and write operations, the backend stores the overrides with a version. Whenever the overrides are returned, the response has an Etag header with the current version.

$ curl -v http://localhost:3100/api/overrides
...
< HTTP/1.1 200 OK
< Content-Type: application/json
< Etag: 1697726795401423
< Date: Wed, 07 Feb 2024 17:49:04 GMT
< Content-Length: 118
...

Requests that modify or delete overrides need to pass the current version using the If-Match header:

curl -X POST -H "If-Match: 1697726795401423" http://localhost:3100/api/overrides --data "..."

This example uses overrides in the overrides.json file with the location in pwd:

curl -X POST -H "X-Scope-OrgID: 3" -H "If-Match: 1697726795401423" http://localhost:3100/api/overrides --data @overrides.json

If the version doesn’t match the version in the backend, the request is rejected with HTTP error 412.

Conflicting runtime overrides check

Overrides set through the user-configurable overrides take priority over runtime overrides. This can lead to misleading scenarios because a value set in the runtime overrides is not actively being used.

To warn users about preexisting runtime overrides, there is an optional check for conflicting runtime overrides. If enabled requests are rejected if:

There are no user-configurable overrides yet for this tenant.
There are runtime overrides set that contain overrides present in the user-configurable overrides.

The check can be enabled in the configuration:

overrides:
  user_configurable_overrides:
    api:
      check_for_conflicting_runtime_overrides: true

You can bypass this check by setting the query parameter skip-conflicting-overrides-check=true:

curl -X POST -H "If-Match: 1697726795401423" http://localhost:3100/api/overrides?skip-conflicting-overrides-check=true --data "..."

Generic forwarding

Wed, 08 Apr 2026 02:52:41 +0000

Generic forwarding

Generic forwarding allows asynchronous replication of ingested traces. The distributor writes received spans to both the ingester and defined endpoints, if enabled. This feature works in a best-effort manner, meaning that no retries happen if an error occurs during replication.

Warning
Generic forwarding doesn’t work retroactively. After it’s enabled, the distributor only replicates freshly ingested spans.

Configure generic forwarding

Enabling generic forwarding requires the configuration of the distributor and overrides.

Define a list of forwarders in the distributor section. Each forwarder must specify a unique name, supported backend, and backend-specific configuration.
Reference these forwarders in the overrides section. This allows for fine-grained control over forwarding and makes it possible to enable this feature globally or on a per-tenant basis.

For a detailed view of all the configuration options for the generic forwarding feature, refer to distributor and overrides configuration pages.

Tune the consistent hash rings

Wed, 08 Apr 2026 02:52:41 +0000

Tune the consistent hash rings

Tempo uses the Consistent Hash Ring implementation from Cortex. By default, the ring is gossiped between all Tempo components. However, it can be configured to use Consul or Etcd, if desired.

There are four consistent hash rings: distributor, ingester, metrics-generator, and compactor. Each hash ring exists for a distinct reason.

Distributor

Participants: Distributors

Used by: Distributors

Unless you are running with limits, this ring does not impact Tempo operation.

This ring is only used when global rate limits are used. The distributors use it to count the other active distributors. Incoming traffic is assumed to be evenly spread across all distributors and (global_rate_limit / # of distributors) is used to rate limit locally.

Ingester

Participants: Ingesters

Used by: Distributors, Queriers

This ring is used by the distributors to load balance traffic into the ingesters. When spans are received the trace id is hashed and they are sent to the appropriate ingesters based on token ownership in the ring. Queriers also use this ring to find the ingesters for querying recent traces.

Metrics-generator

Participants: Metrics-generators

Used by: Distributors, Queriers

This ring is used by distributors to load balance traffic to the metrics-generators. When spans are received, the trace ID is hashed, and the traces are sent to the appropriate metrics-generators based on token ownership in the ring. Queriers also use this ring to generate TraceQL metrics from recent traces.

Compactor

Participants: Compactors

Used by: Compactors

This ring is used by the compactors to shard compaction jobs. Jobs are hashed into the ring and the owning compactor is the only one allowed to compact a specific set of blocks to prevent race conditions on compaction.

Interacting with the rings

Web pages are available at the following endpoints. They show every ring member, their tokens and includes the ability to “Forget” a ring member. “Forgetting” is useful when a ring member leaves the ring without properly shutting down, and therefore leaves its tokens in the ring.

Distributor

Available on: Distributors

Path: /distributor/ring

Unhealthy distributors have little impact but should be forgotten to reduce cost of maintaining the ring .

Ingester

Available on: Distributors

Path: /ingester/ring

Unhealthy ingesters will cause writes to fail. If the ingester is really gone, forget it immediately.

Metrics-generators

Available on: Distributors

Path: /metrics-generator/ring

Unhealthy metrics-generators will cause writes to fail. If the metrics-generator is really gone, forget it immediately.

Compactor

Available on: Compactors

Path: /compactor/ring

Unhealthy compactors will allow the blocklist to grow significantly. If the compactor is really gone, forget it immediately.

Configuring the rings

Ring/Lifecycler configuration control how a component interacts with the ring. Refer to the configuration topic for more details.

Zone-aware replication for ingesters

Wed, 08 Apr 2026 02:52:41 +0000

Zone-aware replication for ingesters

Zone awareness is a feature that ensures data is replicated across failure domains (which we refer to as “zones”) to provide greater reliability. A failure domain is whatever you define it to be, but commonly may be an availability zone, data center, or server rack.

When zone awareness is enabled for ingesters, incoming trace data is guaranteed to be replicated to ingesters in different zones. This allows the system to withstand the loss of one or more zones (depending on the replication factor).

Example:

# use the following fields in _config field of JSonnet config, to enable zone-aware ingesters.
    multi_zone_ingester_enabled: false,
    multi_zone_ingester_migration_enabled: false,
    multi_zone_ingester_replicas: 0,
    multi_zone_ingester_max_unavailable: 25,

For an configuration, refer to the JSonnet microservices operations example