Network options on Grafana Labs

Configure IPv6

Thu, 28 May 2026 17:50:33 +0100

Configure IPv6

Tempo can be configured to communicate between the components using Internet Protocol Version 6, or IPv6.

Note
The underlying infrastructure must support this address family. This configuration may be used in a single-stack IPv6 environment, or in a dual-stack environment where both IPv6 and IPv4 are present. In a dual-stack scenario, only one address family may be configured at a time, and all components must be configured for that address family.

Protocol configuration

This sample listen configuration will allow the gRPC and HTTP servers to listen on IPv6, and configure the various memberlist components to enable IPv6.

memberlist:
  bind_addr:
    - '::'
  bind_port: 7946

# Required only when using the global ingestion rate strategy.
distributor:
  ring:
    enable_inet6: true

# Only needed if the backend-worker ring is enabled (kvstore.store is set).
backend_worker:
  ring:
    enable_inet6: true

metrics_generator:
  ring:
    enable_inet6: true

live_store:
  ring:
    enable_inet6: true

server:
  grpc_listen_address: '::0'
  grpc_listen_port: 9095
  http_listen_address: '::0'
  http_listen_port: 3200

Note
The live_store.partition_ring doesn’t expose enable_inet6. It inherits its address family from the top-level memberlist configuration.

Kubernetes service configuration

Each service fronting the workloads needs to be configured with spec.ipFamilies and spec.ipFamilyPolicy set. Refer to this backend-worker example:

apiVersion: v1
kind: Service
metadata:
  labels:
    name: backend-worker
  name: backend-worker
  namespace: tracing
spec:
  clusterIP: fccb::31a7
  clusterIPs:
    - fccb::31a7
  internalTrafficPolicy: Cluster
  ipFamilies:
    - IPv6
  ipFamilyPolicy: SingleStack
  ports:
    - name: backend-worker-http-metrics
      port: 3200
      protocol: TCP
      targetPort: 3200
  selector:
    app: backend-worker
    name: backend-worker
  sessionAffinity: None
  type: ClusterIP

In addition to the per-component services, Tempo components discover each other through a separate headless gossip-ring Service that exposes the memberlist port. Configure it as IPv6 so that components can join the cluster over IPv6.

apiVersion: v1
kind: Service
metadata:
  name: gossip-ring
  namespace: tracing
spec:
  clusterIP: None
  ipFamilies:
    - IPv6
  ipFamilyPolicy: SingleStack
  ports:
    - name: gossip-ring
      port: 7946
      protocol: TCP
      targetPort: 7946
  selector:
    tempo-gossip-member: "true"

Each Tempo component then joins memberlist by referencing the headless Service’s DNS name in its memberlist.join_members configuration:

memberlist:
  join_members:
    - dns+gossip-ring.tracing.svc.cluster.local.:7946

Verify the listeners

The Tempo container image is distroless and doesn’t include a shell or networking utilities. You can’t exec into a Tempo Pod directly to inspect listeners.

To inspect listening sockets, attach an ephemeral debug container that shares the target Pod’s network namespace, using a debug image that includes the ss utility:

kubectl debug -n tracing -it <pod-name> \
  --image=<your-debug-image> \
  --target=backend-worker \
  -- ss -ltn

You should see IPv6 wildcard listeners on the memberlist, gRPC, and HTTP ports:

State   Recv-Q   Send-Q     Local Address:Port     Peer Address:Port
LISTEN  0        4096                   *:7946                *:*
LISTEN  0        4096                   *:9095                *:*
LISTEN  0        4096                   *:3200                *:*

The * in Local Address indicates the process is listening on all addresses for the configured family.

Configure TLS communication

Thu, 28 May 2026 17:50:33 +0100

Configure TLS communication

Tempo can be configured to communicate between components using Transport Layer Security, or TLS. TLS secures three categories of connections:

Server and client: Communication between internal Tempo components (for example, querier to query-frontend).
Receiver: Incoming trace data from instrumented applications or collectors to the distributor.
Storage and cache: Connections to backend object storage (S3) and caches (Memcached, Redis).

Note
The ciphers and TLS version here are for example purposes only. We aren’t recommending which ciphers or TLS versions to use in production environments.

Server configuration

Every Tempo component exposes gRPC and HTTP endpoints. Use the server block to enable TLS on these endpoints.

server:
  tls_cipher_suites: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
  tls_min_version: VersionTLS12

  grpc_tls_config:
    cert_file: /tls/tls.crt
    key_file: /tls/tls.key
    client_auth_type: VerifyClientCertIfGiven
    client_ca_file: /tls/ca.crt
  http_tls_config:
    cert_file: /tls/tls.crt
    key_file: /tls/tls.key
    client_auth_type: VerifyClientCertIfGiven
    client_ca_file: /tls/ca.crt

Valid values for the client_auth_type are documented in the standard crypto/tls package under ClientAuthType.

Client configuration

Several Tempo components configure gRPC clients to communicate with other components. For example, the querier contacts the query-frontend to request work. If the server endpoint uses TLS, the corresponding client must also enable TLS.

Tempo uses a standard grpc_client_config stanza for each of these client connections.

You can optionally omit tls_min_version, tls_cipher_suites, and tls_insecure_skip_verify. Whether tls_server_name is required depends on your environment.

grpc_client_config:
  tls_enabled: true
  tls_cert_path: /tls/tls.crt
  tls_key_path: /tls/tls.key
  tls_ca_path: /tls/ca.crt
  tls_server_name: tempo.trace.svc.cluster.local
  tls_insecure_skip_verify: false
  tls_cipher_suites: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
  tls_min_version: VersionTLS12

Set this configuration block at the following locations:

live_store_client.grpc_client_config
querier.frontend_worker.grpc_client_config
backend_scheduler_client.grpc_client_config

Additionally, memberlist must also be configured, but the client configuration is nested directly under memberlist as follows. The same configuration options are available as above.

memberlist:
    tls_enabled: true
    tls_cert_path: /tls/tls.crt
    tls_key_path: /tls/tls.key
    tls_ca_path: /tls/ca.crt
    tls_server_name: tempo.trace.svc.cluster.local
    tls_insecure_skip_verify: false

Receiver TLS

Receiver TLS secures the connection between trace sources, such as instrumented applications, OpenTelemetry Collectors, or Grafana Alloy, and the Tempo distributor. The receiver configuration uses the OpenTelemetry Collector TLS settings. For the full set of options, refer to the upstream OTLP receiver TLS documentation and the configtls package reference.

TLS fields

The following fields are available in the receiver tls block:

Field	Required	Description
`cert_file`	Yes	Path to the TLS certificate file.
`key_file`	Yes	Path to the TLS private key file.
`ca_file`	No	Path to a CA certificate bundle used by a TLS client to verify the server’s certificate. Use this field on the sending side (for example, in an Alloy exporter or Kafka client) to trust a custom or internal CA. On receivers, this field has no effect because receivers don’t initiate outbound TLS connections. If omitted, the system root CA pool is used.
`client_ca_file`	No	Path to the CA certificate used by a TLS server to verify client certificates. When set on a receiver, it requires connecting clients to present a valid certificate signed by this CA (mTLS). Use this field to enable mTLS on receivers.
`min_version`	No	Minimum TLS version to accept, for example `"1.2"` or `"1.3"`.

Configure server-only TLS

For server-only TLS, the distributor presents its certificate and clients verify it. Only cert_file and key_file are required:

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          tls:
            cert_file: /tls/tls.crt
            key_file: /tls/tls.key
            min_version: "1.2"

Configure mutual TLS (mTLS)

For mutual TLS, both the server and client present certificates. Add client_ca_file so the distributor requires and verifies client certificates:

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          tls:
            cert_file: /tls/tls.crt
            key_file: /tls/tls.key
            client_ca_file: /tls/ca.crt
            min_version: "1.2"

Supported receiver TLS paths

You can set a tls block on the following receiver configurations:

distributor.receivers.otlp.protocols.grpc.tls
distributor.receivers.otlp.protocols.http.tls
distributor.receivers.kafka.tls (client TLS for connecting to Kafka brokers)
distributor.receivers.zipkin.tls
distributor.receivers.jaeger.protocols.grpc.tls
distributor.receivers.jaeger.protocols.thrift_http.tls

Note
The Kafka receiver TLS block configures client TLS for the connection to Kafka brokers, unlike the other receiver TLS blocks which configure server TLS for incoming connections.

Configure the sending side

When you enable TLS on the Tempo receiver, you must also configure TLS on the client that sends traces. The following example shows a matching Grafana Alloy configuration for an OTLP gRPC exporter with server-only TLS:

otelcol.exporter.otlp "tempo" {
  client {
    endpoint = "tempo.trace.svc.cluster.local:4317"
    tls {
      ca_file = "/tls/ca.crt"
    }
  }
}

For mTLS, include the client certificate and key:

otelcol.exporter.otlp "tempo" {
  client {
    endpoint = "tempo.trace.svc.cluster.local:4317"
    tls {
      ca_file   = "/tls/ca.crt"
      cert_file = "/tls/tls.crt"
      key_file  = "/tls/tls.key"
    }
  }
}

Storage and cache TLS

S3 and S3-compatible storage

If you use a self-managed S3-compatible backend (for example, MinIO) with a custom CA or client certificates, configure TLS on the S3 storage backend. The TLS fields are set inline under storage.trace.s3:

storage:
  trace:
    backend: s3
    s3:
      bucket: tempo-traces
      endpoint: minio.example.com:9000
      tls_ca_path: /tls/ca.crt
      tls_cert_path: /tls/tls.crt
      tls_key_path: /tls/tls.key
      tls_server_name: minio.example.com
      tls_insecure_skip_verify: false
      tls_min_version: VersionTLS12

GCS and Azure Blob Storage rely on their respective SDK defaults for TLS and don’t expose separate TLS fields in the Tempo configuration.

Redis cache

If you use Redis as a cache backend, enable TLS with the tls_enabled field. Redis TLS has limited configuration. It supports tls_enabled and tls_insecure_skip_verify but doesn’t expose CA or client certificate path fields:

cache:
  caches:
    - redis:
        endpoint: redis.example.com:6380
        tls_enabled: true
        tls_insecure_skip_verify: false
      roles:
        - parquet-footer
        - bloom
        - frontend-search

Gateway and ingress TLS

Tempo doesn’t include a built-in gateway component. If you end TLS at an ingress controller, load balancer, or reverse proxy in front of Tempo, configure TLS on that component rather than in the Tempo configuration. When TLS is ended at the ingress, Tempo receivers don’t need TLS configured for internal connections.

Configure TLS with Helm

To configure TLS with the Helm chart, you must have a TLS key-pair and CA certificate stored in a Kubernetes secret. The following example mounts a secret called tempo-distributed-tls into the pods at /tls and modifies the configuration of Tempo to use the files. In this example, the Tempo components share a single TLS certificate. The tls_server_name configuration must match the certificate.

distributor:
  extraVolumeMounts:
    - mountPath: /tls
      name: tempo-distributed-tls
  extraVolumes:
    - name: tempo-distributed-tls
      secret:
        secretName: tempo-distributed-tls
blockBuilder:
  extraVolumeMounts:
    - mountPath: /tls
      name: tempo-distributed-tls
  extraVolumes:
    - name: tempo-distributed-tls
      secret:
        secretName: tempo-distributed-tls
liveStore:
  extraVolumeMounts:
    - mountPath: /tls
      name: tempo-distributed-tls
  extraVolumes:
    - name: tempo-distributed-tls
      secret:
        secretName: tempo-distributed-tls
backendScheduler:
  extraVolumeMounts:
    - mountPath: /tls
      name: tempo-distributed-tls
  extraVolumes:
    - name: tempo-distributed-tls
      secret:
        secretName: tempo-distributed-tls
backendWorker:
  extraVolumeMounts:
    - mountPath: /tls
      name: tempo-distributed-tls
  extraVolumes:
    - name: tempo-distributed-tls
      secret:
        secretName: tempo-distributed-tls
memcached:
  extraArgs:
    - -Z
    - -o
    - ssl_chain_cert=/tls/tls.crt,ssl_key=/tls/tls.key
  extraVolumeMounts:
    - mountPath: /tls
      name: tempo-distributed-tls
  extraVolumes:
    - name: tempo-distributed-tls
      secret:
        secretName: tempo-distributed-tls
metricsGenerator:
  extraVolumeMounts:
    - mountPath: /tls
      name: tempo-distributed-tls
  extraVolumes:
    - name: tempo-distributed-tls
      secret:
        secretName: tempo-distributed-tls
querier:
  extraVolumeMounts:
    - mountPath: /tls
      name: tempo-distributed-tls
  extraVolumes:
    - name: tempo-distributed-tls
      secret:
        secretName: tempo-distributed-tls
queryFrontend:
  extraVolumeMounts:
    - mountPath: /tls
      name: tempo-distributed-tls
  extraVolumes:
    - name: tempo-distributed-tls
      secret:
        secretName: tempo-distributed-tls
tempo:
  readinessProbe:
    httpGet:
      scheme: HTTPS
  structuredConfig:
    memberlist:
      tls_ca_path: /tls/ca.crt
      tls_cert_path: /tls/tls.crt
      tls_enabled: true
      tls_key_path: /tls/tls.key
      tls_server_name: tempo-distributed.trace.svc.cluster.local
    distributor:
      receivers:
        otlp:
          protocols:
            grpc:
              tls:
                ca_file: /tls/ca.crt
                cert_file: /tls/tls.crt
                key_file: /tls/tls.key
    live_store_client:
      grpc_client_config:
        tls_ca_path: /tls/ca.crt
        tls_cert_path: /tls/tls.crt
        tls_enabled: true
        tls_key_path: /tls/tls.key
        tls_server_name: tempo-distributed.trace.svc.cluster.local
    backend_scheduler_client:
      grpc_client_config:
        tls_ca_path: /tls/ca.crt
        tls_cert_path: /tls/tls.crt
        tls_enabled: true
        tls_key_path: /tls/tls.key
        tls_server_name: tempo-distributed.trace.svc.cluster.local
    cache:
      caches:
        - memcached:
            consistent_hash: true
            host: tempo-distributed-memcached
            service: memcached-client
            timeout: 500ms
            tls_ca_path: /tls/ca.crt
            tls_cert_path: /tls/tls.crt
            tls_enabled: true
            tls_key_path: /tls/tls.key
            tls_server_name: tempo-distributed.trace.svc.cluster.local
          roles:
            - parquet-footer
            - bloom
            - frontend-search
    querier:
      frontend_worker:
        grpc_client_config:
          tls_ca_path: /tls/ca.crt
          tls_cert_path: /tls/tls.crt
          tls_enabled: true
          tls_key_path: /tls/tls.key
          tls_server_name: tempo-distributed.trace.svc.cluster.local
    server:
      grpc_tls_config:
        cert_file: /tls/tls.crt
        client_auth_type: VerifyClientCertIfGiven
        client_ca_file: /tls/ca.crt
        key_file: /tls/tls.key
      http_tls_config:
        cert_file: /tls/tls.crt
        client_auth_type: VerifyClientCertIfGiven
        client_ca_file: /tls/ca.crt
        key_file: /tls/tls.key
traces:
  otlp:
    grpc:
      enabled: true

Refer to the prometheus.scrape docs for Alloy to configure TLS on the scrape. A relabel configuration like the following does this configuration for you dynamically.

{
  source_labels: ['__meta_kubernetes_pod_annotation_prometheus_io_scheme'],
  action: 'replace',
  target_label: '__scheme__',
  regex: '(https?)',
  replacement: '$1',
},

Run Tempo distributed with sidecar proxies

Thu, 28 May 2026 17:50:33 +0100

Run Tempo distributed with sidecar proxies

You can route inter-pod gRPC traffic through a sidecar proxy to meet requirements such as custom security, routing, or logging. Common examples include Envoy, Nginx, Traefik, or service meshes like Istio and Linkerd.

This page applies only to the microservices deployment mode. In monolithic (single-binary) mode, components communicate in-process, so there’s no inter-pod gRPC traffic for a sidecar to proxy.

How Tempo pods communicate

In microservices mode, Tempo’s write path runs through Kafka and the read path uses gRPC. Distributors write to Kafka, and live-stores, block-builders, and metrics-generators each consume from Kafka independently. Queriers contact live-stores over gRPC for recent data.

For background on how each component communicates, refer to:

Deployment modes for the overall network model.
Live-store and Querier for the gRPC read path that this page covers.

This page focuses on the querier-to-live-store gRPC traffic, which a sidecar proxy can wrap. Queriers discover live-stores through a shared hash ring: live-stores register their address and port in the ring at startup and deregister on exit.

The overall network looks like this:

You can view the low-level ring data by browsing to the /live-store/ring URL on any component that imports the live-store ring, for example a distributor or querier. It looks like this:

By default, gRPC traffic uses port 9095, but you can change it by customizing server.grpc_listen_port on each pod that needs it. This setting changes the gRPC port for the entire process, so make sure any other components that connect to the pod, for example queriers connecting to a query-frontend, use the new port too.

server:
  grpc_listen_port: 12345

The ring contents reflect the new port:

Run Tempo with proxies

Some installations require that the inter-pod gRPC traffic runs through a sidecar proxy. Running Tempo with proxies requires setting two ports for the live-store: one for the live-store process and one for the sidecar. Additionally, the live-store ring contents must reflect the proxy’s port so that traffic from queriers goes through the proxy.

The overall network looks like this:

You can’t accomplish this by setting the same grpc_listen_port as in the previous example. Instead, the live-store needs to listen on port A but advertise itself on port B. To do this, customize the live-store’s ring instance port:

live_store:
  ring:
    instance_port: 12345

The live-store now listens for regular traffic on port 9095, but queriers route traffic to it on port 12345.

Other components

Only the live-store ring’s instance_port is relevant for this sidecar proxy pattern. Other components have their own rings, but those rings aren’t used by queriers for gRPC routing:

The metrics-generator consumes trace data from Kafka in microservices mode, so distributors don’t reach it through a hash ring. Refer to the metrics-generator architecture.
Distributors and block-builders also use Kafka for the write path. Refer to the distributor and partition ring documentation.
For other gRPC connections, such as querier-to-query-frontend, set server.grpc_listen_port on the destination pod and update the matching client address on the calling component.