Network options on Grafana Labs

Configure IPv6

Thu, 09 Apr 2026 14:59:14 +0000

Configure IPv6

Tempo can be configured to communicate between the components using Internet Protocol Version 6, or IPv6.

Note
The underlying infrastructure must support this address family. This configuration may be used in a single-stack IPv6 environment, or in a dual-stack environment where both IPv6 and IPv4 are present. In a dual-stack scenario, only one address family may be configured at a time, and all components must be configured for that address family.

Protocol configuration

This sample listen configuration will allow the gRPC and HTTP servers to listen on IPv6, and configure the various memberlist components to enable IPv6.

memberlist:
  bind_addr:
    - '::'
  bind_port: 7946

compactor:
  ring:
    kvstore:
      store: memberlist
    enable_inet6: true

metrics_generator:
  ring:
    enable_inet6: true

ingester:
  lifecycler:
    enable_inet6: true

server:
  grpc_listen_address: '::0'
  grpc_listen_port: 9095
  http_listen_address: '::0'
  http_listen_port: 3200

Kubernetes service configuration

Each service fronting the workloads will need to be configured with with spec.ipFamilies and spec.ipFamilyPolicy set. See this compactor example.

apiVersion: v1
kind: Service
metadata:
  labels:
    name: compactor
  name: compactor
  namespace: trace
spec:
  clusterIP: fccb::31a7
  clusterIPs:
    - fccb::31a7
  internalTrafficPolicy: Cluster
  ipFamilies:
    - IPv6
  ipFamilyPolicy: SingleStack
  ports:
    - name: compactor-http-metrics
      port: 3200
      protocol: TCP
      targetPort: 3200
  selector:
    app: compactor
    name: compactor
  sessionAffinity: None
  type: ClusterIP

You can check the listening service from within a pod.

❯ k exec -it compactor-55c778b8d9-2kch2 -- sh
/ # apk add iproute2
OK: 12 MiB in 27 packages
/ # ss -ltn -f inet
State   Recv-Q   Send-Q     Local Address:Port     Peer Address:Port  Process
/ # ss -ltn -f inet6
State    Recv-Q   Send-Q     Local Address:Port     Peer Address:Port  Process
LISTEN   0        4096                   *:7946                *:*
LISTEN   0        4096                   *:9095                *:*
LISTEN   0        4096                   *:3200                *:*

Configure TLS communication

Thu, 09 Apr 2026 14:59:14 +0000

Configure TLS communication

Tempo can be configured to communicate between the components using Transport Layer Security, or TLS.

Note
The ciphers and TLS version here are for example purposes only. We are not recommending which ciphers or TLS versions for use in production environments.

Server configuration

This sample TLS server configuration shows supported options.

server:
  tls_cipher_suites: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
  tls_min_version: VersionTLS12

  grpc_tls_config:
    cert_file: /tls/tls.crt
    key_file: /tls/tls.key
    client_auth_type: VerifyClientCertIfGiven
    client_ca_file: /tls/ca.crt
  http_tls_config:
    cert_file: /tls/tls.crt
    key_file: /tls/tls.key
    client_auth_type: VerifyClientCertIfGiven
    client_ca_file: /tls/ca.crt

Valid values for the client_auth_type are documented in the standard crypt/tls package under ClientAuthType here.

Client configuration

Several components of Tempo need to configure the gRPC clients they use to communicate with other components. For example, when the querier contacts the query-frontend to request work, the client in use must enable TLS if the server is serving a TLS endpoint.

The Tempo configuration uses a standard configuration stanza for each of these client configurations. Below is an example of the configuration.

The optional configuration elements tls_min_version, tls_cipher_suites, and tls_insecure_skip_verify may be omitted. The option tls_server_name may or may not be required, depending on the environment.

grpc_client_config:
  tls_enabled: true
  tls_cert_path: /tls/tls.crt
  tls_key_path: /tls/tls.key
  tls_ca_path: /tls/ca.crt
  tls_server_name: tempo.trace.svc.cluster.local
  tls_insecure_skip_verify: false
  tls_cipher_suites: TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
  tls_min_version: VersionTLS12

The configuration block needs to be set at the following configuration locations.

ingester_client.grpc_client_config
metrics_generator_client.grpc_client_config
querier.query-frontend.grpc_client_config

Additionally, memberlist must also be configured, but the client configuration is nested directly under memberlist as follows. The same configuration options are available as above.

memberlist:
    tls_enabled: true
    tls_cert_path: /tls/tls.crt
    tls_key_path: /tls/tls.key
    tls_ca_path: /tls/ca.crt
    tls_server_name: tempo.trace.svc.cluster.local
    tls_insecure_skip_verify: false

Receiver TLS

Additional receiver configuration can be added to support TLS communication for traces being sent to Tempo. The receiver configuration is pulled in from the Open Telemetry collector, and is documented upstream here. Addition TLS configuration of OTEL components can be found here.

An example tls block might look like the following:

tls:
  ca_file: /tls/ca.crt
  cert_file: /tls/tls.crt
  key_file: /tls/tls.key
  min_version: "1.2"

The above structure can be set on the following receiver configurations:

distributor.receivers.otlp.protocols.grpc.tls
distributor.receivers.otlp.protocols.http.tls
distributor.receivers.zipkin.tls
distributor.receivers.jaeger.protocols.grpc.tls
distributor.receivers.jaeger.protocols.thrift_http.tls

Configure TLS with Helm

To configure TLS with the Helm chart, you must have a TLS key-pair and CA certificate stored in a Kubernetes secret. The following example mounts a secret called tempo-distributed-tls into the pods at /tls and modifies the configuration of Tempo to make use of the files. In this example, the Tempo components share a single TLS certificate. Note that the tls_server_name configuration must match the certificate.

compactor:
  extraVolumeMounts:
    - mountPath: /tls
      name: tempo-distributed-tls
  extraVolumes:
    - name: tempo-distributed-tls
      secret:
        secretName: tempo-distributed-tls
distributor:
  extraVolumeMounts:
    - mountPath: /tls
      name: tempo-distributed-tls
  extraVolumes:
    - name: tempo-distributed-tls
      secret:
        secretName: tempo-distributed-tls
ingester:
  extraVolumeMounts:
    - mountPath: /tls
      name: tempo-distributed-tls
  extraVolumes:
    - name: tempo-distributed-tls
      secret:
        secretName: tempo-distributed-tls
memcached:
  extraArgs:
    - -Z
    - -o
    - ssl_chain_cert=/tls/tls.crt,ssl_key=/tls/tls.key
  extraVolumeMounts:
    - mountPath: /tls
      name: tempo-distributed-tls
  extraVolumes:
    - name: tempo-distributed-tls
      secret:
        secretName: tempo-distributed-tls
metricsGenerator:
  extraVolumeMounts:
    - mountPath: /tls
      name: tempo-distributed-tls
  extraVolumes:
    - name: tempo-distributed-tls
      secret:
        secretName: tempo-distributed-tls
querier:
  extraVolumeMounts:
    - mountPath: /tls
      name: tempo-distributed-tls
  extraVolumes:
    - name: tempo-distributed-tls
      secret:
        secretName: tempo-distributed-tls
queryFrontend:
  extraVolumeMounts:
    - mountPath: /tls
      name: tempo-distributed-tls
  extraVolumes:
    - name: tempo-distributed-tls
      secret:
        secretName: tempo-distributed-tls
tempo:
  readinessProbe:
    httpGet:
      scheme: HTTPS
  structuredConfig:
    memberlist:
      tls_ca_path: /tls/ca.crt
      tls_cert_path: /tls/tls.crt
      tls_enabled: true
      tls_key_path: /tls/tls.key
      tls_server_name: tempo-distributed.trace.svc.cluster.local
    distributor:
      receivers:
        otlp:
          protocols:
            grpc:
              tls:
                ca_file: /tls/ca.crt
                cert_file: /tls/tls.crt
                key_file: /tls/tls.key
    ingester_client:
      grpc_client_config:
        tls_ca_path: /tls/ca.crt
        tls_cert_path: /tls/tls.crt
        tls_enabled: true
        tls_key_path: /tls/tls.key
        tls_server_name: tempo-distributed.trace.svc.cluster.local
    cache:
      caches:
        - memcached:
            consistent_hash: true
            host: tempo-distributed-memcached
            service: memcached-client
            timeout: 500ms
            tls_ca_path: /tls/ca.crt
            tls_cert_path: /tls/tls.crt
            tls_enabled: true
            tls_key_path: /tls/tls.key
            tls_server_name: tempo-distributed.trace.svc.cluster.local
          roles:
            - parquet-footer
            - bloom
            - frontend-search
    metrics_generator_client:
      grpc_client_config:
        tls_ca_path: /tls/ca.crt
        tls_cert_path: /tls/tls.crt
        tls_enabled: true
        tls_key_path: /tls/tls.key
        tls_server_name: tempo-distributed.trace.svc.cluster.local
    querier:
      frontend_worker:
        grpc_client_config:
          tls_ca_path: /tls/ca.crt
          tls_cert_path: /tls/tls.crt
          tls_enabled: true
          tls_key_path: /tls/tls.key
          tls_server_name: tempo-distributed.trace.svc.cluster.local
    server:
      grpc_tls_config:
        cert_file: /tls/tls.crt
        client_auth_type: VerifyClientCertIfGiven
        client_ca_file: /tls/ca.crt
        key_file: /tls/tls.key
      http_tls_config:
        cert_file: /tls/tls.crt
        client_auth_type: VerifyClientCertIfGiven
        client_ca_file: /tls/ca.crt
        key_file: /tls/tls.key
traces:
  otlp:
    grpc:
      enabled: true

Refer to the prometheus.scrape docs for Alloy to configure TLS on the scrape. A relabel configuration like the following will do this configuration for you dynamically.

{
  source_labels: ['__meta_kubernetes_pod_annotation_prometheus_io_scheme'],
  action: 'replace',
  target_label: '__scheme__',
  regex: '(https?)',
  replacement: '$1',
},

Run Tempo distributed with sidecar proxies

Thu, 09 Apr 2026 14:59:14 +0000

Run Tempo distributed with sidecar proxies

You can route inter-pod gRPC traffic run through a sidecar proxy to meet requirements such as custom security, routing, or logging. Common examples include Envoy, Nginx, Traefik, or service meshes like Istio and Linkerd.

How Tempo pods communicate

Tempo pods communicate using gRPC.

The different components like distributors and ingesters find each other by a shared ring with the list of pods, their roles, and their addresses. Pods advertise their address and listening port into the ring when they start, and deregister themselves when they exit.

The overall network looks like this:

The low-level ring data for ingesters can be viewed by browsing to the /ingester/ring URL on a distributor. It looks like this:

By default, gRPC traffic uses port 9095, but this can be changed by customizing the grpc_listen_port for each pod that needs it.

server:
  grpc_listen_port: 12345

The ring contents reflect the new port:

Run Tempo with proxies

Some installations require that the inter-pod gRPC traffic runs through a sidecar proxy. Running Tempo with proxies requires setting two ports for each pod: one for the Tempo process and one for the sidecar. Additionally, the ring contents must reflect the proxy’s port so that traffic from other pods goes through the proxy.

The overall network looks like this:

This cannot be accomplished by setting the same grpc_listen_port as in the previous example. Instead, we need the ingester to listen on port A but advertise itself on port B. This is done by customizing the ingester’s lifecycler port:

ingester:
   lifecycler:
       port: 12345

Now, the ingester is listening for regular traffic on port 9095, but the distributor will route traffic to it on port 12345.

Metrics-generator proxy

You can customize the lifecyler port in the metrics-generator. To set an instance port for the metrics-generator, use this configuration:

metrics_generator:
  ring:
    instance_port: 12345

Replace 12345 with the correct port number.