Deploy Tempo with Tempo Operator on Grafana Labs

Quickstart

Thu, 09 Apr 2026 14:59:14 +0000

Quickstart

One page summary on how to start with Tempo Operator and TempoStack.

Requirements

The easiest way to start with the Tempo Operator is to use Kubernetes kind.

Deploy

To install the operator in an existing cluster, make sure you have cert-manager installed and run:

kubectl apply -f https://github.com/grafana/tempo-operator/releases/latest/download/tempo-operator.yaml

Once you have the operator deployed you need to install a storage backend. For this quick start guide, we will install MinIO as follows:

kubectl apply -f https://raw.githubusercontent.com/grafana/tempo-operator/main/minio.yaml

After minio was deployed, create a secret for MinIO in the namespace you are using:

kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: minio-test
stringData:
  endpoint: http://minio.minio.svc:9000
  bucket: tempo
  access_key_id: tempo
  access_key_secret: supersecret
type: Opaque
EOF

Then create Tempo CR:

kubectl apply -f - <<EOF
apiVersion: tempo.grafana.com/v1alpha1
kind: TempoStack
metadata:
  name: simplest
spec:
  storage:
    secret:
      name: minio-test
      type: s3
  storageSize: 1Gi
  resources:
    total:
      limits:
        memory: 2Gi
        cpu: 2000m
  template:
    queryFrontend:
      jaegerQuery:
        enabled: true
EOF

After create the TempoStack CR, you should see a some pods on the namespace. Wait for the stack to stabilize.

The stack deployed above is configured to receive Jaeger, Zipkin, and OpenTelemetry (OTLP) protocols. Because the Jaeger Query is enabled, you can also use the Jaeger UI to inspect the data.

To do a quick test, deploy a Job that generates some traces.

kubectl apply -f - <<EOF
apiVersion: batch/v1
kind: Job
metadata:
  name: tracegen
spec:
  template:
    spec:
      containers:
        - name: tracegen
          image: ghcr.io/open-telemetry/opentelemetry-collector-contrib/tracegen:latest
          command:
            - "./tracegen"
          args:
            - -otlp-endpoint=tempo-simplest-distributor:4317
            - -otlp-insecure
            - -duration=30s
            - -workers=1
      restartPolicy: Never
  backoffLimit: 4
EOF

Forward the Jaeger Query port to see the traces:

kubectl port-forward svc/tempo-simplest-query-frontend 16686:16686

Visit http://localhost:16686 to view the results.

Object storage

Thu, 09 Apr 2026 14:59:14 +0000

Object storage

Tempo Operator supports AWS S3, Azure, GCS, Minio and OpenShift Data Foundation for TempoStack object storage.

AWS S3

Requirements

Create a bucket on AWS.

Static token installation

Create an Object Storage secret with keys as follows:

kubectl create secret generic tempostack-dev-s3 \
  --from-literal=bucket="<BUCKET_NAME>" \
  --from-literal=endpoint="<AWS_BUCKET_ENDPOINT>" \
  --from-literal=access_key_id="<AWS_ACCESS_KEY_ID>" \
  --from-literal=access_key_secret="<AWS_ACCESS_KEY_SECRET>"

where tempostack-dev-s3 is the secret name.

Create an instance of TempoStack by referencing the secret name and type as s3:

spec:
  storage:
    secret:
      name: tempostack-dev-s3
      type: s3

AWS Security Token Service (STS) installation

Create a custom AWS IAM Role associated with a trust relationship to Tempo’s Kubernetes ServiceAccount:

{
  'Version': '2012-10-17',
  'Statement':
    [
      {
        'Effect': 'Allow',
        'Principal': { 'Federated': 'arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_PROVIDER}' },
        'Action': 'sts:AssumeRoleWithWebIdentity',
        'Condition':
          {
            'StringEquals':
              {
                '${OIDC_PROVIDER}:sub':
                  [
                    'system:serviceaccount:${TEMPOSTACK_NS}:tempo-${TEMPOSTACK_NAME}',
                    'system:serviceaccount:${TEMPOSTACK_NS}:tempo-${TEMPOSTACK_NAME}-query-frontend',
                  ],
              },
          },
      },
    ],
}

Create an AWS IAM role:

aws iam create-role \
--role-name "tempo-s3-access" \
--assume-role-policy-document "file:///tmp/trust.json" \
--query Role.Arn \
--output text

Attach a specific policy to that role:

aws iam attach-role-policy \
--role-name "tempo-s3-access" \
--policy-arn "arn:aws:iam::aws:policy/AmazonS3FullAccess"

Create an Object Storage secret with keys as follows:

kubectl create secret generic tempostack-dev-s3 \
  --from-literal=bucket="<BUCKET_NAME>" \
  --from-literal=region="<AWS_REGION>" \
  --from-literal=role_arn="<ROLE ARN>"

where tempostack-dev-s3 is the secret name.

Create an instance of TempoStack by referencing the secret name and type as s3:

spec:
  storage:
    secret:
      name: tempostack-dev-s3
      type: s3

Azure

Requirements

Create a bucket on Azure.

Installation

Create an Object Storage secret with keys as follows:

kubectl create secret generic tempostack-dev-azure \
  --from-literal=container="<AZURE_CONTAINER_NAME>" \
  --from-literal=account_name="<AZURE_ACCOUNT_NAME>" \
  --from-literal=account_key="<AZURE_ACCOUNT_KEY>"

where tempostack-dev-azure is the secret name.

Create an instance of TempoStack by referencing the secret name and type as azure:

spec:
  storage:
    secret:
      name: tempostack-dev-azure
      type: azure

Google Cloud Storage

Requirements

Create a project on Google Cloud Platform.
Create a bucket under same project.
Create a service account under same project for GCP authentication.

Installation

Copy the service account credentials received from GCP into a file name key.json.

Create an Object Storage secret with keys bucketname and key.json as follows:

kubectl create secret generic tempostack-dev-gcs \
  --from-literal=bucketname="<BUCKET_NAME>" \
  --from-file=key.json="<PATH/TO/KEY.JSON>"

where tempostack-dev-gcs is the secret name, <BUCKET_NAME> is the name of bucket created in requirements step and <PATH/TO/KEY.JSON> is the file path where the key.json was copied to.

Create an instance of TempoStack by referencing the secret name and type as gcs:

spec:
  storage:
    secret:
      name: tempostack-dev-gcs
      type: gcs

MinIO

Requirements

Deploy MinIO on your cluster, e.g. using the MinIO Operator or another method.
Create a bucket on MinIO using the CLI.

Installation

Create an Object Storage secret with keys as follows:

kubectl create secret generic tempostack-dev-minio \
  --from-literal=bucket="<BUCKET_NAME>" \
  --from-literal=endpoint="<MINIO_BUCKET_ENDPOINT>" \
  --from-literal=access_key_id="<MINIO_ACCESS_KEY_ID>" \
  --from-literal=access_key_secret="<MINIO_ACCESS_KEY_SECRET>"

where tempostack-dev-minio is the secret name.

Create an instance of TempoStack by referencing the secret name and type as s3:

spec:
  storage:
    secret:
      name: tempostack-dev-minio
      type: s3

OpenShift Data Foundation

Requirements

Deploy the OpenShift Data Foundation on your cluster.
Create a bucket via an ObjectBucketClaim.

Installation

Create an Object Storage secret with keys as follows:

kubectl create secret generic tempostack-dev-odf \
  --from-literal=bucket="<BUCKET_NAME>" \
  --from-literal=endpoint="https://s3.openshift-storage.svc" \
  --from-literal=access_key_id="<ACCESS_KEY_ID>" \
  --from-literal=access_key_secret="<ACCESS_KEY_SECRET>"

where tempostack-dev-odf is the secret name. You can copy the values for BUCKET_NAME, ACCESS_KEY_ID and ACCESS_KEY_SECRET from your ObjectBucketClaim’s accompanied secret.

Create an instance of TempoStack by referencing the secret name and type as s3:

spec:
  storage:
    secret:
      name: tempostack-dev-odf
      type: s3

Enable multi-tenancy

Thu, 09 Apr 2026 14:59:14 +0000

Enable multi-tenancy

Tempo is a multi-tenant distributed tracing backend. It supports multi-tenancy through the use of a header: X-Scope-OrgID. Refer to multi-tenancy docs for more details. This document outlines how to deploy and use multi-tenant Tempo with the Operator.

Multi-tenancy without authentication

The following Kubernetes Custom Resource (CR) deploys a multi-tenant Tempo instance.

Note
Jaeger query isn’t tenant-aware and, therefore, isn’t supported in this configuration.

apiVersion: tempo.grafana.com/v1alpha1
kind: TempoStack
metadata:
  name: simplest
spec:
  tenants: {}
  storage:
    secret:
      name: minio-test
      type: s3
  storageSize: 1Gi
  resources:
    total:
      limits:
        memory: 2Gi
        cpu: 2000m

OIDC authentication with static RBAC

On Kubernetes, a multi-tenant Tempo instance uses OIDC authentication and static RBAC authorization defined in the CR. The instance should be accessed through service tempo-simplest-gateway, which handles authentication and authorization. The service exposes Jaeger query API and OpenTelemetry gRPC (OTLP) for trace ingestion. The Jaeger UI can be accessed at http://<exposed gateway service>:8080/api/traces/v1/<tenant-name>/search.

apiVersion: tempo.grafana.com/v1alpha1
kind: TempoStack
metadata:
  name: simplest
spec:
  template:
    queryFrontend:
      jaegerQuery:
        enabled: true
    gateway:
      enabled: true
  storage:
    secret:
      type: s3
      name: minio-test
  storageSize: 200M
  tenants:
    mode: static
    authentication:
      - tenantName: test-oidc
        tenantId: test-oidc
        oidc:
          issuerURL: http://dex.default.svc.cluster.local:30556/dex
          redirectURL: http://tempo-simplest-gateway.default.svc.cluster.local:8080/oidc/test-oidc/callback
          usernameClaim: email
          secret:
            name: oidc-test
    authorization:
      roleBindings:
        - name: 'test'
          roles:
            - read-write
          subjects:
            - kind: user
              name: 'admin@example.com'
      roles:
        - name: read-write
          permissions:
            - read
            - write
          resources:
            - traces
          tenants:
            - test-oidc

The secret oidc-test defines fields clientID, clientSecret and issuerCAPath.
The RBAC gives tenant test-oidc read and write access for traces.

OpenShift

On OpenShift, the authentication and authorization does not require any third-party service dependencies. The authentication uses OpenShift OAuth (the user is redirected to the OpenShift login page) and authorization is handled through SubjectAccessReview (SAR).

The instance should be accessed through service tempo-simplest-gateway, which handles authentication and authorization. The service exposes Jaeger query API and OpenTelemetry gRPC (OTLP) for trace ingestion. The Jaeger UI can be accessed at http://<exposed gateway service>:8080/api/traces/v1/<tenant-name>/search.

apiVersion: tempo.grafana.com/v1alpha1
kind: TempoStack
metadata:
  name: simplest
spec:
  storage:
    secret:
      name: object-storage
      type: s3
  storageSize: 1Gi
  tenants:
    mode: openshift
    authentication:
      - tenantName: dev
        tenantId: '1610b0c3-c509-4592-a256-a1871353dbfa'
      - tenantName: prod
        tenantId: '1610b0c3-c509-4592-a256-a1871353dbfb'
  template:
    gateway:
      enabled: true
    queryFrontend:
      jaegerQuery:
        enabled: true

ClusterRole and ClusterRoleBinding objects have to be created to enable reading and writing the data.

RBAC for reading the data

The following RBAC gives authenticated users access to read trace data for dev and prod tenants.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: tempostack-traces-reader
rules:
  - apiGroups:
      - 'tempo.grafana.com'
    resources:
      - dev
      - prod
    resourceNames:
      - traces
    verbs:
      - 'get'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: tempostack-traces-reader
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: tempostack-traces-reader
subjects:
  - kind: Group
    apiGroup: rbac.authorization.k8s.io
    name: system:authenticated

RBAC for writing data

The following RBAC gives service account otel-collector write access for trace data for dev tenant.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: otel-collector
  namespace: otel
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: tempostack-traces-write
rules:
  - apiGroups:
      - 'tempo.grafana.com'
    resources:
      - dev
    resourceNames:
      - traces
    verbs:
      - 'create'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: tempostack-traces
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: tempostack-traces-write
subjects:
  - kind: ServiceAccount
    name: otel-collector
    namespace: otel

OpenTelemetry collector CR configuration with authentication for dev tenant.

spec:
  serviceAccount: otel-collector
  config: |
    extensions:
      bearertokenauth:
        filename: "/var/run/secrets/kubernetes.io/serviceaccount/token"
    exporters:
      # Export the dev tenant traces to a Tempo instance
      otlp/dev:
        endpoint: tempo-simplest-gateway.tempo.svc.cluster.local:8090
        tls:
          insecure: false
          ca_file: "/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt"
        auth:
          authenticator: bearertokenauth
        headers:
          X-Scope-OrgID: "dev"

Monitor Tempo instances and the operator

Thu, 09 Apr 2026 14:59:14 +0000

Monitor Tempo instances and the operator

You can configure the Tempo Operator to monitor TempoStack instances (including all Tempo components like the distributor). In addition, the operator can expose metrics about the operator itself (for example, the number of successful and failed upgrades, etc.).

Monitor TempoStack instances

The Tempo Operator supports monitoring and alerting of each Tempo component (distributor, ingester, etc.). To enable metrics and alerting, the Prometheus Operator or a comparable solution which discovers ServiceMonitor and PrometheusRule objects must be installed and configured in the cluster.

The configuration for monitoring TempoStack instances is exposed in the CR:

apiVersion: tempo.grafana.com/v1alpha1
kind: TempoStack
spec:
  observability:
    metrics:
      createServiceMonitors: true
      createPrometheusRules: true
    tracing:
      sampling_fraction: 1.0
      jaeger_agent_endpoint: localhost:6831

Configure distributed tracing of operands

All Tempo components as well as the Tempo Gateway support the export of traces in thrift_compact format.

Deploy OpenTelemetry collector sidecar

To deploy the OpenTelemetry collector, follow these steps:

Install OpenTelemetry Operator into the cluster.
Create an OpenTelemetryCollector CR that receives trace data in Jaeger Thrift format and exports data via OTLP to the desired trace backend.
Optional: Deploy tracing backend to store trace data.

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: sidecar-for-tempo
spec:
  mode: sidecar
  config: |
    receivers:
      jaeger:
        protocols:
          thrift_compact:

    exporters:
      otlp:
        endpoint: <otlp-endpoint>:4317
        tls:
          insecure: true

    service:
      pipelines:
        traces:
          receivers: [jaeger]
          exporters: [otlp]

Send trace data to OpenTelemetry sidecar

Finally, create a TempoStack instance that sets jaeger_agent_endpoint to report trace data to the localhost. The Tempo operator sets the OpenTelemetry inject annotation sidecar.opentelemetry.io/inject": "true to all TempoStack pods. The OpenTelemetry Operator will recognize the annotation, and it will inject a sidecar into all TempoStack pods.

apiVersion: tempo.grafana.com/v1alpha1
kind: TempoStack
metadata:
  name: simple-stack
spec:
  template:
    queryFrontend:
      jaegerQuery:
        enabled:
  storage:
    secret:
      type: s3
      name: minio-test
  storageSize: 200M
  observability:
    tracing:
      sampling_fraction: '1.0'
      jaeger_agent_endpoint: localhost:6831

Monitor the operator

The Tempo Operator can expose upgrade and other operational metrics about the operator itself, and can create alerts based on these metrics. For example, the operator handles Tempo upgrades and exposes metrics like “the number of successful Tempo upgrades”, “number of failed Tempo upgrades”, and others. The operator also creates alerts to notify system administrators if any Tempo upgrade fails.

Other metrics are internal to the operator itself, for example, the duration of a reconcile loop iteration. This operator-specific component continuously tries to match the expected state as described in the TempoStack custom resource to the actual cluster state. For example, if an object is deleted in the cluster which is managed by the operator, the operator re-creates this object again, to match the expected state of the cluster.

The operator can be configured using the ConfigMap tempo-operator-manager-config in the same namespace as the operator. The following excerpt shows the configuration options to enable the creation of ServiceMonitor (for scraping metrics) and PrometheusRule (for creating alerts) objects:

apiVersion: v1
kind: ConfigMap
data:
  controller_manager_config.yaml: |
    featureGates:
      observability:
        metrics:
          createServiceMonitors: true
          createPrometheusRules: true

Grafana data source

Thu, 09 Apr 2026 14:59:14 +0000

Grafana data source

You can use Grafana to query and visualize traces of the TempoStack instance by configuring a Tempo data source in Grafana.

Use Grafana Operator

If your Grafana instance is managed by the Grafana Operator, you can instruct the Tempo Operator to create a data source (GrafanaDatasource custom resource):

apiVersion: tempo.grafana.com/v1alpha1
kind: TempoStack
spec:
  observability:
    grafana:
      createDatasource: true

Note
The feature gate featureGates.grafanaOperator must be enabled in the Tempo Operator configuration.

Manual data source configuration

You can choose to either use Tempo Operator’s gateway or not:

If the TempoStack is deployed using the gateway, you’ll need to provide authentication information to Grafana, along with the URL of the tenant from which you expect to see the traces.
If the gateway is not used, then you need to make sure Grafana can access the query-frontend endpoints.

For more information, refer to the Tempo data source for Grafana.

Use with gateway

The gateway, an optional component deployed as part of Tempo Operator, provides secure access to Tempo’s distributor (for example, for pushing spans) and query-frontend (for example, for querying traces) via consulting an OAuth/OIDC endpoint for the request subject.

The OIDC configuration expects clientID and clientSecret. They should be provided via a Kubernetes secret that the TempoStack admin provides upfront.

The gateway exposes all Tempo query endpoints, so you can use the endpoint as a Tempo data source for Grafana.

If Grafana is configured with some OAuth provider, such as generic OAuth, the TempoStack with the gateway should be deployed using the same clientID and clientSecret:

apiVersion: v1
kind: Secret
metadata:
  name: oidc-test
stringData:
  clientID: <clientID used for grafana authentication>
  clientSecret: <clientSecret used for grafana authentication>
type: Opaque

Then deploy TempoStack with gateway enabled:

spec:
  template:
    gateway:
      enabled: true
  tenants:
    mode: static
    authentication:
      - tenantName: test-oidc
        tenantId: test-oidc
        oidc:
        issuerURL: http://dex:30556/dex
        redirectURL: http://tempo-foo-gateway:8080/oidc/test-oidc/callback
        usernameClaim: email
        secret:
          name: oidc-test

Set the data source URL parameter to http://<HOST>:<PORT>/api/traces/v1/{tenant}/tempo/, where {tenant} is the name of the tenant.

To use it as a data source, set the Authentication Method to Forward Oauth Identify using the same clientID and clientSecret for gateway and for the OAuth configuration. This will forward the access_token to the gateway so it can authenticate the client.

If you prefer to set the Bearer token directly and not use the Forward Oauth Identify, you can add it to the “Authorization” Header.

Without the gateway

If you are not using the gateway, make sure your Grafana can access to the query-frontend endpoints, you can do this by creating an ingress or a route in OpenShift.

Once you have the endpoint, you can set it as URL when you create the Tempo data source.

Monolithic deployment

Thu, 09 Apr 2026 14:59:14 +0000

Monolithic deployment

The TempoMonolithic Custom Resource (CR) creates a Tempo deployment in Monolithic mode. In this mode, a single container has all components of the Tempo deployment, including the compactor, distributor, ingester, querier, and query-frontend.

This type of deployment is ideal for small deployments, demo, and test setups, and supports storing traces in memory, in a Persistent Volume and in object storage.

Note
The monolithic deployment of Tempo doesn’t scale horizontally. If you require horizontal scaling, use the TempoStack CR for a Tempo deployment in Microservices mode.

Quickstart

The following manifest creates a Tempo monolithic deployment with trace ingestion over OTLP/gRPC and OTLP/HTTP, storing traces in a 2 GiB tmpfs volume (in-memory storage).

apiVersion: tempo.grafana.com/v1alpha1
kind: TempoMonolithic
metadata:
  name: sample
spec:
  storage:
    traces:
      backend: memory
      size: 2Gi

After the Pod is ready, you can send traces to tempo-sample:4317 (OTLP/gRPC) and tempo-sample:4318 (OTLP/HTTP) inside the cluster.

To configure a Grafana data source, use the URL http://tempo-sample:3200 (available inside the cluster).

CRD specification

A manifest with all available configuration options is available here: tempo.grafana.com_tempomonolithics.yaml.

Note
This file is auto-generated and does not constitute a valid CR.

It provides an overview of the structure, the available configuration options and help texts.

Operator configuration and CRD specifications

Thu, 09 Apr 2026 14:59:14 +0000

Operator configuration and CRD specifications

The Operator configuration can be found at tempo-operator/docs/operator/config.yaml
The TempoStack CRD documentation can be found at tempo-operator/docs/spec/tempo.grafana.com_tempostacks.yaml
The TempoMonolithic CRD documentation can be found at tempo-operator/docs/spec/tempo.grafana.com_tempomonolithics.yaml