--- title: "Deploy on Kubernetes with Tanka | Grafana Tempo documentation" description: "Learn how to deploy Tempo using Kubernetes and Tanka" --- > For a curated documentation index, see [llms.txt](/llms.txt). For the complete documentation index, see [llms-full.txt](/llms-full.txt). # Deploy on Kubernetes with Tanka Using this deployment guide, you can deploy Tempo to Kubernetes using a Jsonnet library and [Grafana Tanka](https://tanka.dev) to create a development cluster or sand-boxed environment. This procedure uses MinIO to provide object storage regardless of the cloud platform or on-premise storage you use. In a production environment, you can use your cloud provider’s object storage service to avoid the operational overhead of running object storage in production. To set up Tempo using Kubernetes with Tanka, you need to: 1. Configure Kubernetes and install Tanka 2. Set up the Tanka environment 3. Install libraries 4. Deploy MinIO object storage 5. Optional: Enable metrics-generator 6. Deploy Tempo with the Tanka command > Note > > This configuration isn’t suitable for a production environment but can provide a useful way to learn about Tempo. ## Before you begin To deploy Tempo to Kubernetes with Tanka, you need: - A Kubernetes cluster with at least 40 CPUs and 46GB of memory for the default configuration. Small ingest or query volumes could use a far smaller configuration. - `kubectl` (version depends upon the API version of Kubernetes in your cluster) ## Configure Kubernetes and install Tanka Follow these steps to configure Kubernetes and install Tanka. 1. Create a new directory for the installation, and make it your current working directory: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash mkdir tempo cd tempo ``` 2. Create a Kubernetes namespace. You can replace the namespace`tempo` in this example with a name of your choice. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash kubectl create namespace tempo ``` 3. Install Grafana Tanka; refer to [Installing Tanka](https://tanka.dev/install). 4. Install `jsonnet-bundler`; refer to the [`jsonnet-bundler` README](https://github.com/jsonnet-bundler/jsonnet-bundler/#install). ## Set up the Tanka environment Tanka requires the current context for your Kubernetes environment. 1. Check the current context for your Kubernetes cluster and ensure it’s correct: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash kubectl config current-context ``` 2. Initialize Tanka. This uses the current Kubernetes context: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tk init --k8s=false tk env add environments/tempo tk env set environments/tempo \ --namespace=tempo \ --server-from-context=$(kubectl config current-context) ``` ## Install libraries Install the `k.libsonnet`, Jsonnet, and Memcached libraries. 1. Install `k.libsonnet` for your version of Kubernetes. Set `K8S_VERSION` to match your cluster’s minor version (for example, `1.32`): Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash mkdir -p lib export K8S_VERSION=1.32 jb install github.com/jsonnet-libs/k8s-libsonnet/${K8S_VERSION}@main cat < lib/k.libsonnet import 'github.com/jsonnet-libs/k8s-libsonnet/${K8S_VERSION}/main.libsonnet' EOF ``` 2. Install the Tempo Jsonnet library and its dependencies. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash jb install github.com/grafana/tempo/operations/jsonnet/microservices@main ``` 3. Install the Memcached library and its dependencies. Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash jb install github.com/grafana/jsonnet-libs/memcached@master ``` ## Deploy MinIO object storage [MinIO](https://min.io) is an open source Amazon S3-compatible object storage service that is freely available and runs on Kubernetes. 1. Create a file named `minio.yaml` and copy the following YAML configuration into it. You may need to remove/modify the `storageClassName` depending on your Kubernetes platform. GKE, for example, may not support `local-path` name but may support another option such as `standard`. YAML ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: # This name uniquely identifies the PVC. Will be used in deployment below. name: minio-pv-claim labels: app: minio-storage-claim spec: # Read more about access modes here: http://kubernetes.io/docs/user-guide/persistent-volumes/#access-modes accessModes: - ReadWriteOnce storageClassName: local-path resources: # This is the request for storage. Should be available in the cluster. requests: storage: 50Gi --- apiVersion: apps/v1 kind: Deployment metadata: name: minio spec: selector: matchLabels: app: minio strategy: type: Recreate template: metadata: labels: # Label is used as selector in the service. app: minio spec: # Refer to the PVC created earlier volumes: - name: storage persistentVolumeClaim: # Name of the PVC created earlier claimName: minio-pv-claim initContainers: - name: create-buckets image: busybox:1.28 command: - 'sh' - '-c' - 'mkdir -p /storage/tempo-data' volumeMounts: - name: storage # must match the volume name, above mountPath: '/storage' containers: - name: minio # Pulls the default Minio image from Docker Hub image: minio/minio:latest args: - server - /storage - --console-address - ':9001' env: # MinIO root credentials - name: MINIO_ROOT_USER value: 'minio' - name: MINIO_ROOT_PASSWORD value: 'minio123' ports: - containerPort: 9000 - containerPort: 9001 volumeMounts: - name: storage # must match the volume name, above mountPath: '/storage' --- apiVersion: v1 kind: Service metadata: name: minio spec: type: ClusterIP ports: - port: 9000 targetPort: 9000 protocol: TCP name: api - port: 9001 targetPort: 9001 protocol: TCP name: console selector: app: minio ``` 2. Run the following command to apply the minio.yaml file: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash kubectl apply --namespace tempo -f minio.yaml ``` 3. To check that MinIO is correctly configured, sign in to MinIO and verify that a bucket has been created. Without these buckets, no data will be stored. 1. Port-forward MinIO to port 9001: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash kubectl port-forward --namespace tempo service/minio 9001:9001 ``` 2. Navigate to the MinIO admin bash using your browser: `http://localhost:9001`. The sign-in credentials are username `minio` and password `minio123`. 3. Verify that the Buckets page lists `tempo-data`. 4. Configure the Tempo cluster using the MinIO object storage by updating the contents of the `environments/tempo/main.jsonnet` file by running the following command: jsonnet ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```jsonnet cat < environments/tempo/main.jsonnet // The jsonnet file used to generate the Kubernetes manifests. local tempo = import 'microservices/tempo.libsonnet'; local k = import 'ksonnet-util/kausal.libsonnet'; local container = k.core.v1.container; local containerPort = k.core.v1.containerPort; tempo { _images+:: { tempo: 'grafana/tempo:latest', }, tempo_distributor_container+:: container.withPorts([ containerPort.new('jaeger-grpc', 14250), containerPort.new('otlp-grpc', 4317), ]), _config+:: { namespace: 'tempo', query_frontend+: { replicas: 2, }, querier+: { replicas: 3, }, block_builder+: { replicas: 2, }, live_store+: { replicas: 2, pvc_size: '10Gi', pvc_storage_class: 'standard', }, backend_scheduler+: { pvc_size: '1Gi', pvc_storage_class: 'standard', }, backend_worker+: { replicas: 1, }, distributor+: { replicas: 3, receivers: { jaeger: { protocols: { grpc: { endpoint: '0.0.0.0:14250', }, }, }, otlp: { protocols: { grpc: { endpoint: '0.0.0.0:4317', }, }, }, }, }, metrics_generator+: { replicas: 1, ephemeral_storage_request_size: '10Gi', ephemeral_storage_limit_size: '11Gi', pvc_size: '10Gi', pvc_storage_class: 'standard', }, memcached+: { replicas: 3, }, bucket: 'tempo-data', backend: 's3', }, tempo_config+:: { storage+: { trace+: { s3: { bucket: $._config.bucket, access_key: 'minio', secret_key: 'minio123', endpoint: 'minio:9000', insecure: true, }, }, }, ingest+: { kafka+: { address: 'kafka:9092', topic: 'tempo-ingest', }, }, block_builder+: { consume_cycle_duration: '30s', }, metrics_generator+: { processor: { span_metrics: {}, service_graphs: {}, }, registry+: { external_labels: { source: 'tempo', }, }, }, overrides+: { defaults+: { metrics_generator+: { processors: ['service-graphs', 'span-metrics'], }, }, }, }, } EOF ``` > Note > > This configuration requires a Kafka-compatible system. You’ll need to deploy Kafka (or a compatible system like Redpanda) before deploying Tempo. Update `ingest.kafka.address` in the configuration to point to your Kafka instance. ### Optional: Enable metrics-generator In the preceding configuration, [metrics generation](/docs/tempo/next/configuration/#metrics-generator) is enabled. However, you still need to specify where to send the generated metrics data. If you’d like to remote write these metrics onto a Prometheus compatible instance (such as Grafana Cloud metrics or a Mimir instance), you’ll need to include the configuration block below in the `metrics_generator` section of the `tempo_config` block above (this assumes basic auth is required, if not then remove the `basic_auth` section). You can find the details for your Grafana Cloud metrics instance for your Grafana Cloud account by using the [Cloud Portal](/docs/grafana-cloud/account-management/cloud-portal/). jsonnet ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```jsonnet storage+: { remote_write: [ { url: 'https:///api/v1/write', send_exemplars: true, basic_auth: { username: '', password: '', }, } ], }, ``` > Note > > Enabling metrics generation and remote writing them to Grafana Cloud Metrics produces extra active series that could potentially impact your billing. For more information on billing, refer to [Billing and usage](/docs/grafana-cloud/billing-and-usage/). For more information on metrics generation, refer the [Metrics-generator documentation](/docs/tempo/next/metrics-from-traces/metrics-generator/). ### Optional: Enable KEDA autoscaling The microservices Jsonnet library includes optional KEDA-based horizontal autoscaling for distributor, metrics-generator, backend-worker, live-store, and block-builder components. All KEDA scalers are disabled by default (`enabled: false`), and you enable each component independently under `_config..keda`. Before you enable this option, make sure your cluster has the KEDA operator and CRDs installed. The following example enables all supported KEDA scalers. Set `autoscaling_prometheus_url` to the address of your Prometheus-compatible backend. If your backend is a multi-tenant system such as Grafana Mimir, also set `autoscaling_prometheus_tenant` to your tenant ID so that KEDA sends the `X-Scope-OrgID` header on every scrape request: jsonnet ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```jsonnet _config+:: { autoscaling_prometheus_url: 'http://prometheus-operated.monitoring.svc.cluster.local:9090', // autoscaling_prometheus_tenant: 'my-tenant', // Required for multi-tenant backends (e.g. Grafana Mimir) distributor+: { keda+: { enabled: true, min_replicas: 2, max_replicas: 200, target_cpu: '330m', }, }, metrics_generator+: { keda+: { enabled: true, min_replicas: 1, max_replicas: 200, target_cpu: '500m', }, }, backend_worker+: { keda+: { enabled: true, min_replicas: 3, max_replicas: 200, threshold: 200, }, }, live_store+: { keda+: { enabled: true, min_replicas: 1, max_replicas: 200, // window_seconds: 1800, // retention window; >= complete_block_timeout + query_backend_after // bytes_per_replica: 16800000000, // ~16 GiB at 10 MB/s per pod over 30m }, }, block_builder+: { keda+: { enabled: true, // scaling controls how block-builder replicas track live-store: // 'rollout-operator' (default): rollout-operator mirrors live-store zone-a replicas // directly to block-builder. Faster on both scale-up and scale-down. // Requires live_store.keda.enabled=true. rollout_operator_replica_template_access_enabled // is set automatically. // 'keda': a dedicated KEDA ScaledObject uses a kubernetes-workload trigger counting // live-store zone-a pods. Works with or without live-store KEDA. Reaction time is // bounded by KEDA's polling cycle and stabilization windows. // scaling: 'rollout-operator', }, }, }, ``` Tempo uses these trigger types when KEDA is enabled: CPU for distributor and metrics-generator, Prometheus for backend-worker and live-store. Block-builder autoscaling is configured independently under `_config.block_builder.keda`. The default approach (`scaling: 'rollout-operator'`) uses the rollout-operator to mirror the live-store zone-a replica count directly to block-builder. This is the faster approach on both scale-up and scale-down: block-builder reacts as soon as the ReplicaTemplate changes, which is the same signal zone-a responds to. Block-builder intentionally stays alive through the live-store drain window so that in-flight partition data is not lost before it is written to the backend. This approach requires `live_store.keda.enabled=true`; `rollout_operator_replica_template_access_enabled` is set automatically. Alternatively, set `scaling: 'keda'` to use a dedicated KEDA ScaledObject (kubernetes-workload trigger) that counts live-store zone-a pods. This approach works with or without live-store KEDA enabled. Because KEDA must observe pod counts through its polling cycle and apply stabilization windows before acting, reaction time is longer than the rollout-operator approach — on scale-up, block-builder only reacts after new zone-a pods are running and counted; on scale-down, block-builder only reacts after zone-a pods have fully terminated. The block-builder KEDA defaults use aggressive scaling windows to minimize this delay. The value of `_config.block_builder.partitions_per_instance` (default: `1`) is injected into the block-builder config when block-builder KEDA is active under either approach. ### Optional: Reduce component system requirements Smaller ingestion and query volumes could allow the use of smaller resources. If you wish to lower the resources allocated to components, then you can do this via a container configuration. For example, to change the CPU and memory resource allocation for the block-builders or live-stores. To change the resources requirements, follow these steps: 1. Open the `environments/tempo/main.jsonnet` file. 2. Add a new configuration block for the appropriate component (in this case, the block-builder): jsonnet ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```jsonnet tempo_block_builder_container+:: { resources+: { limits+: { cpu: '3', memory: '5Gi', }, requests+: { cpu: '200m', memory: '2Gi', }, }, }, ``` 3. Save the changes to the file. > Note > > Lowering these requirements can impact overall performance. ## Deploy Tempo using Tanka 1. Deploy Tempo using the Tanka command: Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy ```bash tk apply environments/tempo/main.jsonnet ``` > Note > > If the live-stores don’t start after deploying Tempo with the Tanka command, this may be related to the storage class selected for persistent storage. If this is the case, add an appropriate storage class to the live-store configuration. For example, to add a standard instead of fast storage class, add the following to the `_config` (not `tempo_config`) section in the previous step: > > Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy > > ```bash > live_store+: { pvc_storage_class: 'standard', }, > ``` ## Next steps The Tempo instance will now accept the two configured trace protocols (OTLP gRPC and Jaeger gRPC) via the distributor service at `distributor.tempo.svc.cluster.local` on the relevant ports: - OTLP gRPC: `4317` - Jaeger gRPC: `14250` You can query Tempo using the `query-frontend.tempo.svc.cluster.local` service on port `3200`. Now that you’ve configured a Tempo cluster, you’ll need to validate your deployment. Refer to the [Validate Kubernetes deployment using a test application](/docs/tempo/next/set-up-for-tracing/setup-tempo/test/set-up-test-app/) for instructions.