Hosted storage on Grafana Labs

Amazon S3 and S3-compatible storage

Thu, 28 May 2026 17:50:33 +0100

Amazon S3 and S3-compatible storage

Tempo supports Amazon S3 and S3-compatible object stores as backends for trace storage. For general storage configuration options, refer to the storage section on the configuration page.

Authentication

The following authentication methods are supported:

AWS environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
Static access key and secret credentials specified in access_key and secret_key
MinIO environment variables MINIO_ACCESS_KEY and MINIO_SECRET_KEY
AWS shared credentials configuration file
MinIO client credentials configuration file
AWS IAM (IRSA via WebIdentity,
AWS EC2 instance role)
AWS EKS Pod Identity

IAM policy

The following IAM policy shows minimal permissions required by Tempo, where the bucket has already been created.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "TempoPermissions",
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:ListBucket",
        "s3:DeleteObject",
        "s3:GetObjectTagging",
        "s3:PutObjectTagging"
      ],
      "Resource": ["arn:aws:s3:::<bucketname>/*", "arn:aws:s3:::<bucketname>"]
    }
  ]
}

Lifecycle policy

A lifecycle policy is recommended that deletes incomplete multipart uploads after one day.

S3-compatible local stores for testing

Note
The tools in this section are provided for local testing and evaluation only. They have not been fully tested with Tempo and are not recommended for production use.

You can run an S3-compatible object store locally to test Tempo with the s3 storage backend.

SeaweedFS is the recommended option for local testing, with a single-command startup and a built-in web UI.

rclone serve s3 is an alternative that serves any local directory as an S3-compatible endpoint. It is classified as experimental by the rclone project and has known limitations.

MinIO is also supported but its open source repository has been archived and the community edition is now distributed as source code only. Pre-compiled binaries are no longer published.

Set up a local S3-compatible object store

Choose a tab below to set up your preferred object store:

SeaweedFS

rclone serve s3 (experimental)

MinIO

Note
SeaweedFS has not been fully tested with Tempo and is provided here as an alternative for local evaluation only. It isn’t recommended for production use with Tempo.

SeaweedFS is an Apache 2.0-licensed distributed storage system with a built-in S3 gateway.

Download and install SeaweedFS from the releases page.
Create a data directory and start SeaweedFS in mini mode:
Bash
```
sudo mkdir -p /data/seaweedfs
sudo chown -R $USER:$USER /data/seaweedfs
weed mini -dir=/data/seaweedfs
```
The weed mini command starts a complete single-node setup including the S3 gateway on port 8333. SeaweedFS runs in the foreground, so open a new terminal for the remaining steps.
Create a bucket called tempo using the AWS CLI:
Bash
```
aws --endpoint-url http://localhost:8333 s3 mb s3://tempo --no-sign-request
```
You need the AWS CLI installed. SeaweedFS mini mode allows anonymous access, so the --no-sign-request flag skips credential checks.

Note
rclone serve s3 is classified as experimental by the rclone project and hasn’t been fully tested with Tempo. It’s provided as an alternative for local evaluation only and isn’t recommended for production use. Refer to the rclone documentation for current limitations.

rclone can serve any local directory as an S3-compatible endpoint.

Install rclone by following the rclone install guide.

Create a data directory and start the S3 server:

sudo mkdir -p /data/rclone-s3
sudo chown -R $USER:$USER /data/rclone-s3
rclone serve s3 /data/rclone-s3 --auth-key tempokey,temposecret --addr :8080

The server runs in the foreground on port 8080. Open a new terminal for the remaining steps.

Create a bucket called tempo using the AWS CLI:
Bash
```
AWS_ACCESS_KEY_ID=tempokey AWS_SECRET_ACCESS_KEY=temposecret \
  aws --endpoint-url http://localhost:8080 s3 mb s3://tempo
```
You need the AWS CLI installed. Use the credentials you set with the --auth-key flag.

Note
The MinIO open source repository has been archived and the community edition is now source-only. Pre-compiled binaries are no longer published. You must build MinIO from source using Go 1.24 or later.

Install MinIO from source:
Bash
```
go install github.com/minio/minio@latest
```
Refer to the MinIO repository for alternative installation methods including building a Docker image.
Create a data directory and start MinIO:
Bash
```
sudo mkdir -p /data/minio
sudo chown -R $USER:$USER /data/minio
minio server /data/minio --console-address ':9001'
```
By default, MinIO uses minioadmin for both the access key and secret key. MinIO runs in the foreground, so open a new terminal for the remaining steps.

Create a bucket called tempo using the MinIO Client (mc):

mc alias set local http://localhost:9000 minioadmin minioadmin
mc mb local/tempo

Tempo configuration for S3-compatible stores

The following example configuration uses the S3 backend. Replace the <S3_ENDPOINT>, <S3_ACCESS_KEY>, and <S3_SECRET_KEY> placeholders with the values for your object store.

This example configuration includes the metrics-generator. To disable it, remove the metrics_generator block and the processors list from the overrides.

stream_over_http_enabled: true

server:
  http_listen_port: 3200

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: "0.0.0.0:4317"
        http:
          endpoint: "0.0.0.0:4318"

backend_scheduler:
  provider:
    compaction:
      compaction:
        block_retention: 1h

backend_worker:
  compaction:
    block_retention: 1h

metrics_generator:
  registry:
    external_labels:
      source: tempo
      cluster: linux-monolithic
  storage:
    path: /tmp/tempo/generator/wal
    remote_write:
      - url: http://<PROMETHEUS_URL>/api/v1/write
        send_exemplars: true

storage:
  trace:
    backend: s3
    s3:
      endpoint: <S3_ENDPOINT>
      bucket: tempo
      access_key: <S3_ACCESS_KEY>
      secret_key: <S3_SECRET_KEY>
      insecure: true
    wal:
      path: /var/tempo/wal

overrides:
  defaults:
    metrics_generator:
      processors: [service-graphs, span-metrics]

usage_report:
  reporting_enabled: false

Replace the <PROMETHEUS_URL> placeholder with the address of your Prometheus-compatible storage instance (for example, localhost:9090). To disable the metrics-generator, remove the processors list from the overrides and the metrics_generator block.

Use the following endpoint and credential values for each object store:

SeaweedFS

rclone serve s3 (experimental)

MinIO

SeaweedFS mini mode allows anonymous access, so the access_key and secret_key fields can be omitted or set to any value:

storage:
  trace:
    backend: s3
    s3:
      endpoint: localhost:8333
      bucket: tempo
      insecure: true

Use the credentials you set with the --auth-key flag when starting rclone:

storage:
  trace:
    backend: s3
    s3:
      endpoint: localhost:8080
      bucket: tempo
      access_key: tempokey
      secret_key: temposecret
      insecure: true

storage:
  trace:
    backend: s3
    s3:
      endpoint: localhost:9000
      bucket: tempo
      access_key: minioadmin
      secret_key: minioadmin
      insecure: true

Verify data in your S3-compatible store

After traces start flowing, verify that your storage bucket has received data:

SeaweedFS

rclone serve s3 (experimental)

MinIO

Open the SeaweedFS admin UI at http://localhost:23646, or list the bucket contents using the AWS CLI:

aws --endpoint-url http://localhost:8333 s3 ls s3://tempo/ --recursive --no-sign-request

You should see files such as single-tenant/<block-id>/data.parquet and single-tenant/<block-id>/meta.json.

List the bucket contents using the AWS CLI:

AWS_ACCESS_KEY_ID=tempokey AWS_SECRET_ACCESS_KEY=temposecret \
  aws --endpoint-url http://localhost:8080 s3 ls s3://tempo/ --recursive

You should see files such as single-tenant/<block-id>/data.parquet and single-tenant/<block-id>/meta.json. rclone serve s3 does not provide a web UI.

Open the MinIO Console at http://localhost:9001 and check the tempo bucket for files such as work.json and a tenant data directory.

Azure blob storage permissions and management

Thu, 28 May 2026 17:50:33 +0100

Azure blob storage permissions and management

Tempo supports Azure blob storage for both monolithic and distributed modes. Some of the supported features include:

Object layout: custom container_name and optional prefix to nest objects in a shared container.
Performance: hedged requests (hedge_requests_at, hedge_requests_up_to) to reduce long-tail latency.
Regional or sovereign clouds: configurable endpoint suffix (for example, US Gov, Germany) using endpoint_suffix.
Local development: Azurite emulator support (non-blob.\* endpoint style is auto-detected).
Ops guidance: compatible with Azure Storage lifecycle policies for cleanup (example provided in the doc).

Tempo supports the following authentication methods:

Shared key
Managed Identity (system/user-assigned): use use_managed_identity, user_assigned_id
Azure Workload Identity (federated token): use use_federated_token

Before you begin

Tempo requires the following configuration to authenticate to and access Azure blob storage:

Storage Account name specified in the configuration file as storage_account_name or in the environment variable AZURE_STORAGE_ACCOUNT.
Credentials for accessing the Storage Account that are one of the following:
- Storage Account access key specified in the configuration file as storage_account_key or in the environment variable AZURE_STORAGE_KEY.
- An Azure Managed Identity that’s either system or user assigned. To use Azure Managed Identities, you need to set use_managed_identity to true in the configuration file or set user_assigned_id to the client ID for the managed identity you’d like to use.
  - System-assigned managed identity don’t require additional configuration.
  - User-assigned managed identity require you to set user_assigned_id to the client ID for the managed identity in the configuration file.
- Via Azure Workload Identity. To use Azure Workload Identity, you need to enable Azure Workload Identity on your cluster, add the required label and annotation to the service account and the required Pod label. Additionally, you need to set use_federated_token to true to utilize Azure Workload Identity.

Sample configuration for Tempo monolithic mode

This sample configuration shows how to set up Azure blob storage using Helm charts and an access key from Kubernetes secrets.

tempo:
  storage:
    trace:
      backend: azure
      azure:
        container_name: container-name
        storage_account_name: storage-account-name
        storage_account_key: ${STORAGE_ACCOUNT_ACCESS_KEY}

  extraArgs:
    config.expand-env: true
  extraEnv:
    - name: STORAGE_ACCOUNT_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          name: secret-name
          key: STORAGE_ACCOUNT_ACCESS_KEY

Azure Workload Identity

Here is an example configuration using Azure Workload Identity.

tempo:
  storage:
    trace:
      backend: azure
      azure:
        container_name: container-name
        storage_account_name: storage-account-name
        use_federated_token: true

Sample configuration for Tempo distributed mode

In distributed mode, the trace configuration needs to be applied against the storage object, which resides at the root of the Values object. Additionally, the extraArgs and extraEnv configuration need to be applied to each of the following services:

distributor
blockBuilder
liveStore
querier
queryFrontend
backendScheduler
backendWorker

Distributed mode is usually installed using a Helm chart, like tempo-distributed. To use this example, add it to your custom.yaml or values.yaml file.

storage:
  trace:
    backend: azure
    azure:
      container_name: tempo-traces
      storage_account_name: stgappgeneraluks
      storage_account_key: ${STORAGE_ACCOUNT_ACCESS_KEY}

distributor:
  extraArgs:
    - "-config.expand-env=true"
  extraEnv:
    - name: STORAGE_ACCOUNT_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          name: tempo-traces-stg-key
          key: tempo-traces-key

blockBuilder:
  extraArgs:
    - "-config.expand-env=true"
  extraEnv:
    - name: STORAGE_ACCOUNT_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          name: tempo-traces-stg-key
          key: tempo-traces-key

liveStore:
  extraArgs:
    - "-config.expand-env=true"
  extraEnv:
    - name: STORAGE_ACCOUNT_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          name: tempo-traces-stg-key
          key: tempo-traces-key

querier:
  extraArgs:
    - "-config.expand-env=true"
  extraEnv:
    - name: STORAGE_ACCOUNT_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          name: tempo-traces-stg-key
          key: tempo-traces-key

queryFrontend:
  extraArgs:
    - "-config.expand-env=true"
  extraEnv:
    - name: STORAGE_ACCOUNT_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          name: tempo-traces-stg-key
          key: tempo-traces-key

backendScheduler:
  extraArgs:
    - "-config.expand-env=true"
  extraEnv:
    - name: STORAGE_ACCOUNT_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          name: tempo-traces-stg-key
          key: tempo-traces-key

backendWorker:
  extraArgs:
    - "-config.expand-env=true"
  extraEnv:
    - name: STORAGE_ACCOUNT_ACCESS_KEY
      valueFrom:
        secretKeyRef:
          name: tempo-traces-stg-key
          key: tempo-traces-key

Use `metrics-generator` with Azure storage

The metrics-generator consumes trace data from Kafka and writes derived metrics using the Prometheus remote write protocol. If the metrics-generator needs access to Azure blob storage, list it in the env var expansion configuration so the STORAGE_ACCOUNT_ACCESS_KEY has the secret value.

You can use this configuration example with Helm charts, like tempo-distributed. Replace any values in all caps with the values for your Helm deployment.

generator:
  extraArgs:
    - "-config.expand-env=true"
  extraEnv:
    - name: <STORAGE_ACCOUNT_ACCESS_KEY>
      valueFrom:
        secretKeyRef:
          name: <TEMPO-TRACES-STG-KEY>
          key: <TEMPO-TRACES-KEY>

For more information, refer to Configure TraceQL metrics.

Additional configuration options

The following sections provide additional configuration options for Azure blob storage.

Use Azurite for local development

You can use the Azurite emulator to test your Tempo configuration locally. Refer to the Azurite emulator documentation for more details.

Tempo treats any Azure endpoint_suffix that doesn’t start with blob. as Azurite and automatically switches to the emulator URL style. For more information about the Azurite URL style, refer to the Azure Storage documentation.

Set backend to azure, supply your Azurite account and key, and point endpoint_suffix to the emulator host:port. Tempo handles the Azurite URL format automatically.

If you encounter any issues, try using the fully qualified domain name (FQDN) for the Azurite emulator. For example, azurite-host.azure.local:10000.

In this example, replace the example values with your Azure configuration values and then update your Helm deployment.

storage:
  trace:
    blocklist_poll: 1s
    backend: azure
    azure:
      container_name: container-name # how to store data in azure
      endpoint_suffix: azurite-host.svc.cluster.local:10000 # Azurite emulator host:port
      storage_account_name: "<STORAGE-ACCOUNT-NAME>"
      storage_account_key: "<STORAGE_ACCOUNT_ACCESS_KEY>"

Azure blocklist polling

If you are hosting Tempo on Azure, you may need to update two values to ensure consistent successful blocklist polling. If you experience this issue, try setting blocklist_poll_tenant_index_builders to 1.

Additionally, if you are seeing DNS failures like the ones below, try increasing blocklist_poll_jitter_ms. Refer to the discussion in GitHub issue 1462.

For example:

reading storage container: Head "https://tempoe**************.blob.core.windows.net/tempo/single-tenant/d8aafc48-5796-4221-ac0b-58e001d18515/meta.compacted.json?timeout=61": dial tcp: lookup tempoe**************.blob.core.windows.net on 10.0.0.10:53: dial udp 10.0.0.10:53: operation was canceled

Your final configuration may look something like:

storage:
  trace:
    blocklist_poll_tenant_index_builders: 1
    blocklist_poll_jitter_ms: 500

(Optional) Storage Account management policy for cleaning up the storage container

The following Storage Account management policy shows an example of cleaning up files from the container after they have been deleted for a period of time.

{
  "id": "/subscriptions/00000000-0000-0000000000000000000000/resourceGroups/resourceGroupName/providers/Microsoft.Storage/storageAccounts/accountName/managementPolicies/default",
  "lastModifiedTime": "2021-11-30T19:19:54.855455+00:00",
  "name": "DefaultManagementPolicy",
  "policy": {
    "rules": [
      {
        "definition": {
          "actions": {
            "baseBlob": {
              "delete": {
                "daysAfterLastAccessTimeGreaterThan": null,
                "daysAfterModificationGreaterThan": 60.0
              },
              "enableAutoTierToHotFromCool": null,
              "tierToArchive": null,
              "tierToCool": null
            },
            "snapshot": null,
            "version": null
          },
          "filters": {
            "blobIndexMatch": null,
            "blobTypes": ["blockBlob"],
            "prefixMatch": ["tempo-data"]
          }
        },
        "enabled": true,
        "name": "TempoBlobRetention",
        "type": "Lifecycle"
      },
      {
        "definition": {
          "actions": {
            "baseBlob": null,
            "snapshot": null,
            "version": {
              "delete": {
                "daysAfterCreationGreaterThan": 7.0
              },
              "tierToArchive": null,
              "tierToCool": null
            }
          },
          "filters": {
            "blobIndexMatch": null,
            "blobTypes": ["blockBlob"],
            "prefixMatch": []
          }
        },
        "enabled": true,
        "name": "VersionRetention",
        "type": "Lifecycle"
      }
    ]
  },
  "resourceGroup": "resource-group-name",
  "type": "Microsoft.Storage/storageAccounts/managementPolicies"
}

Google Cloud Storage

Thu, 28 May 2026 17:50:33 +0100

Google Cloud Storage

For configuration options, check the storage section on the configuration page.

Permissions

The following authentication methods are supported:

Google Cloud Platform environment variable GOOGLE_APPLICATION_CREDENTIALS
Google Cloud Platform Workload Identity

The (service-)account that will communicate towards GCS should be assigned to the bucket which will receive the traces and should have the following IAM policies within the bucket:

storage.objects.create
storage.objects.delete
storage.objects.get
storage.buckets.get
storage.objects.list

Hosted storage on Grafana Labs

Amazon S3 and S3-compatible storage

Amazon S3 and S3-compatible storage

Authentication

IAM policy

Lifecycle policy

S3-compatible local stores for testing

Set up a local S3-compatible object store

Tempo configuration for S3-compatible stores

Verify data in your S3-compatible store

Azure blob storage permissions and management

Azure blob storage permissions and management

Before you begin

Sample configuration for Tempo monolithic mode

Azure Workload Identity

Sample configuration for Tempo distributed mode

Use metrics-generator with Azure storage

Additional configuration options

Use Azurite for local development

Azure blocklist polling

(Optional) Storage Account management policy for cleaning up the storage container

Google Cloud Storage

Google Cloud Storage

Permissions

Use `metrics-generator` with Azure storage