This is documentation for the next version of Grafana Tempo documentation. For the latest stable release, go to the latest version.

Configure

Hosted storage

S3 storage

Open source

Amazon S3 and S3-compatible storage

Tempo supports Amazon S3 and S3-compatible object stores as backends for trace storage. For general storage configuration options, refer to the storage section on the configuration page.

Authentication

The following authentication methods are supported:

AWS environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
Static access key and secret credentials specified in access_key and secret_key
MinIO environment variables MINIO_ACCESS_KEY and MINIO_SECRET_KEY
AWS shared credentials configuration file
MinIO client credentials configuration file
AWS IAM (IRSA via WebIdentity,
AWS EC2 instance role)
AWS EKS Pod Identity

IAM policy

The following IAM policy shows minimal permissions required by Tempo, where the bucket has already been created.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "TempoPermissions",
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "s3:ListBucket",
        "s3:DeleteObject",
        "s3:GetObjectTagging",
        "s3:PutObjectTagging"
      ],
      "Resource": ["arn:aws:s3:::<bucketname>/*", "arn:aws:s3:::<bucketname>"]
    }
  ]
}

Lifecycle policy

A lifecycle policy is recommended that deletes incomplete multipart uploads after one day.

S3-compatible local stores for testing

Note
The tools in this section are provided for local testing and evaluation only. They have not been fully tested with Tempo and are not recommended for production use.

You can run an S3-compatible object store locally to test Tempo with the s3 storage backend.

SeaweedFS is the recommended option for local testing, with a single-command startup and a built-in web UI.

rclone serve s3 is an alternative that serves any local directory as an S3-compatible endpoint. It is classified as experimental by the rclone project and has known limitations.

MinIO is also supported but its open source repository has been archived and the community edition is now distributed as source code only. Pre-compiled binaries are no longer published.

Set up a local S3-compatible object store

Choose a tab below to set up your preferred object store:

Note
SeaweedFS has not been fully tested with Tempo and is provided here as an alternative for local evaluation only. It isn’t recommended for production use with Tempo.

SeaweedFS is an Apache 2.0-licensed distributed storage system with a built-in S3 gateway.

Download and install SeaweedFS from the releases page.
Create a data directory and start SeaweedFS in mini mode:
Bash
```
sudo mkdir -p /data/seaweedfs
sudo chown -R $USER:$USER /data/seaweedfs
weed mini -dir=/data/seaweedfs
```
The weed mini command starts a complete single-node setup including the S3 gateway on port 8333. SeaweedFS runs in the foreground, so open a new terminal for the remaining steps.
Create a bucket called tempo using the AWS CLI:
Bash
```
aws --endpoint-url http://localhost:8333 s3 mb s3://tempo --no-sign-request
```
You need the AWS CLI installed. SeaweedFS mini mode allows anonymous access, so the --no-sign-request flag skips credential checks.

Note
rclone serve s3 is classified as experimental by the rclone project and hasn’t been fully tested with Tempo. It’s provided as an alternative for local evaluation only and isn’t recommended for production use. Refer to the rclone documentation for current limitations.

rclone can serve any local directory as an S3-compatible endpoint.

Install rclone by following the rclone install guide.

Create a data directory and start the S3 server:

sudo mkdir -p /data/rclone-s3
sudo chown -R $USER:$USER /data/rclone-s3
rclone serve s3 /data/rclone-s3 --auth-key tempokey,temposecret --addr :8080

The server runs in the foreground on port 8080. Open a new terminal for the remaining steps.

Create a bucket called tempo using the AWS CLI:
Bash
```
AWS_ACCESS_KEY_ID=tempokey AWS_SECRET_ACCESS_KEY=temposecret \
  aws --endpoint-url http://localhost:8080 s3 mb s3://tempo
```
You need the AWS CLI installed. Use the credentials you set with the --auth-key flag.

Note
The MinIO open source repository has been archived and the community edition is now source-only. Pre-compiled binaries are no longer published. You must build MinIO from source using Go 1.24 or later.

Install MinIO from source:
Bash
```
go install github.com/minio/minio@latest
```
Refer to the MinIO repository for alternative installation methods including building a Docker image.
Create a data directory and start MinIO:
Bash
```
sudo mkdir -p /data/minio
sudo chown -R $USER:$USER /data/minio
minio server /data/minio --console-address ':9001'
```
By default, MinIO uses minioadmin for both the access key and secret key. MinIO runs in the foreground, so open a new terminal for the remaining steps.

Create a bucket called tempo using the MinIO Client (mc):

mc alias set local http://localhost:9000 minioadmin minioadmin
mc mb local/tempo

Tempo configuration for S3-compatible stores

The following example configuration uses the S3 backend. Replace the <S3_ENDPOINT>, <S3_ACCESS_KEY>, and <S3_SECRET_KEY> placeholders with the values for your object store.

This example configuration includes the metrics-generator. To disable it, remove the metrics_generator block and the processors list from the overrides.

stream_over_http_enabled: true

server:
  http_listen_port: 3200

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: "0.0.0.0:4317"
        http:
          endpoint: "0.0.0.0:4318"

backend_scheduler:
  provider:
    compaction:
      compaction:
        block_retention: 1h

backend_worker:
  compaction:
    block_retention: 1h

metrics_generator:
  registry:
    external_labels:
      source: tempo
      cluster: linux-monolithic
  storage:
    path: /tmp/tempo/generator/wal
    remote_write:
      - url: http://<PROMETHEUS_URL>/api/v1/write
        send_exemplars: true

storage:
  trace:
    backend: s3
    s3:
      endpoint: <S3_ENDPOINT>
      bucket: tempo
      access_key: <S3_ACCESS_KEY>
      secret_key: <S3_SECRET_KEY>
      insecure: true
    wal:
      path: /var/tempo/wal

overrides:
  defaults:
    metrics_generator:
      processors: [service-graphs, span-metrics]

usage_report:
  reporting_enabled: false

Replace the <PROMETHEUS_URL> placeholder with the address of your Prometheus-compatible storage instance (for example, localhost:9090). To disable the metrics-generator, remove the processors list from the overrides and the metrics_generator block.

Use the following endpoint and credential values for each object store:

SeaweedFS mini mode allows anonymous access, so the access_key and secret_key fields can be omitted or set to any value:

storage:
  trace:
    backend: s3
    s3:
      endpoint: localhost:8333
      bucket: tempo
      insecure: true

Use the credentials you set with the --auth-key flag when starting rclone:

storage:
  trace:
    backend: s3
    s3:
      endpoint: localhost:8080
      bucket: tempo
      access_key: tempokey
      secret_key: temposecret
      insecure: true

storage:
  trace:
    backend: s3
    s3:
      endpoint: localhost:9000
      bucket: tempo
      access_key: minioadmin
      secret_key: minioadmin
      insecure: true

Verify data in your S3-compatible store

After traces start flowing, verify that your storage bucket has received data:

Open the SeaweedFS admin UI at http://localhost:23646, or list the bucket contents using the AWS CLI:

aws --endpoint-url http://localhost:8333 s3 ls s3://tempo/ --recursive --no-sign-request

You should see files such as single-tenant/<block-id>/data.parquet and single-tenant/<block-id>/meta.json.

List the bucket contents using the AWS CLI:

AWS_ACCESS_KEY_ID=tempokey AWS_SECRET_ACCESS_KEY=temposecret \
  aws --endpoint-url http://localhost:8080 s3 ls s3://tempo/ --recursive

You should see files such as single-tenant/<block-id>/data.parquet and single-tenant/<block-id>/meta.json. rclone serve s3 does not provide a web UI.

Open the MinIO Console at http://localhost:9001 and check the tempo bucket for files such as work.json and a tenant data directory.

Was this page helpful?

Suggest an edit in GitHub

Create a GitHub issue

Email docs@grafana.com

Help and support

Community

Amazon S3 and S3-compatible storage

Authentication

IAM policy

Lifecycle policy

S3-compatible local stores for testing

Set up a local S3-compatible object store

Tempo configuration for S3-compatible stores

Verify data in your S3-compatible store

Was this page helpful?

Still have questions?

Get every update

Amazon S3 and S3-compatible storage

Authentication

IAM policy

Lifecycle policy

S3-compatible local stores for testing

Set up a local S3-compatible object store

Tempo configuration for S3-compatible stores

Verify data in your S3-compatible store

Was this page helpful?

Related resources from Grafana Labs