Important: This documentation is about an older version. It's relevant only to the release noted, many of the features and functions have been updated or replaced. Please view the current version.

Deployment and operations

Backend search - serverless GCP setup

Open source

Google Cloud Run for serverless backend search

This document walks you through setting up a Google Cloud Run for serverless backend search. For more guidance on configuration options for full backend search check here.

Build the docker image:

cd ./cmd/tempo-serverless && make build-docker-gcr

This will create the Docker container image to be deployed to Google Cloud Run. The docker image will be named: tempo-serverless:latest and tempo-serverless:<branch>-<commit hash>. Here is an example of that name:

$ docker images | grep tempo-serverless
tempo-serverless                                                           cloud-run-3be4efa               146c9d9fa63c   58 seconds ago   47.9MB
tempo-serverless                                                           latest                          146c9d9fa63c   58 seconds ago   47.9MB

Push the image to a Google Container Registry repo.

Provision the Google Cloud Run service. This example uses Terraform. Configuration values should be adjusted to meet the needs of your installation.

locals {
  // this can be increased if you would like to use multiple functions
  count = 1
}

resource "google_cloud_run_service" "run" {
  count = local.count

  name     = "<service name>"
  location = "<appropriate region>"

  metadata {
    annotations = {
        "run.googleapis.com/ingress"      = "internal",     # this annotation can be used to limit connectivity to the service
    }
  }

  template {
    metadata {
      annotations = {
          "autoscaling.knative.dev/minScale"                   = "1",
          "autoscaling.knative.dev/maxScale"                   = "1000",
          "autoscaling.knative.dev/panic-threshold-percentage" = "110.0",  # default 200.0. how aggressively to go into panic mode and start scaling heavily
          "autoscaling.knative.dev/window"                     = "10s",    # default 60s. window over which to average metrics to make scaling decisions
      }
    }
    spec {
      container_concurrency = 4
      containers {
        image = "<container image created above>"
        resources {
          limits = {
              cpu = "2"
              memory = "1Gi"
          }
        }
        env {
          name = "TEMPO_GCS_BUCKET_NAME"
          value = "<gcs bucket where tempo data is stored>"
        }
        env {
          name = "TEMPO_BACKEND"
          value = "gcs"
        }
        env {
          name = "TEMPO_GCS_HEDGE_REQUESTS_AT"
          value = "400ms"
        }
        env {
          name = "TEMPO_GCS_HEDGE_REQUESTS_UP_TO"
          value = "2"
        }
        env {
          name = "GOGC"
          value = "400"
        }
      }
    }
  }

  traffic {
    percent         = 100
    latest_revision = true
  }
}

Add the newly-created cloud run service as external endpoints in your querier configuration. The endpoint can be retrieved from the “details” tab in Google Cloud Run:
```
querier:
  search:
    external_endpoints:
    - <trigger url from console>
```