Important: This documentation is about an older version. It's relevant only to the release noted, many of the features and functions have been updated or replaced. Please view the current version.

Documentation Tempo documentation Deployment and operations Backend search - serverless GCP setup

Google Cloud Run for serverless backend search

This document walks you through setting up a Google Cloud Run for serverless backend search. For more guidance on configuration options for full backend search check here.

1. Build the docker image:

   bash Copy

   cd ./cmd/tempo-serverless && make build-docker-gcr

   This will create the Docker container image to be deployed to Google Cloud Run. The docker image will be named: tempo-serverless:latest and tempo-serverless:<branch>-<commit hash>. Here is an example of that name:

   bash Copy

   $ docker images | grep tempo-serverless
   tempo-serverless                                                           cloud-run-3be4efa               146c9d9fa63c   58 seconds ago   47.9MB
   tempo-serverless                                                           latest                          146c9d9fa63c   58 seconds ago   47.9MB

2. Push the image to a Google Container Registry repo.

3. Provision the Google Cloud Run service. This example uses Terraform. Configuration values should be adjusted to meet the needs of your installation.

   Copy

   Expand code

   locals {
     // this can be increased if you would like to use multiple functions
     count = 1
   }
   
   resource "google_cloud_run_service" "run" {
     count = local.count
   
     name     = "<service name>"
     location = "<appropriate region>"
   
     metadata {
       annotations = {
           "run.googleapis.com/ingress"      = "internal",     # this annotation can be used to limit connectivity to the service
       }
     }
   
     template {
       metadata {
         annotations = {
             "autoscaling.knative.dev/minScale"                   = "1",
             "autoscaling.knative.dev/maxScale"                   = "1000",
             "autoscaling.knative.dev/panic-threshold-percentage" = "110.0",  # default 200.0. how aggressively to go into panic mode and start scaling heavily
             "autoscaling.knative.dev/window"                     = "10s",    # default 60s. window over which to average metrics to make scaling decisions
         }
       }
       spec {
         container_concurrency = 4
         containers {
           image = "<container image created above>"
           resources {
             limits = {
                 cpu = "2"
                 memory = "1Gi"
             }
           }
           env {
             name = "TEMPO_GCS_BUCKET_NAME"
             value = "<gcs bucket where tempo data is stored>"
           }
           env {
             name = "TEMPO_BACKEND"
             value = "gcs"
           }
           env {
             name = "TEMPO_GCS_HEDGE_REQUESTS_AT"
             value = "400ms"
           }
           env {
             name = "TEMPO_GCS_HEDGE_REQUESTS_UP_TO"
             value = "2"
           }
           env {
             name = "GOGC"
             value = "400"
           }
         }
       }
     }
   
     traffic {
       percent         = 100
       latest_revision = true
     }
   }

4. Add the newly-created cloud run service as external endpoints in your querier configuration. The endpoint can be retrieved from the “details” tab in Google Cloud Run:

   Copy

   querier:
     search:
       external_endpoints:
       - <trigger url from console>