DocumentationGrafana TempoManageSearch with AWS Lambda

Search with AWS Lambda

This document explains how to set up AWS Lambda for serverless backend search. Read improve search performance for more guidance on configuration options for backend search.

1. Build the code package:

   bash Copy

   cd ./cmd/tempo-serverless && make build-lambda-zip

   This will create a ZIP file containing the binary required for the function. The file name will be of the form: ./lambda/tempo-<branch name>-<commit hash>.zip. Here is an example of that name:

   bash Copy

   ls lambda/*.zip
   lambda/tempo-serverless-backend-search-297172a.zip

2. Provision an S3 bucket.

3. Copy the ZIP file into your bucket.

   Copy

   aws s3 cp lambda/tempo-serverless-backend-search-297172a.zip gs://<newly provisioned gcs bucket>

4. Provision the Lambda. For a Lambda function to be invoked via HTTP, we also need to create an ALB and some other resources. This example uses Terraform and only includes the function definition. Additionally, you will need a VPC, security groups, IAM roles, an ALB, target groups, etc., but that is beyond the scope of this guide.

   Copy

   locals {
     // this can be increased if you would like to use multiple functions
     count = 1
   }
   
   resource "aws_lambda_function" "lambda" {
       count = local.count
   
       function_name    = "${local.name}-${count.index}"
       description      = "${local.name}-${count.index}"
       role             = <arn of appropriate role>
       handler          = "main"
       runtime          = "go1.x"
       timeout          = "60"
       s3_key           = "tempo-serverless-backend-search-297172a.zip"
       s3_bucket        = <S3 bucket created above>
       memory_size      = 1769 # 1 vcpu
   
       vpc_config {
           subnet_ids         = <appropriate subnets>
           security_group_ids = [<appropriate security groups>]
       }
   
       environment {
           variables = {
           "TEMPO_S3_BUCKET"               = "<S3 bucket name backing your Tempo instance>"
           "TEMPO_BACKEND"                 = "s3"
           "TEMPO_S3_HEDGE_REQUESTS_AT"    = "400ms"
           "TEMPO_S3_HEDGE_REQUESTS_UP_TO" = "2"
           "TEMPO_S3_ENDPOINT"             = "s3.dualstack.us-east-2.amazonaws.com"
           }
       }
   }

5. Add the hostname of the newly-created ALB to the querier config:

   Copy

   querier:
     search:
       external_endpoints:
       - http://<alb dns hostname>