This is documentation for the next version of Grafana Tempo documentation. For the latest stable release, go to the latest version.

Documentation Grafana Tempo Manage Tempo Vulture

Tempo Vulture

Tempo Vulture is a testing tool that validates the end-to-end functionality of Tempo by continuously writing traces and verifying they can be read back correctly. It is useful for monitoring data integrity in production environments and for post-deployment validation testing.

Overview

Tempo Vulture performs the following operations:

• Write traces: Pushes test traces to Tempo using OTLP over gRPC
• Read traces by ID: Queries Tempo to retrieve traces by their trace ID
• Search traces: Uses TraceQL to search for traces by attributes
• Metrics queries: Validates TraceQL metrics queries (enabled by default)

Tempo Vulture can run in two modes:

• Continuous mode (default): Runs indefinitely, continuously writing and reading traces while exposing Prometheus metrics
• Validation mode: Executes a fixed number of write/read cycles and exits with a status code indicating success or failure

Run Tempo Vulture

Tempo Vulture is available as a Docker image:

Bash Copy

docker run grafana/tempo-vulture:latest [arguments...]

You can also build it from source:

Bash Copy

make tempo-vulture
./tempo-vulture [arguments...]

Configuration flags

Configure Tempo Vulture using command-line flags.

Required flags

Optional flags

Validation mode flags

Continuous mode

In continuous mode, Tempo Vulture runs indefinitely and exposes Prometheus metrics that can be used for monitoring and alerting.

Example

Bash Copy

docker run grafana/tempo-vulture:latest \
  --tempo-query-url=http://tempo:3200 \
  --tempo-push-url=http://tempo:4317 \
  --tempo-org-id=my-tenant \
  --tempo-write-backoff-duration=15s \
  --tempo-read-backoff-duration=30s \
  --tempo-search-backoff-duration=60s

Prometheus metrics

Tempo Vulture exposes the following key metrics:

Error types include:

• incorrectresult: Retrieved trace doesn’t match the expected trace
• incorrect_metrics_result: Metrics query returned unexpected results
• missingspans: Trace has missing spans
• notfound_byid: Trace not found by ID
• notfound_search: Trace not found via search
• notfound_traceql: Trace not found via TraceQL
• notfound_metrics: Trace not found via metrics query
• notfound_search_attribute: No searchable attribute found in trace
• inaccurate_metrics: Metrics count doesn’t match actual span count
• requestfailed: Request to Tempo failed

Alerting

You can configure alerts based on Tempo Vulture metrics. For example, the Tempo mixin includes a TempoVultureHighErrorRate alert that fires when the error rate exceeds a configurable threshold.

Validation mode

Validation mode runs Tempo Vulture as a one-shot test that can be integrated into CI/CD pipelines for post-deployment validation. Instead of running continuously, it executes a fixed number of write/read cycles and exits with a status code indicating success or failure.

Exit codes

Environment variables

Validation mode requires authentication:

Example

Bash Copy

docker run \
  -e TEMPO_ACCESS_POLICY_TOKEN=your-token \
  grafana/tempo-vulture:latest \
  --tempo-query-url=https://tempo.example.com \
  --tempo-push-url=https://tempo.example.com:4317 \
  --tempo-org-id=my-tenant \
  --validation-mode \
  --validation-cycles=5 \
  --validation-timeout=10m

CI/CD pipeline integration

Validation mode is useful for:

• Post-deployment testing: Verify Tempo is functioning correctly after a deployment
• Smoke testing: Quick validation that write and read paths are working
• Integration testing: Ensure end-to-end trace pipeline is operational

Example CI/CD usage in a GitHub Actions workflow:

YAML Copy

- name: Validate Tempo deployment
  run: |
    docker run \
      -e TEMPO_ACCESS_POLICY_TOKEN=${{ secrets.TEMPO_TOKEN }} \
      grafana/tempo-vulture:latest \
      --tempo-query-url=${{ vars.TEMPO_URL }} \
      --tempo-push-url=${{ vars.TEMPO_PUSH_URL }} \
      --tempo-org-id=${{ vars.TEMPO_ORG_ID }} \
      --validation-mode \
      --validation-cycles=3 \
      --validation-timeout=5m

Validation process

In validation mode, Tempo Vulture performs these steps:

1. For each cycle (default: 3):

   • Write a test trace to Tempo
   • Wait briefly for ingestion
   • Read the trace back by ID and verify it matches

2. After all cycles complete:

   • Wait for traces to be searchable (if search is enabled)
   • Search for each trace using a random attribute
   • Verify all traces are found

3. Exit with code 0 if all validations pass, or 1 if any fail

Docker Compose example

The Tempo repository includes Docker Compose examples that demonstrate how to run Tempo Vulture. Refer to the example configurations for complete setups.

Example from a Docker Compose file:

YAML Copy

vulture:
  image: grafana/tempo-vulture:latest
  command:
    - "-tempo-push-url=http://distributor:4317"
    - "-tempo-query-url=http://query-frontend:3200"
    - "-tempo-org-id="