API documentation

Tempo’s API

Tempo exposes an API for pushing and querying traces, and operating the cluster itself.

For the sake of clarity, in this document we have grouped API endpoints by service, but keep in mind that they’re exposed both when running Tempo in microservices and singly-binary mode:

  • Microservices: each service exposes its own endpoints
  • Single-binary: the Tempo process exposes all API endpoints for the services running internally

Endpoints

API Service Type Endpoint
Configuration All services HTTP GET /config
Readiness probe All services HTTP GET /ready
Metrics All services HTTP GET /metrics
Pprof All services HTTP GET /debug/pprof
Ingest traces Distributor - See section for details
Querying traces Query-frontend HTTP GET /api/traces/<traceID>
Query Path Readiness Check Query-frontend HTTP GET /api/echo
Memberlist Distributor, Ingester, Querier, Compactor HTTP GET /memberlist
Flush Ingester HTTP GET,POST /flush
Shutdown Ingester HTTP GET,POST /shutdown
Distributor ring status (*) Distributor HTTP GET /distributor/ring
Ingesters ring status Distributor, Querier HTTP GET /ingester/ring
Compactor ring status Compactor HTTP GET /compactor/ring

(*) This endpoint is not always available, check the specific section for more details.

Configuration

GET /config

Displays the configuration currently applied to Tempo (in YAML format), including default values and settings via CLI flags. Sensitive data is masked. Please be aware that the exported configuration doesn’t include the per-tenant overrides.

Readiness probe

GET /ready

Returns status code 200 when Tempo is ready to serve traffic.

Metrics

GET /metrics

Returns the metrics for the running Tempo service in the Prometheus exposition format.

Pprof

GET /debug/pprof/heap
GET /debug/pprof/block
GET /debug/pprof/profile
GET /debug/pprof/trace
GET /debug/pprof/goroutine
GET /debug/pprof/mutex

Returns the runtime profiling data in the format expected by the pprof visualization tool. There are many things which can be profiled using this including heap, trace, goroutine, etc.

For more information, please check out the official documentation of pprof.

Ingest

Tempo distributor uses the OpenTelemetry Receivers as a shim to ingest trace data. Note that these APIs are meant to be consumed by the corresponding client SDK or a pipeline service like Grafana Agent / OpenTelemetry Collector / Jaeger Agent.

Protocol Type Docs
OpenTelemetry GRPC Link
OpenTelemetry HTTP Link
Jaeger Thrift Compact Link
Jaeger Thrift Binary Link
Jaeger Thrift HTTP Link
Jaeger GRPC Link
Zipkin HTTP Link

For information on how to use the Zipkin endpoint with curl (for debugging purposes) check here.

Query

Tempo’s Query API is simple. The following request is used to retrieve a trace from the query frontend service in a microservices deployment, or the Tempo endpoint in a single binary deployment.

GET /api/traces/<traceid>

The following query API is also provided on the querier service for debugging purposes.

GET /querier/api/traces/<traceid>?mode=xxxx&blockStart=0000&blockEnd=FFFF

Parameters:

  • mode = (blocks|ingesters|all) Specifies whether the querier should look for the trace in blocks, ingesters or both (all). Default = all
  • blockStart = (GUID) Specifies the blockID start boundary. If specified, the querier will only search blocks with IDs > blockStart. Default = 00000000-0000-0000-0000-000000000000 Example: blockStart=12345678-0000-0000-1235-000001240000
  • blockEnd = (GUID) Specifies the blockID finish boundary. If specified, the querier will only search blocks with IDs < blockEnd. Default = FFFFFFFF-FFFF-FFFF-FFFF-FFFFFFFFFFFF Example: blockStart=FFFFFFFF-FFFF-FFFF-FFFF-456787652341

Note that this API is not meant to be used directly unless for debugging the sharding functionality of the query frontend.

Query Path Readiness Check

GET /api/echo

Returns status code 200 and body echo when the query frontend is up and ready to receive requests.

Note: Meant to be used in a Query Visualization UI like Grafana to test that the Tempo datasource is working.

Flush

GET,POST /flush

Triggers a flush of all in-memory traces to the WAL. Useful at the time of rollout restarts and unexpected crashes.

Shutdown

GET,POST /shutdown

Flushes all in-memory traces and the WAL to the long term backend. Gracefully exits from the ring. Shuts down the ingester service.

Note: This is usually used at the time of scaling down a cluster.

Distributor ring status

Note: this endpoint is only available when Tempo is configured with the global override strategy.

GET /distributor/ring

Displays a web page with the distributor hash ring status, including the state, healthy and last heartbeat time of each distributor.

For more information, check the page on consistent hash ring.

Ingesters ring status

GET /ingester/ring

Displays a web page with the ingesters hash ring status, including the state, healthy and last heartbeat time of each ingester.

For more information, check the page on consistent hash ring.

Compactor ring status

GET /compactor/ring

Displays a web page with the compactor hash ring status, including the state, healthy and last heartbeat time of each compactor.

For more information, check the page on consistent hash ring.