Tempo HTTP API on Grafana Labs

Metrics summary API

Thu, 09 Apr 2026 12:40:42 +0000

Metrics summary API

Warning
The metrics summary API is deprecated as of Tempo 2.7. Features powered by the metrics summary API, like the Aggregate by table, are also deprecated in Grafana Cloud and Grafana 11.3 and later. It will be removed in a future release.

This document explains how to use the metrics summary API in Tempo. This API returns RED metrics (span count, erroring span count, and latency information) for kind=server spans sent to Tempo in the last hour, grouped by a user-specified attribute.

Deprecation in favor of TraceQL metrics

The metrics summary API is now redundant given the release of TraceQL metrics. This API is therefore being deprecated to reduce our maintenance burden.

TraceQL metrics queries are significantly more powerful than what metrics summary API provides. TraceQL metrics can look at arbitrary time windows (not just the last hour), return time series information (rather than just a single instant value over the past hour), and can look at all spans (not just kind=server).

To provide an example, if you were to aggregate by the attribute resource.cloud.region with the metrics summary API, you could get the same results with a couple TraceQL queries:

Rate of requests by resource.cloud.region

{ } | rate() by (resource.cloud.region)

Error rate by resource.cloud.region

{ status=error } | rate() by (resource.cloud.region)

The 50th, 90th, and 99th percentile latencies (for example, p99, p90, and p50) by resource.cloud.region:

{ } | quantile_over_time(duration, .99, .9, .5) by (resource.cloud.region)

If you want something faster than typing these queries out in Explore’s code mode, use Grafana Traces Drilldown, a queryless experience for navigating your trace data stored in Tempo and powered by TraceQL metrics queries under the hood.

Activate metrics summary

To enable the deprecated) metrics summary API, you must turn on the local blocks processor in the metrics generator. Be aware that the generator uses considerably more resources, including disk space, if it’s enabled:

overrides:
  defaults:
    metrics_generator:
      processors: [..., 'local-blocks']

In Grafana and Grafana Cloud, the Metrics summary API is disabled by default. To enable it in Grafana Cloud, contact Grafana Support.

Request

To make a request to this API, use the following endpoint on the query-frontend:

GET http://<tempo>/api/metrics/summary

Query Parameters

All query parameters must be URL-encoded to preserve non-URL-safe characters in the query such as &.

Name	Examples	Definition	Required?
`q`	`{ resource.service.name = "foo" && span.http.status_code != 200 }`	The TraceQL query with full syntax. All spans matching this query are included in the calculations. Any valid TraceQL query is supported.	Yes
`groupBy`	`name` `.foo` `resource.namespace` `span.http.url,span.http.status_code`	One or more TraceQL values to group by. Any valid intrinsic or attribute with scope. To group by multiple values use a comma-delimited list.	Yes
`start`	1672549200	Start of time range in Unix seconds. If not specified, then all recent data is queried.	No
`end`	1672549200	End of the time range in Unix seconds. If not specified, then all recent data is queried.	No

Example:

curl "$URL/api/metrics/summary" --data-urlencode 'q={resource.service.name="checkout-service"}' --data-urlencode 'groupBy=name'

Response

The Tempo response is a SpanMetricsSummary object defined in tempo.proto, relevant section pasted below:

message SpanMetricsSummaryResponse {
  repeated SpanMetricsSummary summaries = 1;
}

message SpanMetricsSummary {
  uint64 spanCount = 1;
  uint64 errorSpanCount = 2;
  TraceQLStatic static = 3;
  uint64 p99 = 4;
  uint64 p95 = 5;
  uint64 p90 = 6;
  uint64 p50 = 7;
}

message TraceQLStatic {
  int32 type = 1;
  int64 n = 2;
  double f = 3;
  string s = 4;
  bool b = 5;
  uint64 d = 6;
  int32 status = 7;
  int32 kind = 8;
}

The response is returned as JSON following standard protobuf->JSON mapping rules.

Note
The uint64 fields can’t be fully expressed by JSON numeric values so the fields are serialized as strings.

Example:

{
   "summaries": [
       {
           "spanCount": "20",
           "series" : [
               {
                   "key": ".attr1",
                   "value": {
                       "type": 5,
                       "s": "foo"
                   },
               },
               ...
           ],
           "p99": "68719476736",
           "p95": "1073741824",
           "p90": "1017990479",
           "p50": "664499239"
       },

Field	Notes
`summaries`	The list of metrics per group.
`.spanCount`	Number of spans in this group.
`.errorSpanCount`	Number of spans with `status`=`error`. (This field isn’t present if the value is `0`.)
`.series`	The unique values for this group. A key/value pair is returned for each entry in `groupBy`.
`.key`	Key name.
`.value`	Value with TraceQL underlying type.
`.type`	Data type enum`` defined [here](https://github.com/grafana/tempo/blob/main/pkg/traceql/enum_statics.go#L8) (This field will not be present if the value is 0`.) <br/>0 =` nil`<br/>3 =` integer`<br/> 4 =` float`<br/> 5 =`string`<br/> 6 =` bool`<br/> 7 =` duration` 8 = span status 9 = span kind
`.n`	Populated if this is an integer value.
`.s`	Populated if this is a string value.
`.f`	Populated if this is a float value.
`.b`	Populated if this is a boolean value.
`.d`	Populated if this is a duration value.
`.status`	Populated if this is a span status value.
`.kind`	Populated if this is a span kind value.
`.p99`	The p99 latency of this group in nanoseconds.
`.p95`	The p95 latency of this group in nanoseconds.
`.p90`	The p90 latency of this group in nanoseconds.
`.p50`	The p50 latency of this group in nanoseconds.

Tempo MCP server

Thu, 09 Apr 2026 12:40:42 +0000

Model Context Protocol (MCP) Server

Tempo includes an MCP (Model Context Protocol) server that provides AI assistants and Large Language Models (LLMs) with direct access to distributed tracing data through TraceQL queries and other endpoints.

For examples on how you can use the MCP server, refer to LLM-powered insights into your tracing data: introducing MCP support in Grafana Cloud Traces.

For more information on MCP, refer to the MCP documentation.

Configuration

Enable the MCP server in your Tempo configuration:

query_frontend:
  mcp_server:
    enabled: true

Warning
Be aware that using this feature will likely cause tracing data to be passed to an LLM or LLM provider. Consider the content of your tracing data and organizational policies when enabling this.

Quick start

To experiment with the MCP server using dummy data and Claude Code:

Run the local docker-compose example in /example/docker-compose/local. This exposes the MCP server at http://localhost:3200/api/mcp
Run claude mcp add --transport=http tempo http://localhost:3200/api/mcp to add a reference to Claude Code.
Run claude and ask some questions.

This MCP server has also been tested successfully in cursor using the mcp-remote package.

Push spans with HTTP

Thu, 09 Apr 2026 12:40:42 +0000

Push spans with HTTP

Sometimes using a tracing system is intimidating because it seems like you need complex application instrumentation or a span ingestion pipeline in order to push spans. This guide aims to show an extremely basic technique for pushing spans with HTTP/JSON from a Bash script using the OpenTelemetry receiver.

Before you begin

This procedure uses an example Docker Compose setup to run Tempo, so you don’t need an existing installation. The Docker image also includes a Grafana container which lets you visualize traces.

To use this procedure, you need to have Docker and docker compose installed.

Start Tempo using the quick start

Use the instructions in the Quick start for Tempo documentation to start a local instance of Tempo and Grafana.

Push spans with OTLP

Now that Tempo is running and listening on port 4318 for OTLP spans, you can push a span to Tempo using curl.

Before you can use this example, you need to update the start and end time.

The startTimeUnixNano and endTimeUnixNano fields are in nanoseconds, where 100000000 nanoseconds is 100 milliseconds. The nanosecond value can be obtained by any tool that provides the epoch date in nanoseconds. For example, under Linux, you can use date +%s%N. You can also use an online tool such as Epoch Converter to get the current time in nanoseconds.

Copy and paste the curl command into a text editor.

curl -X POST -H 'Content-Type: application/json' http://localhost:4318/v1/traces -d '
{
	"resourceSpans": [{
    	"resource": {
        	"attributes": [{
            	"key": "service.name",
            	"value": {
                	"stringValue": "my.service"
            	}
        	}]
    	},
    	"scopeSpans": [{
        	"scope": {
            	"name": "my.library",
            	"version": "1.0.0",
            	"attributes": [{
                	"key": "my.scope.attribute",
                	"value": {
                    	"stringValue": "some scope attribute"
                	}
            	}]
        	},
        	"spans": [
        	{
            	"traceId": "5B8EFFF798038103D269B633813FC700",
            	"spanId": "EEE19B7EC3C1B100",
            	"name": "I am a span!",
            	"startTimeUnixNano": 1689969302000000000,
            	"endTimeUnixNano": 1689970000000000000,
            	"kind": 2,
            	"attributes": [
            	{
                	"key": "my.span.attr",
                	"value": {
                    	"stringValue": "some value"
                	}
            	}]
        	}]
    	}]
	}]
}'

Replace startTimeUnixNano and endTimeUnixNano with current values for the last 24 hours to allow you to search for them using a 24 hour relative time range. You can get this in seconds and milliseconds from the Unix Epoch Clock. For startTimeUnixNano, use the current time minus 24 hours. Use the link above to figure out the value. For endTimeUnixNano, use the current time.

Multiply the milliseconds value by 1,000,000 to turn it into nanoseconds. You can do this from a bash terminal with the following command. Replace <epochTimeMilliseconds> with the milliseconds value.
Bash
```
echo $((<epochTimeMilliseconds> * 1000000))
```
Copy the updated curl command to a terminal window and run it.

View the trace in Grafana:
1. Open a browser window to http://localhost:3000.
2. Open the Explorer page and select the Tempo data source.
3. Select the Search query type.
4. Select Run query to list available traces.
5. Select the trace ID (yellow box) to view details about the trace and its spans.

Retrieve traces

The easiest way to get the trace is to execute a simple curl command to Tempo. The returned format is OTLP.

Replace the trace ID in the curl command with the trace ID that was generated from the push. This information is in the data that’s sent with the curl. You could use Grafana’s Explorer page to find this, as shown in the previous section.

curl http://localhost:3200/api/v2/traces/5b8efff798038103d269b633813fc700

{"trace": {"resourceSpans":[{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"my.service"}}]},"scopeSpans":[{"scope":{"name":"my.library","version":"1.0.0"},"spans":[{"traceId":"W47/95gDgQPSabYzgT/HAA==","spanId":"7uGbfsPBsQA=","name":"I am a span!","kind":"SPAN_KIND_SERVER","startTimeUnixNano":"1689969302000000000","endTimeUnixNano":"1689970000000000000","attributes":[{"key":"my.span.attr","value":{"stringValue":"some value"}}],"status":{}}]}]}]}}

Copy and paste the updated curl command into a terminal window.

Use TraceQL to search for a trace

Alternatively, you can also use TraceQL to search for the trace that was pushed. You can search by using the unique trace attributes that were set:

curl -G -s http://localhost:3200/api/search --data-urlencode 'q={ .service.name = "my.service" }'

{"traces":[{"traceID":"5b8efff798038103d269b633813fc700","rootServiceName":"my.service","rootTraceName":"I am a span!","startTimeUnixNano":"1694718625557000000","durationMs":10000,"spanSet":{"spans":[{"spanID":"eee19b7ec3c1b100","startTimeUnixNano":"1694718625557000000","durationNanos":"10000000000","attributes":[{"key":"service.name","value":{"stringValue":"my.service"}}]}],"matched":1},"spanSets":[{"spans":[{"spanID":"eee19b7ec3c1b100","startTimeUnixNano":"1694718625557000000","durationNanos":"10000000000","attributes":[{"key":"service.name","value":{"stringValue":"my.service"}}]}],"matched":1}]}],"metrics":{"inspectedBytes":"292781","completedJobs":1,"totalJobs":1}}

To format this in a more human-readable output, consider using a tool such as jq, which lets you to run the same curl command and pipe it to jq to format the block. For example:

curl -G -s http://localhost:3200/api/search --data-urlencode 'q={ .service.name = "my.service" }' | jq

Spans from everything

Tracing isn’t limited to enterprise languages with complex frameworks. As you can see, it’s easy to store and track events from your js, python or bash scripts. You can use Tempo and distributed tracing today to trace CI pipelines, long running bash processes, python data processing flows, or anything else you can think of.

Happy tracing!