---
title: "TraceQL cookbook | Grafana Cloud documentation"
description: "Practical TraceQL cookbook with copy-paste queries for common tasks"
---

# TraceQL cookbook

Use these copy‑pasteable TraceQL recipes in Grafana Explore with a Tempo data source. Adjust service names, routes, and attributes to your environment.

All of these examples use the OpenTelemetry semantic conventions. Refer to [Trace semantic suggestions](https://opentelemetry.io/docs/specs/semconv/general/trace/) for OpenTelemetry for more information. Use your attribute names if they differ from these conventions.

> Tip
> 
> If a query returns no results, widen the time range and confirm the correct Tempo data source is selected. Prefer trace‑level intrinsic fields like `trace:duration` for speed.

## Tips for running queries

Before you run a query, check:

- Select the correct Tempo data source and tenant in Grafana Explore.
- Set a broad time range when starting out.
- Use the Search builder to sanity‑check that traces are present.
- Confirm your services are emitting spans (and the attributes you query).

To speed up your queries, use the following tips:

- Prefer `trace:*` intrinsic fields where possible (for example, `trace:duration`, `trace:rootService`).
- Filter early by service/time; add detail filters iteratively.
- Avoid broad regular expression filters until you’ve narrowed results.

## Basics

Select by service and operation name:

traceql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```traceql
{ resource.service.name = "frontend" && span:name = "POST /api/orders" }
```

Filter by environment/namespace:

traceql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```traceql
{
  resource.service.namespace = "ecommerce" &&
  resource.service.name = "frontend" &&
  resource.deployment.environment = "production"
}
```

HTTP success range (span attribute numeric):

traceql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```traceql
{ span.http.response.status_code >= 200 && span.http.response.status_code < 300 }
```

Find error spans (status type):

traceql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```traceql
{ span:status = error }
```

Prefer trace‑level intrinsic fields for speed (root name/service):

traceql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```traceql
{ trace:rootService = "gateway" && trace:rootName = "HTTP GET /health" }
```

Minimum total trace duration:

traceql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```traceql
{ trace:duration > 2s }
```

Array attribute contains value (vParquet4+):

traceql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```traceql
{ span.http.request.header.Accept = "application/json" }
```

Quoted attribute names (keys containing spaces or dots):

traceql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```traceql
{ span."attribute name with space" = "value" }
```

## HTTP and database

Find GET and DELETE requests:

traceql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```traceql
{ span.http.request.method =~ "GET|DELETE" }
```

5xx server errors on an endpoint:

traceql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```traceql
{ span:name = "POST /api/orders" && span.http.response.status_code >= 500 }
```

Slow endpoints (&gt; 500 ms span duration):

traceql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```traceql
{ span:duration > 500ms && span.http.target!= nil }
```

Specific DB systems:

traceql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```traceql
{ span.db.system =~ "postgresql|mysql" }
```

Database access during a given API call:

traceql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```traceql
{ span:name = "GET /api/products/{id}"} && { span.db.system = "postgresql" }
```

## Structural operators

Frontend calls a downstream service that errors (descendant):

traceql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```traceql
{ resource.service.name = "frontend" } >> { span:status = error }
```

Direct parent/child relationship (parent then child):

traceql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```traceql
{ resource.service.name = "api" } > { span:name = "SELECT product" }
```

Sibling services involved in the same trace:

traceql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```traceql
{ resource.service.name = "productcatalogservice" } ~ { resource.service.name = "frontend" }
```

Leaf spans in a service (experimental not‑parent):

traceql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```traceql
{ } !< { resource.service.name = "productcatalogservice" }
```

Return both the failing endpoint and all failing descendants (union structural):

traceql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```traceql
{ span:name = "POST /checkout" && span:status = error } &>> { span:status = error }
```

## Aggregation and grouping

Traces with more than 10 spans:

traceql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```traceql
count() > 10
```

At least 3 success spans:

traceql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```traceql
{ span.http.response.status_code = 200 } | count() > 3
```

Average span duration over 20 ms:

traceql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```traceql
avg(span:duration) > 20ms
```

Group by service to find services with multiple errors:

traceql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```traceql
{ span:status = error } | by(resource.service.name) | count() > 1
```

Select fields for fast tabular inspection:

traceql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```traceql
{ span:status = error } | select(span.http.response.status_code, span.url.full)
```

## Advanced scopes

Events (exceptions):

traceql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```traceql
{ event.exception.message =~ ".*something went wrong.*" }
```

Links (OpenTracing ref type):

traceql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```traceql
{ link.opentracing.ref_type = "child_of" }
```

Instrumentation scope by language:

traceql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```traceql
{ instrumentation.language = "java" }
```

List libraries producing spans for a service:

traceql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```traceql
{ resource.service.name = "frontend" } | rate() by (instrumentation:name)
```

## Service Graph → TraceQL

Error spike on a node (service):

traceql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```traceql
{ resource.service.name = "payments" && span:status = error }
```

High latency edge (API → DB):

traceql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```traceql
{ span.http.route = "/pay" && span.http.request.method = "POST" } >> { span.db.system = "postgresql" && span:duration > 200ms }
```

Alternative (span name):

traceql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```traceql
{ span:name = "POST /pay" } >> { span.db.system = "postgresql" && span:duration > 200ms }
```

## TraceQL metrics examples

Run these in Metrics mode in Explore. TraceQL metrics queries typically have a 24‑hour default time‑range limit.

Error count by service:

traceql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```traceql
{ span:status = error } | count() by (resource.service.name)
```

Average span duration by route:

traceql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```traceql
{ span.http.route != nil } | avg(span:duration) by (span.http.route)
```

Traces by environment:

traceql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```traceql
{ resource.deployment.environment != nil } | count() by (resource.deployment.environment)
```

## Common attribute keys

Expand table

| Group       | Keys                                                                                                              |
|-------------|-------------------------------------------------------------------------------------------------------------------|
| HTTP/URL    | `span.http.request.method`, `span.http.response.status_code`, `span.http.route`, `span.url.path`, `span.url.full` |
| Database    | `span.db.system`, `span.db.name`, `span.db.statement` (if present)                                                |
| Service/Env | `resource.service.name`, `resource.deployment.environment`, `resource.service.namespace`                          |

For more details, refer to the OpenTelemetry Trace semantic conventions: [https://opentelemetry.io/docs/specs/semconv/general/trace/](https://opentelemetry.io/docs/specs/semconv/general/trace/)

## Use dashboard variables

Replace hard‑coded values with dashboard variables when using these queries in panels. For example, using `$service`:

traceql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```traceql
{ resource.service.name = "$service" && span:status = error }
```

## More resources

- TraceQL reference and concepts: [Construct a TraceQL query](/docs/tempo/next/traceql/construct-traceql-queries/)
- Query editor modes in Grafana: [Query tracing data](/docs/grafana-cloud/send-data/traces/traces-query-editor/)

> Note
> 
> This cookbook uses stable operators and attributes. Experimental operators and hints (for example, `most_recent=true`) are intentionally excluded. Feature availability can vary by Tempo version; refer to the reference docs when in doubt.