Documentation Grafana Cloud Instrument and send data Traces Query tracing data TraceQL cookbook

TraceQL cookbook

Use these copy‑pasteable TraceQL recipes in Grafana Explore with a Tempo data source. Adjust service names, routes, and attributes to your environment.

All of these examples use the OpenTelemetry semantic conventions. Refer to Trace semantic suggestions for OpenTelemetry for more information. Use your attribute names if they differ from these conventions.

Tips for running queries

Before you run a query, check:

• Select the correct Tempo data source and tenant in Grafana Explore.
• Set a broad time range when starting out.
• Use the Search builder to sanity‑check that traces are present.
• Confirm your services are emitting spans (and the attributes you query).

To speed up your queries, use the following tips:

• Prefer trace:* intrinsic fields where possible (for example, trace:duration, trace:rootService).
• Filter early by service/time; add detail filters iteratively.
• Avoid broad regular expression filters until you’ve narrowed results.

Basics

Select by service and operation name:

traceql Copy

{ resource.service.name = "frontend" && span:name = "POST /api/orders" }

Filter by environment/namespace:

traceql Copy

{
  resource.service.namespace = "ecommerce" &&
  resource.service.name = "frontend" &&
  resource.deployment.environment = "production"
}

HTTP success range (span attribute numeric):

traceql Copy

{ span.http.response.status_code >= 200 && span.http.response.status_code < 300 }

Find error spans (status type):

traceql Copy

{ span:status = error }

Prefer trace‑level intrinsic fields for speed (root name/service):

traceql Copy

{ trace:rootService = "gateway" && trace:rootName = "HTTP GET /health" }

Minimum total trace duration:

traceql Copy

{ trace:duration > 2s }

Array attribute contains value (vParquet4+):

traceql Copy

{ span.http.request.header.Accept = "application/json" }

Quoted attribute names (keys containing spaces or dots):

traceql Copy

{ span."attribute name with space" = "value" }

HTTP and database

Find GET and DELETE requests:

traceql Copy

{ span.http.request.method =~ "GET|DELETE" }

5xx server errors on an endpoint:

traceql Copy

{ span:name = "POST /api/orders" && span.http.response.status_code >= 500 }

Slow endpoints (> 500 ms span duration):

traceql Copy

{ span:duration > 500ms && span.http.target!= nil }

Specific DB systems:

traceql Copy

{ span.db.system =~ "postgresql|mysql" }

Database access during a given API call:

traceql Copy

{ span:name = "GET /api/products/{id}"} && { span.db.system = "postgresql" }

Structural operators

Frontend calls a downstream service that errors (descendant):

traceql Copy

{ resource.service.name = "frontend" } >> { span:status = error }

Direct parent/child relationship (parent then child):

traceql Copy

{ resource.service.name = "api" } > { span:name = "SELECT product" }

Sibling services involved in the same trace:

traceql Copy

{ resource.service.name = "productcatalogservice" } ~ { resource.service.name = "frontend" }

Leaf spans in a service (experimental not‑parent):

traceql Copy

{ } !< { resource.service.name = "productcatalogservice" }

Return both the failing endpoint and all failing descendants (union structural):

traceql Copy

{ span:name = "POST /checkout" && span:status = error } &>> { span:status = error }

Aggregation and grouping

Traces with more than 10 spans:

traceql Copy

count() > 10

At least 3 success spans:

traceql Copy

{ span.http.response.status_code = 200 } | count() > 3

Average span duration over 20 ms:

traceql Copy

avg(span:duration) > 20ms

Group by service to find services with multiple errors:

traceql Copy

{ span:status = error } | by(resource.service.name) | count() > 1

Select fields for fast tabular inspection:

traceql Copy

{ span:status = error } | select(span.http.response.status_code, span.url.full)

Advanced scopes

Events (exceptions):

traceql Copy

{ event.exception.message =~ ".*something went wrong.*" }

Links (OpenTracing ref type):

traceql Copy

{ link.opentracing.ref_type = "child_of" }

Instrumentation scope by language:

traceql Copy

{ instrumentation.language = "java" }

List libraries producing spans for a service:

traceql Copy

{ resource.service.name = "frontend" } | rate() by (instrumentation:name)

Service Graph → TraceQL

Error spike on a node (service):

traceql Copy

{ resource.service.name = "payments" && span:status = error }

High latency edge (API → DB):

traceql Copy

{ span.http.route = "/pay" && span.http.request.method = "POST" } >> { span.db.system = "postgresql" && span:duration > 200ms }

Alternative (span name):

traceql Copy

{ span:name = "POST /pay" } >> { span.db.system = "postgresql" && span:duration > 200ms }

TraceQL metrics examples

Run these in Metrics mode in Explore. TraceQL metrics queries typically have a 24‑hour default time‑range limit.

Error count by service:

traceql Copy

{ span:status = error } | count() by (resource.service.name)

Average span duration by route:

traceql Copy

{ span.http.route != nil } | avg(span:duration) by (span.http.route)

Traces by environment:

traceql Copy

{ resource.deployment.environment != nil } | count() by (resource.deployment.environment)

Common attribute keys

For more details, refer to the OpenTelemetry Trace semantic conventions: https://opentelemetry.io/docs/specs/semconv/general/trace/

Use dashboard variables

Replace hard‑coded values with dashboard variables when using these queries in panels. For example, using $service:

traceql Copy

{ resource.service.name = "$service" && span:status = error }

More resources

• TraceQL reference and concepts: Construct a TraceQL query
• Query editor modes in Grafana: Query tracing data