DocumentationGrafana CloudAlerts and IRMMachine LearningSiftSlow requests

Sift analysis - Slow requests

Overview

This analysis searches for slow traces during the investigation time period.

Inputs

• Required:

  • A Tempo datasource (This analysis will attempt to discover this datasource without the need for user input)

• Optional:

  • None

How it works

Step 1: Evaluate any specified services

If services are specified, for each service, search Tempo for traces matching a few conditions where the duration is configurable:

• span.http.route != ”/metrics”
• span.http.route !~ “/debug/pprof*”
• duration > %ms
• resource.service.name = “$service”

Step 2: Search without specified services

If services are not specified, search Tempo for traces matching similar conditions, but instead of filtering using service name, convert labels to ‘known’ OTel semantic conventions if possible, and filter using those instead:

• span.http.route != ”/metrics”
• span.http.route !~ “/debug/pprof*”
• duration > %ms
• resource.k8s.namespace.name = “$namespace”
• resource.k8s.cluster.name = “$cluster”

Step 3: Return results

Return all matching traces.

What resources does this analysis use?

• Tempo’s search API

How does this analysis determine when a result is interesting?

• If there is at least one slow request, the result is considered interesting.

What configurations options are available for this analysis?

Threshold

The threshold above which traces are considered ‘slow’.

• Default: 3 seconds
• Minimum: 1 second