Documentation Grafana Cloud Alerts and IRM Machine Learning Sift Sift analyses Slow requests

Sift analysis - Slow requests

This analysis searches for slow traces during the investigation time period.

Inputs

• Required:
  • A Tempo datasource (This analysis will attempt to discover this datasource without the need for user input)
  • label/value pairs in the investigation which can be mapped to span or resource attributes in the Tempo datasource, such as cluster, namespace, node, or deployment. (Sift will convert known attributes like cluster to k8s.cluster.name to match OpenTelemetry Semantic Conventions)

How it works

Step 1: Evaluate any specified services

If services are specified, for each service, search Tempo for traces matching a few conditions where the duration is configurable:

• span.http.route != ”/metrics”
• span.http.route !~ “/debug/pprof*”
• duration > %ms
• resource.service.name = “$service”

Step 2: Search without specified services

If services are not specified, search Tempo for traces matching similar conditions, but instead of filtering using service name, convert labels to ‘known’ OTel semantic conventions if possible, and filter using those instead:

• span.http.route != ”/metrics”
• span.http.route !~ “/debug/pprof*”
• duration > %ms
• resource.k8s.namespace.name = “$namespace”
• resource.k8s.cluster.name = “$cluster”

Step 3: Return results

Return all matching traces.

What resources does this analysis use?

• Tempo’s search API

How does this analysis determine when a result is interesting?

• If there is at least one slow request, the result is considered interesting.

What configurations options are available for this analysis?

Threshold

The threshold above which traces are considered ‘slow’.

• Default: 3 seconds
• Minimum: 1 second