Documentation Grafana Cloud Telemetry signals Use signals together Troubleshooting

Troubleshoot signal correlation

This guide helps you diagnose and fix common signal correlation issues.

Can’t navigate from metrics to logs

Symptoms

• Clicking a metric doesn’t show related logs.
• Logs panel is empty when navigating from metrics.
• “No logs found” message appears.

Possible causes

• Labels don’t match between data sources.
• Time ranges are misaligned.
• Label names are different (for example, service vs service_name).
• No logs exist for that time range or label combination.

Solutions

Verify label names match:

1. Check Prometheus metric labels:

   promql Copy

   up{service="api"}

2. Check Loki log labels:

   logql Copy

   {service="api"}

3. Compare label names. They must be identical (case-sensitive).

If label names differ:

Use Alloy or collector configuration to normalize labels:

Alloy Copy

// Prometheus remote write
prometheus.remote_write "default" {
  external_labels = {
    service = "api",
  }
}

// Loki write
loki.write "default" {
  external_labels = {
    service = "api",  // Use same name as Prometheus
  }
}

Verify labels have the same values:

promql Copy

up{service="api"}  # Prometheus shows: service="api"

logql Copy

{service="api"}    # Loki must also show: service="api"

If values differ (for example, api vs api-service), fix in your instrumentation or configuration.

Check time ranges:

Ensure both queries use the same time range. In Drilldown or Explore, the time range selector should show the same values for both panes.

Trace IDs don’t appear in logs

Symptoms

• Log entries don’t show trace IDs.
• No clickable links to traces in log view.
• Derived fields don’t work.

Possible causes

• Trace context not propagated in application.
• Logging format doesn’t include trace context.
• OpenTelemetry SDK not configured correctly.

Solutions

Verify application propagates trace context:

Check your application code includes trace context in logs. For OpenTelemetry-instrumented apps:

Python:

Python Copy

from opentelemetry.instrumentation.logging import LoggingInstrumentor

# Must be called during app initialization
LoggingInstrumentor().instrument()

Go:

Go Copy

import (
    "go.opentelemetry.io/contrib/instrumentation/log/slog"
)

// Use OpenTelemetry-aware logger
logger := slog.New(slog.NewJSONHandler(os.Stdout))

Check log output format:

View raw logs to confirm trace IDs are present:

Bash Copy

kubectl logs <pod-name> | grep -i trace

Expected output should include trace_id:

Bash Copy

{"level":"info","trace_id":"abc123...","msg":"request processed"}

If trace IDs are missing:

Add manual instrumentation to include trace context:

Python Copy

import logging
from opentelemetry import trace

logger = logging.getLogger(__name__)

def handle_request():
    span = trace.get_current_span()
    trace_id = format(span.get_span_context().trace_id, "032x")

    logger.info("Processing request", extra={"trace_id": trace_id})

Derived fields don’t work

Symptoms

• Trace IDs appear in logs but aren’t clickable.
• Clicking trace ID shows “No trace found”.
• Derived field configured but not appearing.

Possible causes

• Regex doesn’t match log format.
• Derived field not configured correctly.
• Tempo data source not selected.

Solutions

Test your regular expression pattern:

1. Find a log entry with a trace ID.
2. Copy the full log line.
3. Test your regular expression against it.

Example log line:

Bash Copy

{"timestamp":"2024-01-01T12:00:00Z","trace_id":"abc123def456","level":"info"}

Matching regular expression:

Bash Copy

"trace_id":\s*"(\w+)"

Verify derived field configuration:

1. In Grafana Cloud, go to Connections > Data sources.
2. Select your Loki data source.
3. Check Derived fields section:
   • Name: Must be unique (for example, traceId)
   • Regex: Must match your log format
   • Internal link: Must be toggled ON
   • Data source: Must select Tempo data source
   • Query: Should be ${__value.raw}

Test the configuration:

Query logs that should have trace IDs:

logql Copy

{service="api"} | json | trace_id != ""

Look for trace IDs that are underlined/clickable. If they’re not clickable, the regular expression or data source configuration is incorrect.

Exemplars aren’t showing on graphs

Symptoms

• No diamond points appear on Time series graphs.
• Can’t click from metrics to traces.
• Exemplars enabled but not visible.

Possible causes

• Application not emitting exemplars.
• Panel type isn’t Time series.
• Exemplars not enabled in panel settings.
• send_exemplars not enabled in Alloy/Collector.

Solutions

Verify application emits exemplars:

Check your metrics endpoint:

Bash Copy

curl -H "Accept: application/openmetrics-text" http://your-app:9090/metrics | grep -i "traceid"

Expected output:

Bash Copy

http_request_duration_seconds_bucket{le="0.1"} 45 # {trace_id="abc123"} 0.05

If exemplars are missing:

Update your instrumentation to include exemplars:

Go Copy

histogram.ObserveWithExemplar(
    duration,
    prometheus.Labels{"trace_id": traceID},
)

Check panel configuration:

1. Edit the dashboard panel.
2. Verify Panel type is “Time series” (not “Graph”).
3. In the legend, toggle Exemplars ON.

Verify Alloy/collector configuration:

Alloy Copy

prometheus.remote_write "default" {
  endpoint {
    url            = "..."
    send_exemplars = true  // Must be true
  }
}

Check data source configuration:

The Prometheus data source in Grafana must have the Tempo data source configured for exemplar linking:

1. In Grafana Cloud, go to Connections > Data sources.
2. Select Prometheus data source.
3. Scroll to Exemplars.
4. Verify Tempo data source is selected.

Can’t link traces to profiles

Symptoms

• No “Profiles” link appears on trace spans.
• Clicking profiles link shows “No data”.
• Profiles appear but for wrong time range.

Possible causes

• Span profiles not configured.
• Tempo data source not configured for traces-to-profiles.
• Label mappings incorrect.
• Profiles not available for the time range.

Solutions

Verify span profiles are enabled:

Check your application sends span profiles:

Go Copy

pyroscope.Start(pyroscope.Config{
    EnableSpanProfiling: true,  // Must be true
})

Check Tempo data source configuration:

1. In Grafana Cloud, go to Connections > Data sources.
2. Select Tempo data source.
3. Scroll to Traces to profiles.
4. Verify:
   • Enabled toggle is ON
   • Pyroscope data source is selected
   • Profile type is set (for example, process_cpu:cpu:nanoseconds:cpu:nanoseconds)
   • Label mappings are correct

Verify attribute-to-label mappings:

Trace attributes must map to profile labels:

Trace attributes (from Tempo):

Bash Copy

resource.service.name = "api"

Profile labels (in Pyroscope):

Bash Copy

service_name = "api"

Mapping configuration:

• Span attribute (from traces): service.name
• Label (in profiles): service_name

Test manually:

1. Note the service name and time range from a trace.

2. Go to Explore with Pyroscope data source.

3. Query profiles manually:

   Bash Copy

   {service_name="api"}

4. Verify profiles exist for that time range.

If no profiles exist, your application may not be sending them.

Labels don’t match between signals

Symptoms

• Can’t correlate across different signal types.
• Each signal has different label names or values.
• Inconsistent data when filtering.

Possible causes

• Different instrumentation libraries use different conventions.
• External labels not configured consistently.
• Resource attributes not mapped correctly.

Solutions

Standardize label and attribute names:

Create a mapping of identifiers across signals:

Configure external labels consistently:

In Alloy:

Alloy Copy

prometheus.remote_write "default" {
  external_labels = {
    service     = "api",
    environment = "prod",
  }
}

loki.write "default" {
  external_labels = {
    service     = "api",  // Same name and value
    environment = "prod",
  }
}

For OpenTelemetry:

Use resource attributes for traces:

YAML Copy

processors:
  resource:
    attributes:
      - key: service.name
        value: api
        action: upsert

Tempo automatically uses resource.service.name, and you can map it in the Tempo data source configuration.

Verify label and attribute consistency:

Query each data source with the same identifier:

promql Copy

up{service="api"}                           # Prometheus

logql Copy

{service="api"}                             # Loki

traceql Copy

{resource.service.name="api"}              # Tempo

Copy

{service_name="api"}                       # Pyroscope

All queries should return data for the same service.

Profiles not available

Symptoms

• Profiles don’t appear in Explore or Profiles Drilldown.
• “No data” when querying Pyroscope.
• Profiles link from traces shows empty results.

Possible causes

• Daily ingestion limit reached.
• Profiles not being sent from application.
• Incorrect service_name label.
• Time range mismatch.

Solutions

Check daily ingestion limit:

If you have configured a daily ingestion limit for profiles, data is discarded when the limit is reached. Monitor your usage:

1. In the Grafana main menu, click Dashboards. The Dashboards page lists all of the available dashboards in your Grafana instance.
2. Click the Usage Insights dashboard.
3. Check the grafanacloud_profiles_instance_period_ingested_megabytes metric.
4. Compare against grafanacloud_profiles_instance_ingest_limit_megabytes.

If limit is reached, data won’t be available until the limit resets at 00:00 UTC.

Verify profiles are being sent:

Check your application’s Pyroscope configuration:

Go Copy

pyroscope.Start(pyroscope.Config{
    ApplicationName: "my-service",
    ServerAddress:   "https://profiles-xxx.grafana.net",
    // Ensure correct authentication
    BasicAuthUser:     "<PYROSCOPE_USERNAME>",
    BasicAuthPassword: "<GRAFANA_CLOUD_API_KEY>",
})

Check service_name label:

Profiles use service_name (with underscore), not service.name (with dot):

Bash Copy

{service_name="my-service"}  // Correct for Pyroscope
{service.name="my-service"}  // Wrong, this is trace attribute syntax

Verify time range:

Profiles are continuous but may have gaps. Ensure you’re querying a time range when your application was actively running and sending data.

Ingestion limits exceeded

Symptoms

• Data gaps in metrics, logs, or traces.
• “Rate limit exceeded” errors in collector logs.
• Incomplete correlation due to missing data.

Possible causes

• Ingestion rate exceeds configured limits.
• Burst of traffic exceeded burst limits.
• High cardinality causing excessive series/streams.

Solutions

Check current limits and usage:

Query the Usage Insights data source:

promql Copy

# Metrics ingestion rate
rate(grafanacloud_instance_samples_per_second[5m])

# Logs ingestion rate
rate(grafanacloud_logs_instance_bytes_received_total[5m])

Compare against your limits (check usage limits documentation).

Reduce ingestion rate:

• For metrics: Use Adaptive Metrics to aggregate unused metrics.
• For logs: Filter noisy log lines at the collector level.
• For traces: Implement tail sampling to keep only important traces.

Monitor for dropped data:

Check collector logs for rate limit errors:

Bash Copy

# In Grafana Alloy logs
grep -i "rate limit" /var/log/alloy/*.log

Query results truncated

Symptoms

• “Results may be incomplete” warning.
• Fewer results than expected.
• Query times out.

Possible causes

• Query returns too many series or streams.
• Time range too long for data volume.
• High cardinality query.

Solutions

Add more specific label filters:

Instead of:

promql Copy

http_requests_total

Use:

promql Copy

http_requests_total{service="api", environment="prod"}

Reduce time range:

Start with a shorter time range and expand if needed:

• Start with 1 hour instead of 24 hours.
• Use relative time ranges (Last 15 minutes) for initial investigation.

Use aggregation functions:

For metrics, use topk() or aggregations:

promql Copy

topk(10, sum by (service) (rate(http_requests_total[5m])))

For logs, add filters before aggregating:

logql Copy

{service="api"} |= "error" | json | line_format "{{.message}}"

Check query limits:

Data older than retention unavailable

Symptoms

• Historical correlation fails.
• “No data” for older time ranges.
• Only recent data appears.

Possible causes

• Data deleted due to retention policy.
• Different retention periods across signals.

Solutions

Check retention periods:

Default retention varies by signal:

Plan for retention alignment:

If you need to correlate signals over longer periods:

• Extend logs retention (configurable from 30 days to 1 year via API).
• Export logs to long-term storage before retention expires.

Use metrics for historical context:

When logs or traces are no longer available:

• Use metrics (longer retention) to understand historical patterns.
• Correlate forward from metrics to recent logs/traces.

Verification checklist

Use this checklist to systematically verify correlation:

Shared labels and attributes

• Label and attribute names match across Prometheus, Loki, Tempo, and Pyroscope.
• Label and attribute values match exactly (case-sensitive).
• Query each data source confirms labels and attributes exist.
• Time ranges align when testing.

Trace IDs in logs

• Application code propagates trace context.
• Raw logs contain trace_id fields.
• Trace ID format is 32-character hex.
• All log entries from traced requests have trace IDs.

Derived fields (Loki to Tempo)

• Derived field configured in Loki data source.
• Regex pattern matches actual log format exactly.
• Tempo data source selected as target.
• Trace IDs are underlined/clickable in logs.
• Clicking trace ID opens correct trace.

Exemplars (Prometheus to Tempo)

• Application code emits exemplars with trace IDs.
• Metrics endpoint shows exemplars (OpenMetrics format).
• Alloy/collector has send_exemplars = true.
• Dashboard uses Time series panel type.
• Exemplars toggle enabled in panel.
• Prometheus data source links to Tempo.
• Exemplar points visible as diamonds on graph.

Traces to profiles (Tempo to Pyroscope)

• Application instrumented for both traces and profiles.
• Span profiles enabled if using span-level linking.
• Tempo data source configured for traces-to-profiles.
• Pyroscope data source selected.
• Label mappings configured correctly.
• Profiles link appears on trace spans.
• Clicking opens profile for correct time range.

Get help

If you’re still experiencing issues:

• Grafana Community Forums: https://community.grafana.com
• Grafana Cloud Support: Available in the Grafana Cloud portal
• Documentation:
  • Configure exemplars
  • Use Traces to profiles
  • Grafana Cloud usage limits
  • Cardinality management
  • Traces sampling
  • Profiles ingestion control