---
title: "Troubleshoot signal correlation | Grafana Cloud documentation"
description: "Solutions for common signal correlation problems."
---

> For a curated documentation index, see [llms.txt](/llms.txt). For the complete documentation index, see [llms-full.txt](/llms-full.txt).

# Troubleshoot signal correlation

This guide helps you diagnose and fix common signal correlation issues.

## Can’t navigate from metrics to logs

### Symptoms

- Clicking a metric doesn’t show related logs.
- Logs panel is empty when navigating from metrics.
- “No logs found” message appears.

### Possible causes

- Labels don’t match between data sources.
- Time ranges are misaligned.
- Label names are different (for example, `service` vs `service_name`).
- No logs exist for that time range or label combination.

### Solutions

**Verify label names match:**

1. Check Prometheus metric labels:
   
   promql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy
   
   ```promql
   up{service="api"}
   ```
2. Check Loki log labels:
   
   logql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy
   
   ```logql
   {service="api"}
   ```
3. Compare label names. They must be identical (case-sensitive).

**If label names differ:**

Use Alloy or collector configuration to normalize labels:

Alloy ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```alloy
// Prometheus remote write
prometheus.remote_write "default" {
  external_labels = {
    service = "api",
  }
}

// Loki write
loki.write "default" {
  external_labels = {
    service = "api",  // Use same name as Prometheus
  }
}
```

**Verify labels have the same values:**

promql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```promql
up{service="api"}  # Prometheus shows: service="api"
```

logql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```logql
{service="api"}    # Loki must also show: service="api"
```

If values differ (for example, `api` vs `api-service`), fix in your instrumentation or configuration.

**Check time ranges:**

Ensure both queries use the same time range. In Drilldown or Explore, the time range selector should show the same values for both panes.

## Trace IDs don’t appear in logs

### Symptoms

- Log entries don’t show trace IDs.
- No clickable links to traces in log view.
- Derived fields don’t work.

### Possible causes

- Trace context not propagated in application.
- Logging format doesn’t include trace context.
- OpenTelemetry SDK not configured correctly.

### Solutions

**Verify application propagates trace context:**

Check your application code includes trace context in logs. For OpenTelemetry-instrumented apps:

**Python:**

Python ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```python
from opentelemetry.instrumentation.logging import LoggingInstrumentor

# Must be called during app initialization
LoggingInstrumentor().instrument()
```

**Go:**

Go ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```go
import (
    "go.opentelemetry.io/contrib/instrumentation/log/slog"
)

// Use OpenTelemetry-aware logger
logger := slog.New(slog.NewJSONHandler(os.Stdout))
```

**Check log output format:**

View raw logs to confirm trace IDs are present:

Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```bash
kubectl logs <pod-name> | grep -i trace
```

Expected output should include `trace_id`:

Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```bash
{"level":"info","trace_id":"abc123...","msg":"request processed"}
```

**If trace IDs are missing:**

Add manual instrumentation to include trace context:

Python ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```python
import logging
from opentelemetry import trace

logger = logging.getLogger(__name__)

def handle_request():
    span = trace.get_current_span()
    trace_id = format(span.get_span_context().trace_id, "032x")

    logger.info("Processing request", extra={"trace_id": trace_id})
```

## Derived fields don’t work

### Symptoms

- Trace IDs appear in logs but aren’t clickable.
- Clicking trace ID shows “No trace found”.
- Derived field configured but not appearing.

### Possible causes

- Regex doesn’t match log format.
- Derived field not configured correctly.
- Tempo data source not selected.

### Solutions

**Test your regular expression pattern:**

1. Find a log entry with a trace ID.
2. Copy the full log line.
3. Test your regular expression against it.

Example log line:

Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```bash
{"timestamp":"2024-01-01T12:00:00Z","trace_id":"abc123def456","level":"info"}
```

Matching regular expression:

Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```bash
"trace_id":\s*"(\w+)"
```

**Verify derived field configuration:**

1. In Grafana Cloud, go to **Connections** &gt; **Data sources**.
2. Select your Loki data source.
3. Check **Derived fields** section:
   
   - **Name**: Must be unique (for example, `traceId`)
   - **Regex**: Must match your log format
   - **Internal link**: Must be toggled ON
   - **Data source**: Must select Tempo data source
   - **Query**: Should be `${__value.raw}`

**Test the configuration:**

Query logs that should have trace IDs:

logql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```logql
{service="api"} | json | trace_id != ""
```

Look for trace IDs that are underlined/clickable. If they’re not clickable, the regular expression or data source configuration is incorrect.

## Exemplars aren’t showing on graphs

### Symptoms

- No diamond points appear on Time series graphs.
- Can’t click from metrics to traces.
- Exemplars enabled but not visible.

### Possible causes

- Application not emitting exemplars.
- Panel type isn’t Time series.
- Exemplars not enabled in panel settings.
- `send_exemplars` not enabled in Alloy/Collector.

### Solutions

**Verify application emits exemplars:**

Check your metrics endpoint:

Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```bash
curl -H "Accept: application/openmetrics-text" http://your-app:9090/metrics | grep -i "traceid"
```

Expected output:

Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```bash
http_request_duration_seconds_bucket{le="0.1"} 45 # {trace_id="abc123"} 0.05
```

**If exemplars are missing:**

Update your instrumentation to include exemplars:

Go ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```go
histogram.ObserveWithExemplar(
    duration,
    prometheus.Labels{"trace_id": traceID},
)
```

**Check panel configuration:**

1. Edit the dashboard panel.
2. Verify **Panel type** is “Time series” (not “Graph”).
3. In the legend, toggle **Exemplars** ON.

**Verify Alloy/collector configuration:**

Alloy ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```alloy
prometheus.remote_write "default" {
  endpoint {
    url            = "..."
    send_exemplars = true  // Must be true
  }
}
```

**Check data source configuration:**

The Prometheus data source in Grafana must have the Tempo data source configured for exemplar linking:

1. In Grafana Cloud, go to **Connections** &gt; **Data sources**.
2. Select Prometheus data source.
3. Scroll to **Exemplars**.
4. Verify Tempo data source is selected.

## Can’t link traces to profiles

### Symptoms

- No “Profiles” link appears on trace spans.
- Clicking profiles link shows “No data”.
- Profiles appear but for wrong time range.

### Possible causes

- Span profiles not configured.
- Tempo data source not configured for traces-to-profiles.
- Label mappings incorrect.
- Profiles not available for the time range.

### Solutions

**Verify span profiles are enabled:**

Check your application sends span profiles:

Go ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```go
pyroscope.Start(pyroscope.Config{
    EnableSpanProfiling: true,  // Must be true
})
```

**Check Tempo data source configuration:**

1. In Grafana Cloud, go to **Connections** &gt; **Data sources**.
2. Select Tempo data source.
3. Scroll to **Traces to profiles**.
4. Verify:
   
   - Enabled toggle is ON
   - Pyroscope data source is selected
   - Profile type is set (for example, `process_cpu:cpu:nanoseconds:cpu:nanoseconds`)
   - Label mappings are correct

**Verify attribute-to-label mappings:**

Trace attributes must map to profile labels:

**Trace attributes** (from Tempo):

Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```bash
resource.service.name = "api"
```

**Profile labels** (in Pyroscope):

Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```bash
service_name = "api"
```

**Mapping configuration:**

- **Span attribute** (from traces): `service.name`
- **Label** (in profiles): `service_name`

**Test manually:**

1. Note the service name and time range from a trace.
2. Go to **Explore** with Pyroscope data source.
3. Query profiles manually:
   
   Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy
   
   ```bash
   {service_name="api"}
   ```
4. Verify profiles exist for that time range.

If no profiles exist, your application may not be sending them.

## Labels don’t match between signals

### Symptoms

- Can’t correlate across different signal types.
- Each signal has different label names or values.
- Inconsistent data when filtering.

### Possible causes

- Different instrumentation libraries use different conventions.
- External labels not configured consistently.
- Resource attributes not mapped correctly.

### Solutions

**Standardize label and attribute names:**

Create a mapping of identifiers across signals:

Expand table

| Signal     | Identifier for service              |
|------------|-------------------------------------|
| Prometheus | `service` (label)                   |
| Loki       | `service` (label)                   |
| Tempo      | `resource.service.name` (attribute) |
| Pyroscope  | `service_name` (label)              |

**Configure external labels consistently:**

In Alloy:

Alloy ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```alloy
prometheus.remote_write "default" {
  external_labels = {
    service     = "api",
    environment = "prod",
  }
}

loki.write "default" {
  external_labels = {
    service     = "api",  // Same name and value
    environment = "prod",
  }
}
```

**For OpenTelemetry:**

Use resource attributes for traces:

YAML ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```yaml
processors:
  resource:
    attributes:
      - key: service.name
        value: api
        action: upsert
```

Tempo automatically uses `resource.service.name`, and you can map it in the Tempo data source configuration.

**Verify label and attribute consistency:**

Query each data source with the same identifier:

promql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```promql
up{service="api"}                           # Prometheus
```

logql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```logql
{service="api"}                             # Loki
```

traceql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```traceql
{resource.service.name="api"}              # Tempo
```

![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```none
{service_name="api"}                       # Pyroscope
```

All queries should return data for the same service.

## Profiles not available

### Symptoms

- Profiles don’t appear in Explore or Profiles Drilldown.
- “No data” when querying Pyroscope.
- Profiles link from traces shows empty results.

### Possible causes

- Daily ingestion limit reached.
- Profiles not being sent from application.
- Incorrect `service_name` label.
- Time range mismatch.

### Solutions

**Check daily ingestion limit:**

If you have configured a daily ingestion limit for profiles, data is discarded when the limit is reached. Monitor your usage:

1. In the Grafana main menu, click **Dashboards**. The **Dashboards** page lists all of the available dashboards in your Grafana instance.
2. Click the **Usage Insights** dashboard.
3. Check the `grafanacloud_profiles_instance_period_ingested_megabytes` metric.
4. Compare against `grafanacloud_profiles_instance_ingest_limit_megabytes`.

If limit is reached, data won’t be available until the limit resets at 00:00 UTC.

**Verify profiles are being sent:**

Check your application’s Pyroscope configuration:

Go ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```go
pyroscope.Start(pyroscope.Config{
    ApplicationName: "my-service",
    ServerAddress:   "https://profiles-xxx.grafana.net",
    // Ensure correct authentication
    BasicAuthUser:     "<PYROSCOPE_USERNAME>",
    BasicAuthPassword: "<GRAFANA_CLOUD_API_KEY>",
})
```

**Check service\_name label:**

Profiles use `service_name` (with underscore), not `service.name` (with dot):

Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```bash
{service_name="my-service"}  // Correct for Pyroscope
{service.name="my-service"}  // Wrong, this is trace attribute syntax
```

**Verify time range:**

Profiles are continuous but may have gaps. Ensure you’re querying a time range when your application was actively running and sending data.

## Ingestion limits exceeded

### Symptoms

- Data gaps in metrics, logs, or traces.
- “Rate limit exceeded” errors in collector logs.
- Incomplete correlation due to missing data.

### Possible causes

- Ingestion rate exceeds configured limits.
- Burst of traffic exceeded burst limits.
- High cardinality causing excessive series/streams.

### Solutions

**Check current limits and usage:**

Query the Usage Insights data source:

promql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```promql
# Metrics ingestion rate
rate(grafanacloud_instance_samples_per_second[5m])

# Logs ingestion rate
rate(grafanacloud_logs_instance_bytes_received_total[5m])
```

Compare against your limits (check [usage limits documentation](/docs/grafana-cloud/cost-management-and-billing/manage-invoices/understand-your-invoice/usage-limits/)).

**Reduce ingestion rate:**

- For metrics: Use [Adaptive Metrics](/docs/grafana-cloud/adaptive-telemetry/adaptive-metrics/) to aggregate unused metrics.
- For logs: Filter noisy log lines at the collector level.
- For traces: Implement tail sampling to keep only important traces.

**Monitor for dropped data:**

Check collector logs for rate limit errors:

Bash ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```bash
# In Grafana Alloy logs
grep -i "rate limit" /var/log/alloy/*.log
```

## Query results truncated

### Symptoms

- “Results may be incomplete” warning.
- Fewer results than expected.
- Query times out.

### Possible causes

- Query returns too many series or streams.
- Time range too long for data volume.
- High cardinality query.

### Solutions

**Add more specific label filters:**

Instead of:

promql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```promql
http_requests_total
```

Use:

promql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```promql
http_requests_total{service="api", environment="prod"}
```

**Reduce time range:**

Start with a shorter time range and expand if needed:

- Start with 1 hour instead of 24 hours.
- Use relative time ranges (Last 15 minutes) for initial investigation.

**Use aggregation functions:**

For metrics, use `topk()` or aggregations:

promql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```promql
topk(10, sum by (service) (rate(http_requests_total[5m])))
```

For logs, add filters before aggregating:

logql ![Copy code to clipboard](/media/images/icons/icon-copy-small-2.svg) Copy

```logql
{service="api"} |= "error" | json | line_format "{{.message}}"
```

**Check query limits:**

Expand table

| Signal                   | Query limit          | Workaround                      |
|--------------------------|----------------------|---------------------------------|
| Metrics                  | 768 hours (32 days)  | Break into smaller time ranges  |
| Logs                     | 721 hours (~30 days) | Use more specific label filters |
| Traces (TraceQL metrics) | 24 hours             | Contact Support for increase    |

## Data older than retention unavailable

### Symptoms

- Historical correlation fails.
- “No data” for older time ranges.
- Only recent data appears.

### Possible causes

- Data deleted due to retention policy.
- Different retention periods across signals.

### Solutions

**Check retention periods:**

Default retention varies by signal:

Expand table

| Signal   | Default retention |
|----------|-------------------|
| Metrics  | 13 months         |
| Logs     | 30 days           |
| Traces   | 30 days           |
| Profiles | Based on billing  |

> Note
> 
> These are the defaults for Grafana Cloud [paid accounts](/pricing/). Your company may have negotiated and paid for custom retention.

**Plan for retention alignment:**

If you need to correlate signals over longer periods:

- Extend logs retention (configurable from 30 days to 1 year via API).
- Export logs to long-term storage before retention expires.

**Use metrics for historical context:**

When logs or traces are no longer available:

- Use metrics (longer retention) to understand historical patterns.
- Correlate forward from metrics to recent logs/traces.

## Verification checklist

Use this checklist to systematically verify correlation:

### Shared labels and attributes

- Label and attribute names match across Prometheus, Loki, Tempo, and Pyroscope.
- Label and attribute values match exactly (case-sensitive).
- Query each data source confirms labels and attributes exist.
- Time ranges align when testing.

### Trace IDs in logs

- Application code propagates trace context.
- Raw logs contain `trace_id` fields.
- Trace ID format is 32-character hex.
- All log entries from traced requests have trace IDs.

### Derived fields (Loki to Tempo)

- Derived field configured in Loki data source.
- Regex pattern matches actual log format exactly.
- Tempo data source selected as target.
- Trace IDs are underlined/clickable in logs.
- Clicking trace ID opens correct trace.

### Exemplars (Prometheus to Tempo)

- Application code emits exemplars with trace IDs.
- Metrics endpoint shows exemplars (OpenMetrics format).
- Alloy/collector has `send_exemplars = true`.
- Dashboard uses Time series panel type.
- Exemplars toggle enabled in panel.
- Prometheus data source links to Tempo.
- Exemplar points visible as diamonds on graph.

### Traces to profiles (Tempo to Pyroscope)

- Application instrumented for both traces and profiles.
- Span profiles enabled if using span-level linking.
- Tempo data source configured for traces-to-profiles.
- Pyroscope data source selected.
- Label mappings configured correctly.
- Profiles link appears on trace spans.
- Clicking opens profile for correct time range.

## Get help

If you’re still experiencing issues:

- **Grafana Community Forums**: [https://community.grafana.com](https://community.grafana.com)
- **Grafana Cloud Support**: Available in the Grafana Cloud portal
- **Documentation**:
  
  - [Configure exemplars](/docs/grafana-cloud/send-data/traces/configure/exemplars/)
  - [Use Traces to profiles](/docs/grafana-cloud/monitor-applications/profiles/traces-to-profiles/)
  - [Grafana Cloud usage limits](/docs/grafana-cloud/cost-management-and-billing/manage-invoices/understand-your-invoice/usage-limits/)
  - [Cardinality management](/docs/grafana-cloud/cost-management-and-billing/analyze-costs/metrics-costs/prometheus-metrics-costs/cardinality-management/)
  - [Traces sampling](/docs/grafana-cloud/send-data/traces/configure/sampling/)
  - [Profiles ingestion control](/docs/grafana-cloud/monitor-applications/profiles/ingestion-control/)