Configure the Tempo data source on Grafana Labs

Configure trace to logs correlation

Thu, 16 Apr 2026 20:21:47 +0000

Configure trace to logs correlation

Trace to logs correlation lets you navigate from a span in Tempo directly to the matching logs in Loki, and from a log line in Loki back to its trace in Tempo. When configured, a Logs for this span link appears in the trace view and a link to the configured tracing data source appears on log lines that contain a trace ID.

This is a two data source configuration:

Tempo data source: Controls how Grafana queries logs when you click a span, including the time window, query, and tags to match.
Loki data source: Controls how trace IDs are extracted from log lines and linked back to traces.

Both sides must be configured for the correlation to work in either direction.

Before you begin

To configure trace to logs correlation, you need:

A Tempo data source configured in Grafana
A Loki data source configured in Grafana
Applications that emit traces and logs with a shared identifier such as a trace ID, service name, or both
Editor or Admin permissions in Grafana

Note
You can’t modify a provisioned data source from the Grafana UI. If you’re using Grafana Cloud Traces (the pre-configured tracing data source in Grafana Cloud), its settings are read-only. To configure trace to logs, clone the data source to create an editable copy, or update the provisioning file for self-managed instances. Refer to Provision trace to logs settings for details.

Configure the Tempo data source

These settings control how Grafana queries your log data source when you click a span in the trace view. This guide uses Loki, but Trace to logs also supports Elasticsearch, Splunk, OpenSearch, Falcon LogScale, Google Cloud Logging, and VictoriaMetrics Logs.

Go to Connections > Data sources and select your Tempo data source.
Scroll to the Trace to logs section.
Select your Loki data source from the Data source drop-down list.
Configure the Span start time shift and Span end time shift fields.

These fields expand the log query time window around the span’s start and end timestamps. A span’s timestamps are precise to the millisecond, but log lines are often written slightly before or after the span boundary. The default value of 0 can return no logs if timestamps don’t align exactly. A common starting point is -2s for start shift and 2s for end shift.

You can increase this value for spans that represent slow or batch operations, for example 5s.

Configure Tags to map span attributes to Loki label names.

Grafana uses these tags to filter logs to the correct service, Pod, or cluster. The Loki label name you enter must exactly match the label name on your Loki streams. A mismatch silently breaks the correlation with no error shown.

If you don’t configure any tags, Grafana uses the following defaults: cluster, hostname, namespace, pod, service.name (remapped to service_name), and service.namespace (remapped to service_namespace). The automatic dot-to-underscore remapping only applies to these default tags. For custom tags, you must specify the remapped name explicitly.

To customize the mapping, use the Add tag button to add each row. The following table shows common tags and their typical Loki label equivalents:

Span attribute	Loki label
`service.name`	`service_name`
`service.namespace`	`service_namespace`
`cluster`	`cluster`
`namespace`	`namespace`
`pod`	`pod`
`hostname`	`hostname`

Choose labels with low cardinality. In Loki, every unique combination of label values creates a separate stream. Labels that take on many distinct values, such as pod, host, thread, duration, traceId, or spanId, can create hundreds of thousands or even millions of streams, causing slow queries, high memory usage, and log loss at ingest. The labels listed above (service_name, namespace, cluster) are good choices because they have a bounded, small set of values. If you need to query high-cardinality values like trace IDs or Pod names, store them as structured metadata instead. They are queryable without creating additional streams. For more information, refer to Cardinality.

Note
When using Loki, the Logs for this span link only appears on a span if at least one of the configured tags exists on that span. If none of the tags match, the link doesn’t appear and no error is shown.

Configure the Filter by trace ID and Filter by span ID toggles.
- Filter by trace ID: Filters Loki results to all logs for the entire trace, not just the selected span. This works well for most use cases.
- Filter by span ID: Filters to logs for a single span. This only works if your logs include a span ID field.
These toggles are disabled when Use custom query is enabled because the custom query controls filtering directly.

Clicking Logs for this span filters by trace ID, not span ID, so results include all logs for the entire trace that contain a matching trace ID, not only logs emitted during that specific span. If your results look broader than expected, this is expected behavior. To narrow results to a specific span’s activity, write a custom LogQL query that filters on span ID or span-specific attributes if your application logs them, for example {service_name="my-service"} | json | spanId="". For this feature to work at all, your application must inject trace IDs into log lines, for example, using OpenTelemetry SDK structured logging. Without trace IDs in logs, the query falls back to a time-range filter.
Optional: Enable Use custom query and write a LogQL expression to replace the auto-generated query.

Use ${__tags} to inject the mapped tag filters automatically:
logql
```
{${__tags}} | logfmt | trace_id=`${__trace.traceId}`
```
If a tag like pod is stored as Loki structured metadata instead of an indexed label, it can’t appear in the stream selector {}. Move it to a pipeline filter instead:
logql
```
{${__tags}} | pod=`${__span.tags["k8s.pod.name"]}` |= `${__trace.traceId}`
```
Refer to Custom query variables for the full list of available variables.
Click Save & test.

Configure the Loki data source

These settings enable the Tempo link on Loki log lines, letting you navigate from a log entry to its trace.

Note
Configuring the Tempo data source without also configuring the Loki data source is a common mistake. If you skip this step, clicking a log line doesn’t open a trace. Both sides are required.

Go to Connections > Data sources and select your Loki data source.
In the Additional settings section, scroll to Derived fields.
Click Add to add a new derived field.
Enter a Name for the field, for example TraceID.

Select the Type and enter a pattern to extract the trace ID from log lines.

If you select Regex in log line, enter a regular expression with one capture group. The following table shows common patterns:

Log format	Regex
`traceID=<VALUE>`	`traceID=(\w+)`
`trace_id=<VALUE>`	`trace_id=(\w+)`
JSON field `"traceId": "<VALUE>"`	`"traceId":"(\w+)"`

If you select Label, enter a regular expression that matches the label key, for example trace[_]?id to match both traceid and trace_id.

Tip
If your applications use different trace ID field names, you need a separate derived field entry for each format. To simplify this, standardize on OpenTelemetry structured metadata so trace IDs are stored as a consistent label.

Enable the Internal link toggle and select your Tempo data source from the drop-down list.
In the Query field, enter ${__value.raw}.

This passes the extracted trace ID to Tempo when the link is clicked. The field shows ${__value.raw} as placeholder text, but you must enter it as the actual value.
Click Save & test.

For more details about derived field options, refer to Derived fields in the Loki data source documentation.

Provision trace to logs settings

You can provision the trace to logs configuration using the tracesToLogsV2 block in your data source YAML file. For the full provisioning YAML example including all Tempo settings, refer to Provision the Tempo data source. For general provisioning instructions, refer to Provisioning data sources.

Example: NGINX service

This example assumes:

Applications running in Kubernetes that emit OpenTelemetry traces to Tempo
Logs sent to Loki with labels service_name, namespace, and pod
Trace IDs written to logs as trace_id=<VALUE>

The Tempo data source settings are:

Field	Value
Data source	Loki
Span start time shift	`-2s`
Span end time shift	`2s`
Tags	`service.name` mapped to `service_name`, `namespace`, `pod`
Filter by trace ID	Enabled
Filter by span ID	Disabled

The Loki data source derived field settings are:

Field	Value
Name	TraceID
Type	Regex in log line
Regex	`trace_id=(\w+)`
Internal link	Enabled, select Tempo
Query	`${__value.raw}`

Expected result

After configuring both data sources:

In Explore with the Tempo data source selected, click any span and then click Logs for this span. Loki opens filtered to service_name=nginx and namespace=production within the configured time window around the span.
In Explore with the Loki data source selected, any log line containing trace_id=... shows a link to your Tempo data source. Clicking it opens the full trace in Tempo.

Troubleshooting

If trace to logs links aren’t appearing, return no data, or only work for some services, refer to Trace to logs/metrics/profiles issues in the troubleshooting guide.

If the configuration fields are greyed out, your data source is provisioned. Refer to Provision trace to logs settings for options.

Next steps

Configure trace to metrics correlation: Link spans to metrics queries in Prometheus.
Configure trace to profiles correlation: Link spans to profiling data in Grafana Pyroscope.
Configure the Tempo data source: Return to connection, authentication, and streaming settings.

Configure trace to metrics correlation

Tue, 14 Apr 2026 14:59:56 +0000

Configure trace to metrics correlation

Trace to metrics lets you navigate from a span in a trace to metrics in a Prometheus or other metrics data source. When configured, a Metrics for this span link appears in the trace view.

Trace to metrics works with any Prometheus-compatible data source and doesn’t require the Tempo metrics generator. The metrics generator creates new metrics from trace data; trace to metrics links to metrics that already exist in your metrics data source.

Before you begin

To configure trace to metrics, you need:

A Tempo data source configured in Grafana
A Prometheus-compatible metrics data source configured in Grafana
Metrics that correspond to your traced services (for example, requests_total, request_duration_seconds)
Editor or Admin permissions in Grafana

Note
You can’t modify a provisioned data source from the Grafana UI. If you’re using Grafana Cloud Traces (the pre-configured tracing data source in Grafana Cloud), its settings are read-only. To configure trace to metrics, clone the data source to create an editable copy, or update the provisioning file for self-managed instances. Refer to the Provisioning section for details.

There are two ways to configure the trace to metrics feature:

Use a basic configuration with a default query.
Configure one or more custom queries where you can use a template language to interpolate variables from the trace or span.

Set up a basic configuration

To use a basic configuration, follow these steps:

Select a metrics data source from the Data source drop-down.
Optional: Change Span start time shift and Span end time shift. The placeholders show -2m (start) and 2m (end), which are applied if you leave the fields empty.
Optional: Choose any tags to use in the query. Click Add tag to add a tag mapping.

The tags you configure must be present in the spans attributes or resources for a trace to metrics span link to appear. You can optionally configure a new name for the tag. This is useful if the tag has dots in the name and the target data source doesn’t allow using dots in labels. For example, you can remap service.name to service_name.
Don’t select Add query.
Select Save and Test.

Set up custom queries

To use custom queries, you need to configure the tags you’d like to include in the linked queries. For each tag, the key is the span attribute name. In cases where the attribute name would result in an invalid metrics query or doesn’t exactly match the desired label name, you can enter the label name as the second value. For example, you could map the attribute k8s.pod to the label pod.

You can interpolate the configured tags using the $__tags keyword. For example, when you configure the query requests_total{$__tags} with the tags k8s.pod=pod and cluster, it results in requests_total{pod="nginx-554b9", cluster="us-east-1"}. The label values are dynamically inserted based on the span attributes’ values.

You can link to any metric you’d like, but metrics for span durations, counts, and errors filtered by service or span are a great starting point.

To use custom queries with the configuration, follow these steps:

Select a metrics data source from the Data source drop-down.
Optional: Choose any tags to use in the query. Click Add tag to add a tag mapping.

These tags can be used in the custom query with the ${__tags} variable. This variable interpolates the mapped tags as a list in an appropriate syntax for the data source and only includes the tags that were present in the span, omitting those that weren’t present. You can optionally configure a new name for the tag. This is useful in cases where the tag has dots in the name and the target data source doesn’t allow using dots in labels. For example, you can remap service.name to service_name. If you don’t map any tags here, you can still use any tag in the query like this: method="${__span.tags.method}". Refer to Custom query variables for the full list of available variables.
Click Add query to add a custom query.
Specify a custom query to be used to query metrics data.

Each linked query consists of:
- Link Label: (Optional) Descriptive label for the linked query.
- Query: The query ran when navigating from a trace to the metrics data source. Interpolate tags using the $__tags keyword. For example, when you configure the query requests_total{$__tags} with the tags k8s.pod=pod and cluster, the result looks like requests_total{pod="nginx-554b9", cluster="us-east-1"}.
Select Save and Test.

Configuration options

The following table describes options for configuring the Trace to metrics settings:

Setting name	Description
Data source	Defines the target data source.
Span start time shift	Shifts the start time for the metrics query, based on the span’s start time. You can use time units, such as `5s`, `1m`, `3h`. To extend the time to the past, use a negative value. Default: `-2m`.
Span end time shift	Shifts the end time for the metrics query, based on the span’s end time. You can use time units. Default: `2m`.
Tags	Defines the tags used in linked queries. The key sets the span attribute name, and the optional value sets the corresponding metric label name. For example, you can map `k8s.pod` to `pod`. To interpolate these tags into queries, use the `$__tags` keyword.
Link Label	(Optional) Descriptive label for the linked query.
Query	Input to write a custom query. Use variable interpolation to customize it with variables from span.

Provisioning

You can provision the trace to metrics configuration using the tracesToMetrics block in your data source YAML file. For the full provisioning YAML example including all Tempo settings, refer to Provision the Tempo data source.

Link from metrics to traces

To navigate in the reverse direction, from a metric to its associated trace, configure exemplars in your Prometheus data source. Refer to Exemplars for setup instructions.

Troubleshooting

If trace to metrics links aren’t appearing or return no data, refer to Trace to logs/metrics/profiles issues in the troubleshooting guide.

If the configuration fields are greyed out, your data source is provisioned. Refer to the Provisioning section for how to update the configuration via YAML.

Next steps

Configure trace to logs correlation: Navigate from spans to related logs in Loki.
Configure trace to profiles correlation: Link spans to profiling data in Grafana Pyroscope.
Provision the Tempo data source: Configure the Tempo data source using a YAML file.

Configure trace to profiles correlation

Tue, 14 Apr 2026 14:59:56 +0000

Configure trace to profiles correlation

Trace to profiles lets you navigate from a span in a trace directly to profiling data in Grafana Pyroscope. When configured, a Profiles for this span button and an embedded flame graph appear in the span details view.

Before you begin

To configure trace to profiles, you need:

A Tempo data source configured in Grafana
A Grafana Pyroscope data source configured in Grafana
An application instrumented with the OpenTelemetry span profiling bridge so that spans include the pyroscope.profile.id attribute
Editor or Admin permissions in Grafana

Note
You can’t modify a provisioned data source from the Grafana UI. If you’re using Grafana Cloud Traces (the pre-configured tracing data source in Grafana Cloud), its settings are read-only. To configure trace to profiles, clone the data source to create an editable copy, or update the provisioning file for self-managed instances. Refer to the Provisioning section for details.

Using Trace to profiles, you can use the ability to correlate different signals in Grafana by adding the functionality to link between traces and profiles.

Trace to profiles lets you link your Grafana Pyroscope data source to tracing data. When configured, this connection lets you run queries from a trace span into the profile data using Explore. Each span links to your queries. Clicking a link runs the query in a split panel. If tags are configured, Grafana dynamically inserts the span attribute values into the query. The query runs over the time range of the (span start time - 60) to (span end time + 60 seconds).

Embedded flame graphs are also inserted into each span details section that has a linked profile. This lets you see resource consumption in a flame graph visualization for each span without having to navigate away from the current view.

There are two ways to configure the trace to profiles feature:

Use a basic configuration with default query, or
Configure a custom query where you can use a template language to interpolate variables from the trace or span.

Before you begin

To use Trace to profiles, you need:

A Grafana Pyroscope data source connected to your Pyroscope instance.
A Tempo data source configured with the Trace to profiles settings described on this page.
Your application instrumented with all three of the following:
1. Profiling: A Pyroscope SDK or Grafana Alloy sending profiling data to Pyroscope. Refer to Configure the client to send profiles for setup instructions.
2. Tracing: An OpenTelemetry SDK sending trace data to Tempo. Refer to the OpenTelemetry getting started documentation for setup instructions.
3. Span profiling bridge: A language-specific OpenTelemetry integration package that links profiling data with trace spans. Without this package, traces and profiles are independent signals with no connection between them. Refer to Link tracing and profiling with span profiles for per-language setup instructions.

Use a basic configuration

To use a basic configuration, follow these steps:

In the left menu, select Connections > Data sources.
Select your configured Tempo data source from the Data source list.
Scroll down to the Traces to profiles section.
Select a Pyroscope data source in the Data source drop-down.
Optional: Add one or more tags to use in the query. If left blank, the default values of service.name and service.namespace are used.

The tags you configure must be present in the spans attributes or resources for a trace-to-profiles span link to appear.

You can optionally configure a new name for the tag. This is useful if the tag has dots in the name and the target data source doesn’t allow dots in labels. In that case, you can remap service.name to service_name.
Select one or more profile types to use in the query. Select the drop-down and choose options from the menu.

The profile type or app must be selected for the query to be valid. Grafana doesn’t show any data if the profile type or app isn’t selected when a query runs.
Select Save and Test.

If you have configured a Pyroscope data source and no profile data is available or the Profiles for this span button and the embedded flame graph isn’t visible, verify that the pyroscope.profile.id key-value pair exists in your span tags.

Configure a custom query

To use a custom query with the configuration, follow these steps:

In the left menu, select Connections > Data sources.
Select a configured Tempo data source from the Data source list.
Scroll down to the Traces to profiles section.
Select a Pyroscope data source in the Data source drop-down.
Optional: Choose any tags to use in the query. If left blank, the default values of service.name and service.namespace are used.

These tags can be used in the custom query with ${__tags} variable. This variable interpolates the mapped tags as list in an appropriate syntax for the data source. Only tags present in the span are included. Tags that aren’t present are omitted.

You can also configure a name for the tag. Tag names are useful where the tag has dots in the name and the target data source doesn’t allow using dots in labels. For example, you can remap service.name to service_name. If you don’t map any tags here, you can still use any tag in the query, for example: method="${__span.tags.method}". Learn more about custom query variables.
Select one or more profile types to use in the query. Select the drop-down and choose options from the menu.
Switch on Use custom query to enter a custom query.
Specify a custom query to be used to query profile data. You can use various variables to make that query relevant for current span. The link shows only if all the variables are interpolated with non-empty values to prevent creating an invalid query. You can interpolate the configured tags using the $__tags keyword.
Select Save and Test.

Configuration options

The following table describes options for configuring your Trace to profiles settings:

Setting name	Description
Data source	Defines the target data source. You can select a Pyroscope [profiling] data source.
Tags	Defines the tags to use in the profile query. Default: `cluster`, `hostname`, `namespace`, `pod`, `service.name`, `service.namespace`. You can change the tag name for example to remove dots from the name if they’re not allowed in the target data source. For example, map `http.status` to `http_status`.
Profile type	Defines the profile type to use in the query.
Use custom query	Toggles use of custom query with interpolation.
Query	Input to write custom query. Use variable interpolation to customize it with variables from span.

Provisioning

You can provision the trace to profiles configuration using the tracesToProfiles block in your data source YAML file. For the full provisioning YAML example including all Tempo settings, refer to Provision the Tempo data source.

Verify the integration

After configuring trace to profiles, verify the integration:

Open Explore and select your Tempo data source.
Run a query to load a trace.
Expand a span and confirm that the Profiles for this span button appears in the span details.
Click the button and verify that an embedded flame graph loads with profiling data.

Troubleshooting

If trace to profiles links aren’t appearing or the flame graph is empty, refer to Trace to logs/metrics/profiles issues in the troubleshooting guide. For additional diagnosis steps, refer to Troubleshoot trace to profiles in the Pyroscope documentation.

If the configuration fields are greyed out, your data source is provisioned. Refer to the Provisioning section for how to update the configuration via YAML.

Next steps

Configure trace to logs correlation: Navigate from spans to related logs in Loki.
Configure trace to metrics correlation: Link spans to metrics queries in Prometheus.
Provision the Tempo data source: Configure the Tempo data source using a YAML file.

Additional settings

Tue, 14 Apr 2026 14:59:56 +0000

Additional settings

The Additional settings section of the Tempo data source settings contains options that most users don’t need to change. These settings are collapsed by default.

Use the down arrow to expand the Additional settings section to view these options.

This page covers the following settings:

Advanced HTTP settings: Configure cookie forwarding and request timeout
Service graph: Link a Prometheus data source for Service Graph visualization
Node graph: Enable the node graph above the trace view
Tempo search: Show or hide the search option in Explore
TraceID query: Narrow TraceID lookups to a time range for performance
Tags time range: Control the time window for tag and tag-value queries
Tag limit: Set the maximum number of tags retrieved from Tempo
Span bar: Display duration or a custom tag in the span bar
Private data source connect: Connect to data sources in private networks

Advanced HTTP settings

The Grafana Proxy deletes forwarded cookies. Use the Allowed cookies field to specify cookies by name that should be forwarded to the data source.

The Timeout field sets the HTTP request timeout in seconds.

Service graph

The Service graph setting configures the Service Graph data.

You must link a Prometheus data source in the Data source setting for the Service Graph to display data. Without this, Service Graph queries return empty results.

The linked Prometheus data source must contain metrics generated by Grafana Alloy or the Tempo metrics generator. For the full list of prerequisites and troubleshooting steps, refer to Service Graph and Service Graph view.

Note
If you cloned a provisioned data source (for example, the Grafana Cloud Traces data source) and the Service graph section doesn’t appear in the cloned data source’s settings, verify that the cloned data source has the same Tempo backend URL and authentication configured. The Service graph section only appears when the data source can connect to Tempo successfully. Refer to Clone a provisioned data source for Grafana Cloud for setup steps.

Node graph

The Node graph setting enables the node graph visualization, which isn’t activated by default.

After it’s activated, Grafana displays the node graph above the trace view.

Tempo search

The Search setting configures Tempo search.

You can configure the Hide search setting to hide the search query option in Explore if search is not configured in the Tempo instance.

TraceID query

The TraceID query setting modifies how TraceID queries are run. The time range can be used when there are performance issues or timeouts since it narrows down the search to the defined range. This setting is disabled by default.

You can configure this setting as follows:

Name	Description
Use time range in query	Use a time range in the TraceID query. Default: `disabled`.
Time shift for start of search	Time shift for start of search. Default: `30m`.
Time shift for end of search	Time shift for end of search. Default: `30m`.

Tags time range

The Tags time range feature controls how tag and tag-value queries are executed by specifying the time window applied to these requests. You can select one of the following options to constrain your queries:

Name	Description
Last 30 minutes	Last 30 minutes of selected range.
Last 3 hours	Last 3 hours of selected range.
Last 24 hours	Last 24 hours of selected range.
Last 3 days	Last 3 days of selected range.
Last 7 days	Last 7 days of selected range.

Tag limit

The Tag limit setting modifies the max number of tags and tag values to retrieve from Tempo. Default: 5000

Span bar

The Span bar setting helps you display additional information in the span bar row.

You can choose one of three options:

Name	Description
None	Adds nothing to the span bar row.
Duration	(Default) Displays the span duration on the span bar row.
Tag	Displays the span tag on the span bar row. You must also specify which tag key to use to get the tag value, such as `component`.

Private data source connect

Note
This feature is only available in Grafana Cloud.

Use private data source connect (PDC) to connect to and query data within a secure network without opening that network to inbound traffic from Grafana Cloud.

Refer to Private data source connect for more information on how PDC works and Configure Grafana private data source connect (PDC) for steps on setting up a PDC connection.

Use the drop-down list to select a configured private data source. If you make changes, select Test & save to preserve your changes.

Use Manage private data source connect to configure and manage any private data sources you have configured.

Next steps

Configure the Tempo data source: Return to connection, authentication, and streaming settings.
Provision the Tempo data source: Configure the Tempo data source using a YAML file.
Service Graph and Service Graph view: Visualize service dependencies and monitor RED metrics.

Provision the Tempo data source

Tue, 14 Apr 2026 14:59:56 +0000

Provision the Tempo data source

You can define and configure the Tempo data source in YAML files as part of the Grafana provisioning system.

You can use version control, like Git, to track and manage file changes. Changes can be updated or rolled back as needed.

Note
Provisioning via YAML is primarily used for self-managed Grafana instances.

For more information about provisioning and available configuration options, refer to Provisioning Grafana.

Provisioned data sources

You can’t modify a provisioned data source from the Grafana UI. The settings form is read-only and the Save & test button is replaced by Test (you can test the connection but not save changes).

To make changes, you can either:

Clone the data source: Create a new data source of the same type and copy the settings from the provisioned data source. Refer to Clone a provisioned data source for Grafana Cloud for detailed steps.
Update the provisioning file: Edit the YAML configuration file and restart Grafana, or wait for the provisioning system to pick up the change. Any setting, including span time shifts on trace to logs, can be changed by editing the YAML.

Example file

This example provisioning YAML file sets up the equivalents of the options available in the Tempo data source UI. Replace grafana-pyroscope-datasource with the actual UID of your Pyroscope datasource, and verify the other datasource Uid values match what’s actually provisioned.

apiVersion: 1

datasources:
  - name: Tempo
    type: tempo
    uid: EbPG8fYoz
    url: http://localhost:3200
    access: proxy
    basicAuth: false
    jsonData:
      tracesToLogsV2:
        # Field with an internal link pointing to a logs data source in Grafana.
        # datasourceUid value must match the uid value of the logs data source.
        datasourceUid: 'loki'
        spanStartTimeShift: '-1h'
        spanEndTimeShift: '1h'
        tags: [{ key: 'job' }, { key: 'instance' }, { key: 'pod' }, { key: 'namespace' }]
        filterByTraceID: false
        filterBySpanID: false
        customQuery: true
        query: 'method="$${__span.tags.method}"'
      tracesToMetrics:
        datasourceUid: 'prom'
        spanStartTimeShift: '-1h'
        spanEndTimeShift: '1h'
        tags: [{ key: 'service.name', value: 'service' }, { key: 'job' }]
        queries:
          - name: 'Sample query'
            query: 'sum(rate(traces_spanmetrics_latency_bucket{$$__tags}[5m]))'
      tracesToProfiles:
        datasourceUid: 'grafana-pyroscope-datasource'
        tags: [{ key: 'job' }, { key: 'instance' }, { key: 'pod' }, { key: 'namespace' }]
        profileTypeId: 'process_cpu:cpu:nanoseconds:cpu:nanoseconds'
        customQuery: true
        query: 'method="$${__span.tags.method}"'
      serviceMap:
        datasourceUid: 'prometheus'
      nodeGraph:
        enabled: true
      search:
        hide: false
      traceQuery:
        timeShiftEnabled: true
        spanStartTimeShift: '-1h'
        spanEndTimeShift: '1h'
      spanBar:
        type: 'Tag'
        tag: 'http.path'
      streamingEnabled:
        search: true
        metrics: true

For details about individual settings, refer to:

Configure trace to logs correlation: tracesToLogsV2 block
Configure trace to metrics correlation: tracesToMetrics block
Configure trace to profiles correlation: tracesToProfiles block
Additional settings: serviceMap, nodeGraph, search, traceQuery, spanBar blocks

Clone a provisioned data source for Grafana Cloud

If you have a data source that is provisioned by a configuration file in Grafana Cloud, you can clone that provisioned data source and then edit the new data source in the Grafana UI.

For example, if you want to edit the trace to logs settings in your Tempo data source that is provisioned on Grafana Cloud, you can enable traceID and spanID filtering by cloning the data source.

To clone a provisioned data source, follow these steps:

Create a viewer Cloud Access Policy token in the Grafana Cloud Portal, making sure it has read permissions at least for the data types you are trying to clone.
Create a new data source of the same type you want to clone.
Copy all of the settings from the existing provisioned data source into the new data source while replacing the password with the API key you created.

The easiest way to do this is to open separate browser windows with the provisioned data source in one and the newly created data source in another.

After copying the HTTP and Auth section details, pasting the Cloud Access Policy token into the Password field, and changing any of the other options that you want, you can save and test the data source.

Next steps

Configure the Tempo data source: Return to connection, authentication, and streaming settings.
Configure trace to logs correlation: Set up span-to-log navigation.
Additional settings: Configure Service graph, node graph, search, and other settings.

Trace correlations

Thu, 16 Apr 2026 20:21:47 +0000

Trace correlations

You can use Grafana correlations to embed interactive correlation links in your trace view to jump from spans to related logs, metrics, profiles, or external systems. This guide explains how to configure and manage Trace correlations in Grafana.

What are trace correlations?

Trace correlations let you define rules that inject context-sensitive links into your trace spans. When viewing traces in Explore or the Traces panel, users can click these links to navigate directly to relevant queries or URLs. Correlations are similar but more flexible to the trace to logs, metrics, and profiles links you can configure for the Tempo data source.

Before you begin

To use trace correlations, you need:

Grafana 12 or later
A Tempo data source configured in Grafana
Admin access to configuration settings or provisioning files in Grafana

Set up a trace correlation

Log in to Grafana with an administrator account.
Go to Configuration > Plugins & data > Correlations.
Select Add correlation or Add new.
On step 1, provide a label for the correlation, and an optional description.

On step 2, configure the correlation target.

Select the Type drop-down list and choose Query to link to another data source or choose External for a custom URL.
For a query Target, select the target drop-down list and select the data source that should be queried when the link is clicked. Define the target query.
For an external Target, enter the External URL.
For both query and external targets, you can use the following variables based on trace data. Object variables must be parsed into a value variable with a regular expression transformation.

Variable	Type	Description
`traceId`	String	Trace identifier
`spanID`	String	Span identifier
`parentSpanID`	String	Parent span identifier
`serviceName`	String	Service name
`serviceTags`	Object	Resource attributes
`tags`	Object	Span attributes
`logs`	Object	Trace events
`references`	Object	Trace links

On step 3, configure the correlation data source:
- Select your Tempo data source in the Source drop-down list.
- Enter the trace data variable you use for the correlation in the Results field.
- Optionally, add one or more Transformations to parse the trace data into additional variables. You can use these variables to configure the correlation Target.
Select Save to save the correlation.

Verifying correlations in Explore

Open Explore and select your Tempo tracing source.
Run a query to load spans.
Hover over the span links menu or open the span details to reveal the correlation link buttons.
Click a correlation link to open a split view or navigate to your target system or query.

Examples

Below are several practical correlation configurations to get you started.

Example 1: Trace to logs by service name and trace identifier

In this example, you configure trace to logs by service name and a trace identifier.

On step 1, add a new correlation with the label Logs for this service and trace and an optional description.
On step 2, configure the correlation target:
- Select the target type Query and select your Loki data source as Target.
- Define the Loki query, using serviceName and traceID as variables derived from the span data:
```
{service_name="$serviceName"} | trace_id=`$traceID` |= ``
```
  In this query, service_name is the only stream label used inside {}. Stream labels should be low-cardinality values that describe the source of your logs. The trace_id field appears after the | pipe as a pipeline filter, which searches log content or structured metadata without creating additional streams. Don’t use trace IDs or other high-cardinality values as stream labels because each unique value creates a separate stream, which degrades Loki performance. For more information, refer to Label best practices and Cardinality.
On step 3, configure the correlation source:
- Select your Tempo data source as Source.
- Use traceID as Results field.
- Add a new transformation to extract the serviceName from the span serviceTags using the regular expression:
```
{(?=[^\}]*\bkey":"service.name")[^\}]*\bvalue":"(.*?)".*}
```
Save the correlation.

Example 2: Trace to custom URL

In this example, you configure trace correlations with a custom URL.

On step 1, add a new correlation with the label Open custom URL and an optional description.
On step 2, configure the correlation target:
- Select the target type External.
- Define your target URL, using variables derived from the span data. In this example, we are using serviceName and traceID.
```
https://my-server.example.com/service=$serviceName&trace=$traceID
```
On step 3, configure the correlation source:
- Select your Tempo data source as Source.
- Use traceID as Results field.
- Add a new transformation to extract the serviceName from the span serviceTags using the regular expression:
```
{(?=[^\}]*\bkey":"service.name")[^\}]*\bvalue":"(.*?)".*}
```
Save the correlation.

Best practices

Name clearly: Use descriptive names indicating source and target. For example: Trace to errors in logs.
Use low-cardinality stream labels: When targeting Loki, use only low-cardinality values like service_name, namespace, or cluster inside the stream selector {}. Place high-cardinality values like trace IDs in pipeline filters (after |) or store them as structured metadata. Using trace IDs as stream labels creates excessive streams and degrades Loki performance. For more information, refer to Cardinality and Label best practices. For detailed guidance on mapping span attributes to Loki labels, refer to Configure trace to logs correlation.
Template wisely: Use multiple $variable tokens if you need to inject more than one field.

Next steps

Configure trace to logs correlation: Link spans to log queries in Loki with tag mapping and cardinality guidance.
Configure trace to metrics correlation: Link spans to metrics queries in Prometheus.
Configure trace to profiles correlation: Link spans to profiling data in Grafana Pyroscope.
Configure the Tempo data source: Return to connection, authentication, and streaming settings.