Help build the future of open source observability software Open positions

Check out the open source projects we support Downloads

Grot cannot remember your choice unless you click the consent notice at the bottom.

New in Grafana 8.5: How to jump from traces to Splunk logs

New in Grafana 8.5: How to jump from traces to Splunk logs

May 9, 2022 4 min

The recent release of Grafana 8.5 marks the start of enabling the jump from traces directly to Splunk logs. It’s a big leap that now allows you to draw a straight line between your traces — whether they are coming from Tempo, Zipkin, or Jaeger — to even more third-party logging data, all from the comfort of your traces view. Previously, the Grafana trace to logs enablement included only Loki logs. But with the introduction of this new feature, the deep insights offered by Splunk logs are now one click away.

Traces tell you that something happened, or more specifically, what part of your request path caused the error. But what if you were also empowered to connect the dots even further to understand why it happened as well? This feature helps you do just that, telling you why and when an incident happened. So, in combination, traces to logs is really powerful.

Grafana handles the task of sifting through your immense log data simply and powerfully by allowing you to filter logs by trace ID and/or by span ID, in addition to filtering by any other tags and metadata that may be contained in your logs. 

This was a heavily requested feature from our customers whose data is deeply embedded with Splunk; and now those users can add to their all-in-one approach to working in Grafana.

For more information about the Splunk Enterprise plugin, which is available for customers with a Grafana Cloud account or with a Grafana Enterprise license, check out the Splunk solutions page or contact our team.

How to jump from traces to Splunk logs

Much like the trace to logs feature for Loki, if trace to logs for Splunk is enabled in your data source (see Get started below), we will automatically generate a query link that, when clicked from the trace view, opens a Splunk tab in Grafana. 

This query is offset by 1,000 milliseconds (or 1 second), which is required by Splunk in Grafana to create a valid query. This allows us to return results faster, but it also means that you may need to adjust the time range in the user interface to a longer interval to see relevant results (more details in the instructions below). 

Just as with Loki logs, this traces to Splunk logs functionality is reflected in the URL, allowing the ability to save and share queries.

Get started

If you already have one of the tracing systems installed and your Splunk data source configured, all you have to do is the following: 

  1. Go to Configuration and choose your Trace data source.
Screenshot of configuration page for trace data source in Grafana
Screenshot of configuration page for trace data source in Grafana
  1. From your traces settings page, locate the Trace to logs section. From the Data source dropdown menu, select your configured Splunk data source. Add any tags contained in your data (e.g., jobas shown here), toggle the filter by trace ID and/or span ID feature if desired, and change any other settings you would like to here. Then click Save & test.
Screenshot of authorization configuration in Grafana
Screenshot of authorization configuration in Grafana
  1. Then, from Explore, select Service name, then run the query to see trace views.
Screenshot of selecting service name for trace to logs in Grafana Explore
Screenshot of selecting service name for trace to logs in Grafana Explore
Trace views in Grafana Explore after running service name query
Trace views in Grafana Explore after running service name query
  1. Click the Log link icon from your trace view.
Log link icon in Grafana Explore UI
Log link icon in Grafana Explore UI
  1. If you have any metadata that you’d like to search for that is contained in the _raw column of Splunk logs (e.g. color=white), you can query those key-value pairs directly in the query search section, or just search for specific strings in the data. Be sure to adjust the time range to get your desired data.
Query section in Grafana Explore for trace to logs
Query section in Grafana Explore for trace to logs
  1. Depending on the Select fields you set up in Splunk, they will appear as the Detected fields in the trace to logs view in Grafana; and you can query those directly by key and/or by value. The default fields host, source, and sourceType appear in every log event.
Detected fields in trace to logs view in Grafana
Detected fields in trace to logs view in Grafana

Happy trace to logging!

To learn more about the latest features in Grafana 8.5, you can also check out our What’s new in Grafana 8.5 documentation.

If you’re not already using Grafana Cloud — the easiest way to get started with observability — sign up now for a free 14-day trial of Grafana Cloud Pro, with unlimited metrics, logs, traces, and users, long-term retention, and access to one Enterprise plugin.