Splunk data source for Grafana

The Splunk data source allows you to query and visualize Splunk data with Search Processing Language (SPL) or a visual SPL editor.

Requirements

The Splunk data source has the following requirements:

Known limitations

There are no known limitations.

Install the data source

To install the data source, refer to Installation.

Configure the data source

To add a data source, refer to Add a data source, and fill in the following fields:

Basic fields

Field Description
Name A name for this particular Splunk data source
URL Your access method is Server. The URL needs to be accessible from the Grafana backend server.
Whitelisted Cookies Grafana proxy deletes forwarded cookies by default. Specify the cookies by name that you wan to forward to the data source.

Authentication fields

Most data sources use either Basic Auth or an API key.

Basic auth
Enter a Splunk user name and password. Avoid using the default administration account. TLS Client Auth
The built-in option for authenticating using TLS (Transport Layer Security) Skip TLS Verify
Enable to skip verifying TLS With Credentials
Enable to send credentials, such as cookies or authentication headers with cross-site requests. With CA Cert
Enable to verify self-signed TLS. Forward OAuth Identity
Forward the identity of the Oauth user who is signed in to Grafana, in cases where the same Oauth provider is used for both Grafana and the data source

You can configure custom HTTP header data sources that are managed by Grafana’s provisioning. Doing so allows you to add HTTP headers to all requests that go to that data source. Configure the header name in the jsonData field, and the header value in the secureJsonData field.

Splunk details

Clustering strategy
session key use if your Splunk does not use a load balancer. Splunk does not share session keys between nodes.
basic auth use if your Splunk server uses a load balancer.
Stream mode
(EXPERIMENTAL) Enable to get search results as they become available
Auto cancel
Number of seconds that a job can be inactive before it is automatically canceled. The default is 30 seconds. To never auto cancel, set to 0.
Poll result
Run a search, and then periodically check for a result. Under the hood, this option runs a search/jobs API call with exec_mode set to normal. In such a case, the API request returns the job’s SID (security identifier), and then Grafana periodically checks the job status, in order to get job result.
This option might be helpful for slow queries. This option is disabled by default, and Grafana sets exec_mode to oneshot, which allows you to return the search result in the same API call. For more information about the search/jobs API endpoint, refer to the search/jobs/{search_id}/results in the Splunk documentation.
Internal fields filtration
Enable to hide fields with names that start with _.
Internal field pattern
Regex pattern for removing internal fields from the result.
Time stamp field
Default time stamp field. For more information, refer to Time stamp and time ranges.
Fields search mode
When you use the visual query editor, the data source tries to get a list of available fields for the selected source type.
quick uses the first available result from preview
full waits for the job to finish to get full result
Default earliest time
Time offset format: [+ | -]<integer><time_unit> For example, -1w. Time unit: s, m, h, d, w, mon, q, y
By using a default earliest time, you can prevent a search from spanning all of time, which might slow down Splunk. Some searches, such as template variable queries, cannot use a dashboard time range.
Variables search mode
Search mode for template variable queries.
fast: turn field discovery off for event searches. No event or field data for stats searches.
smart turn field discovery on for event searches. No event or field data for stats searches.
verbose All event and field data.

Data links are typically used in Grafana’s explore mode. The data displays a link that allows you to associate data internally with other Grafana data sources, or externally with data via a URL.

Set up a data link by clicking the + Add button under the Data links header in data source settings.

Field
The field from which to parse data. Logging data in Splunk is often returned in the _raw column.
Label
Given a key and value such as color=white, the label is the key in this key/value pair.
Regex
Given a key and value such as color=white, the regex parses out the value in the key value pair from the field. The regex is a matching regex, so you must provide one set of brackets for the matching expression. To match everything, use /(.*)/. You must wrap the regex in //.

Turn off Internal link to link to a URL that is based on the value parsed out of the Splunk logs. URL: Use the variable ${__value.raw} to hold the value of the data parsed out with the regex. You can use this to construct a URL.

Turn on Internal link to use an external link and link one data source to another data source within Grafana.

Configure the data source with provisioning

It is possible to configure data sources using config files with Grafana’s provisioning system. You can read more about how it works and all the settings you can set for data sources on the provisioning docs page

apiVersion: 1

datasources:
  - name: Splunk
    type: grafana-splunk-datasource
    access: proxy
    basicAuth: true
    basicAuthUser: user
    editable: true
    enabled: true
    jsonData:
      advancedOptions: true
      fieldSearchType: quick
      internalFieldsFiltration: true
      tlsSkipVerify: true
      variableSearchLevel: fast
      streamMode: false
      clusteringStrategy: 1
    secureJsonData:
      basicAuthPassword: password
    url: splunk url
    version: 1

Query the data source

Query editor support two modes: spl and visual. To switch between these modes click hamburger icon at the right side of editor and select Toggle Editor Mode.

SPL mode

Use SPL mode by querying with Search Processing Language (SPL). Find more information on SPL here.

For time series data use timechart command. For example:

index=os sourcetype=cpu | timechart span=1m avg(pctSystem) as system, avg(pctUser) as user, avg(pctIowait) as iowait
index=os sourcetype=ps | timechart span=1m limit=5 useother=false avg(cpu_load_percent) by process_name

Grafana is a time-series-oriented application, and your search query returns time-series data (in the form of a timestamp and value, or a single value). Find more information about the timechart command, refer to timechart. For more search query examples, refer to Splunk® Enterprise Search Reference.

Splunk metrics and mstats

Splunk 7.x provides an mstats command for analyzing metrics, which you need to combine with the timeseries command for it to work properly. You must also set the prestats=t option.

Current syntax:
| mstats prestats=t avg(disk.disk_ops.read) avg(disk.disk_ops.write) WHERE index="collectd" by metric_name span=1m
| timechart avg(disk.disk_ops.read) avg(disk.disk_ops.write) span=1m

Deprecated syntax:
| mstats prestats=t avg(_value) AS Value WHERE index="collectd" metric_name="disk.disk_ops.read" OR metric_name="disk.disk_ops.write" by metric_name span=1m
| timechart avg(_value) span=1m by metric_name

For more information about the mstats command, refer to Splunk’s mstats documentation.

Format as

There are two supported result format modes - Time series (default) and Table. Table mode is suitable for using with a table panel when you want to display aggregated data. That works with raw events (returns all selected fields) and stats search function, which returns table-like data. Results are similar to Statistics tab in Splunk UI. Examples:

index="_internal" sourcetype="scheduler" | fields host, source
index="_internal" sourcetype="splunkd_access" | stats avg(bytes) as bytes, avg(file) as file by status

Read more about stats function usage in Splunk Search Reference

Visual mode

This mode provide visual step-by-step search functionality. This mode creates timechart splunk search by selecting an index, source type, and metrics, and set split by fields as needed. (HOW DO YOU RUN A VISUAL QUERY)

Metric

You can add multiple metrics to search by clicking plus button at the right side of metric row. Metric editor contains list of frequently used aggregations, but you can specify here any other function.

  1. Select or type an aggregation type. The default is avg.
  2. Select or type a field for the aggregation to use.
  3. (Optional) Fill out an alias.
Split by and where

You need to be using Time series mode to use Split by.

Select which fields to split by (these can be removed by selecting remove in the dropdown).

Click the + button after Where to select a where clause type. Each function in the where clause can be edited after selecting a where clause type (remove the where clause by selection remove in the comparator function)

Find out more at here.

Options

To learn more about timechart options, refer to timechart.

Annotations

annotations editor

Use annotations if you want to show Splunk alerts or events on graph. Annotation can be either predefined Splunk alert or regular splunk search.

Splunk alert

Specify alert name or leave field blank to get all fired alerts. Template variables are supported.

Use splunk search to get needed events, for example:

index=os sourcetype=iostat | where total_ops > 400
index=os sourcetype=iostat | where total_ops > $io_threshold

Template variables are supported.

Event field as text option suitable if you want to use field value as annotation text. For example, error message text from logs:

Event field as text: _raw
Regex: WirelessRadioManagerd\[\d*\]: (.*)

Regex allows to extract a part of message.

Templates and variables

To add a new Splunk query variable, refer to Add a query variable. Use your Splunk data source as your data source.

Query with SPL to return list of values, for example with statscommand:

index=os sourcetype="iostat" | stats values(Device)

This query returns list of Device field values from iostat source. Then you can use these device names for time series queries or annotations.

There are two possible types of variable queries can be used in Grafana:

  • A simple query (as present above) which returns a list of values
  • A query that can create a key/value variable. The query should return two columns that are named _text and _value. The _text column value should be unique (if it is not unique then the first value is used). The options in the dropdown will have a text and value that allows you to have a friendly name as text and an id as the value.

This search returns table with columns Name (Docker container name) and Id (container id):

source=docker_inspect | stats count latest(Name) as Name by Id | table Name, Id

In order to use container name as a visible value for variable and id as it’s real value, query should be modified, like:

source=docker_inspect | stats count latest(Name) as Name by Id | table Name, Id | rename Name as "_text", Id as "_value"

Multi-value variables

It’s possible to use multi-value variables in queries. An interpolated search will be depending on variable usage context. There are a number of that contexts which plugin supports. Assume there’s a variable $container with selected values foo and bar:

  • Basic filter for search command

    source=docker_stats $container
    =>
    source=docker_stats (foo OR bar)
    
  • Field-value filter

    source=docker_stats container_name=$container
    =>
    source=docker_stats (container_name=foo OR container_name=bar)
    
  • Field-value filter with the IN operator and in() function

    source=docker_stats container_name IN ($container)
    =>
    source=docker_stats container_name IN (foo, bar)
    
    source=docker_stats | where container_name in($container)
    =>
    source=docker_stats | where container_name in(foo, bar)
    

Multi-value variables and quotes

If variable wrapped in quotes (both double or single), its values also will be quoted:

source=docker_stats container_name="$container"
=>
source=docker_stats (container_name="foo" OR container_name="bar")

source=docker_stats container_name='$container'
=>
source=docker_stats (container_name='foo' OR container_name='bar')

After creating a variable it can be used in your Splunk queries by using this syntax.

For more information on variables refer this.

Learn more