Plugins 〉Splunk

Data Source
grafana
Enterprise

Splunk

  • Overview
  • Installation
  • Change log
  • Related content

Splunk data source for Grafana

The Splunk data source allows you to query and visualize Splunk data with Search Processing Language (SPL) or a visual SPL editor.

Requirements

The Splunk data source has the following requirements:

Known limitations

There are no known limitations.

Walkthrough tutorial

Follow along step-by-step to set up Splunk as a data source in your Grafana account. Try tutorial

Install the data source

To install the data source, refer to Installation.

Configure the data source

To add a data source, refer to Add a data source, and fill in the following fields:

Basic fields

FieldDescription
NameA name for this particular Splunk data source
URLYour access method is Server. The URL needs to be accessible from the Grafana backend server.
Whitelisted CookiesGrafana proxy deletes forwarded cookies by default. Specify the cookies by name that you wan to forward to the data source.

Authentication fields

Most data sources use either Basic Auth or an API key.

Basic auth : Enter a Splunk user name and password. Avoid using the default administration account. TLS Client Auth : The built-in option for authenticating using TLS (Transport Layer Security) Skip TLS Verify : Enable to skip verifying TLS With Credentials : Enable to send credentials, such as cookies or authentication headers with cross-site requests. With CA Cert : Enable to verify self-signed TLS. Forward OAuth Identity : Forward the identity of the Oauth user who is signed in to Grafana, in cases where the same Oauth provider is used for both Grafana and the data source

You can configure custom HTTP header data sources that are managed by Grafana’s provisioning. Doing so allows you to add HTTP headers to all requests that go to that data source. Configure the header name in the jsonData field, and the header value in the secureJsonData field.

Splunk details

Clustering strategy : session key use if your Splunk does not use a load balancer. Splunk does not share session keys between nodes.
: basic auth use if your Splunk server uses a load balancer.

Stream mode : (EXPERIMENTAL) Enable to get search results as they become available

Auto cancel : Number of seconds that a job can be inactive before it is automatically canceled. The default is 30 seconds. To never auto cancel, set to 0.

Poll result : Run a search, and then periodically check for a result. Under the hood, this option runs a search/jobs API call with exec_mode set to normal. In such a case, the API request returns the job’s SID (security identifier), and then Grafana periodically checks the job status, in order to get job result. : This option might be helpful for slow queries. This option is disabled by default, and Grafana sets exec_mode to oneshot, which allows you to return the search result in the same API call. For more information about the search/jobs API endpoint, refer to the search/jobs/{search_id}/results in the Splunk documentation.

Internal fields filtration : Enable to hide fields with names that start with _.

Internal field pattern : Regex pattern for removing internal fields from the result.

Time stamp field : Default time stamp field. For more information, refer to Time stamp and time ranges.

Fields search mode : When you use the visual query editor, the data source tries to get a list of available fields for the selected source type. : quick uses the first available result from preview : full waits for the job to finish to get full result

Default earliest time : Time offset format: [+ | -]<integer><time_unit> For example, -1w. Time unit: s, m, h, d, w, mon, q, y : By using a default earliest time, you can prevent a search from spanning all of time, which might slow down Splunk. Some searches, such as template variable queries, cannot use a dashboard time range.

Variables search mode : Search mode for template variable queries. : fast: turn field discovery off for event searches. No event or field data for stats searches. : smart turn field discovery on for event searches. No event or field data for stats searches. : verbose All event and field data.

Data links

Data links are typically used in Grafana's explore mode. The data displays a link that allows you to associate data internally with other Grafana data sources, or externally with data via a URL.

Set up a data link by clicking the + Add button under the Data links header in data source settings.

Field : The field from which to parse data. Logging data in Splunk is often returned in the _raw column.

Label : Given a key and value such as color=white, the label is the key in this key/value pair.

Regex : Given a key and value such as color=white, the regex parses out the value in the key value pair from the field. The regex is a matching regex, so you must provide one set of brackets for the matching expression. To match everything, use /(.*)/. You must wrap the regex in //.

Turn off Internal link to link to a URL that is based on the value parsed out of the Splunk logs. URL: Use the variable ${__value.raw} to hold the value of the data parsed out with the regex. You can use this to construct a URL.

Turn on Internal link to use an external link and link one data source to another data source within Grafana.

Configure the data source with provisioning

It is possible to configure data sources using config files with Grafana’s provisioning system. You can read more about how it works and all the settings you can set for data sources on the provisioning docs page

apiVersion: 1

datasources:

  • name: Splunk type: grafana-splunk-datasource access: proxy basicAuth: true basicAuthUser: user editable: true enabled: true jsonData: advancedOptions: true fieldSearchType: quick internalFieldsFiltration: true tlsSkipVerify: true variableSearchLevel: fast streamMode: false clusteringStrategy: 1 secureJsonData: basicAuthPassword: password url: splunk url version: 1

Query the data source

Query editor support two modes: spl and visual. To switch between these modes click hamburger icon at the right side of editor and select Toggle Editor Mode.

SPL mode

Use SPL mode by querying with Search Processing Language (SPL). Find more information on SPL here.

For time series data use timechart command. For example:

index=os sourcetype=cpu | timechart span=1m avg(pctSystem) as system, avg(pctUser) as user, avg(pctIowait) as iowait
index=os sourcetype=ps | timechart span=1m limit=5 useother=false avg(cpu_load_percent) by process_name

Grafana is a time-series-oriented application, and your search query returns time-series data (in the form of a timestamp and value, or a single value). Find more information about the timechart command, refer to timechart. For more search query examples, refer to Splunk® Enterprise Search Reference.

Splunk metrics and mstats

Splunk 7.x provides an mstats command for analyzing metrics, which you need to combine with the timeseries command for it to work properly. You must also set the prestats=t option.

Current syntax:
| mstats prestats=t avg(disk.disk_ops.read) avg(disk.disk_ops.write) WHERE index="collectd" by metric_name span=1m
| timechart avg(disk.disk_ops.read) avg(disk.disk_ops.write) span=1m

Deprecated syntax: | mstats prestats=t avg(_value) AS Value WHERE index="collectd" metric_name="disk.disk_ops.read" OR metric_name="disk.disk_ops.write" by metric_name span=1m | timechart avg(_value) span=1m by metric_name

For more information about the mstats command, refer to Splunk’s mstats documentation.

Format as

There are two supported result format modes - Time series (default) and Table. Table mode is suitable for using with a table panel when you want to display aggregated data. That works with raw events (returns all selected fields) and stats search function, which returns table-like data. Results are similar to Statistics tab in Splunk UI. Examples:

index="_internal" sourcetype="scheduler" | fields host, source
index="_internal" sourcetype="splunkd_access" | stats avg(bytes) as bytes, avg(file) as file by status

Read more about stats function usage in Splunk Search Reference

Visual mode

This mode provide visual step-by-step search functionality. This mode creates timechart splunk search by selecting an index, source type, and metrics, and set split by fields as needed. (HOW DO YOU RUN A VISUAL QUERY)

Metric

You can add multiple metrics to search by clicking plus button at the right side of metric row. Metric editor contains list of frequently used aggregations, but you can specify here any other function.

  1. Select or type an aggregation type. The default is avg.
  2. Select or type a field for the aggregation to use.
  3. (Optional) Fill out an alias.
Split by and where

You need to be using Time series mode to use Split by.

Select which fields to split by (these can be removed by selecting remove in the dropdown).

Click the + button after Where to select a where clause type. Each function in the where clause can be edited after selecting a where clause type (remove the where clause by selection remove in the comparator function)

Find out more at here.

Options

To learn more about timechart options, refer to timechart.

Annotations

annotations editor

Use annotations if you want to show Splunk alerts or events on graph. Annotation can be either predefined Splunk alert or regular splunk search.

Splunk alert

Specify alert name or leave field blank to get all fired alerts. Template variables are supported.

Splunk search

Use splunk search to get needed events, for example:

index=os sourcetype=iostat | where total_ops > 400
index=os sourcetype=iostat | where total_ops > $io_threshold

Template variables are supported.

Event field as text option suitable if you want to use field value as annotation text. For example, error message text from logs:

Event field as text: _raw
Regex: WirelessRadioManagerd\[\d*\]: (.*)

Regex allows to extract a part of message.

Templates and variables

To add a new Splunk query variable, refer to Add a query variable. Use your Splunk data source as your data source.

Query with SPL to return list of values, for example with statscommand:

index=os sourcetype="iostat" | stats values(Device)

This query returns list of Device field values from iostat source. Then you can use these device names for time series queries or annotations.

There are two possible types of variable queries can be used in Grafana:

  • A simple query (as present above) which returns a list of values
  • A query that can create a key/value variable. The query should return two columns that are named _text and _value. The _text column value should be unique (if it is not unique then the first value is used). The options in the dropdown will have a text and value that allows you to have a friendly name as text and an id as the value.

This search returns table with columns Name (Docker container name) and Id (container id):

source=docker_inspect | stats count latest(Name) as Name by Id | table Name, Id

In order to use container name as a visible value for variable and id as it's real value, query should be modified, like:

source=docker_inspect | stats count latest(Name) as Name by Id | table Name, Id | rename Name as "_text", Id as "_value"

Multi-value variables

It's possible to use multi-value variables in queries. An interpolated search will be depending on variable usage context. There are a number of that contexts which plugin supports. Assume there's a variable $container with selected values foo and bar:

  • Basic filter for search command

    source=docker_stats $container
    =>
    source=docker_stats (foo OR bar)
    
  • Field-value filter

    source=docker_stats container_name=$container
    =>
    source=docker_stats (container_name=foo OR container_name=bar)
    
  • Field-value filter with the IN operator and in() function

    source=docker_stats container_name IN ($container)
    =>
    source=docker_stats container_name IN (foo, bar)
    

    source=docker_stats | where container_name in($container) => source=docker_stats | where container_name in(foo, bar)

Multi-value variables and quotes

If variable wrapped in quotes (both double or single), its values also will be quoted:

source=docker_stats container_name="$container"
=>
source=docker_stats (container_name="foo" OR container_name="bar")

source=docker_stats container_name='$container' => source=docker_stats (container_name=‘foo’ OR container_name=‘bar’)

After creating a variable it can be used in your Splunk queries by using this syntax.

For more information on variables refer this.

Import a dashboard for Splunk

Follow these instructions for importing a dashboard.

Imported dashboards can be found in Configuration > Data Sources > select your Splunk data source > select the Dashboards tab to see available pre-made dashboards.

Learn more

For more information, visit the docs on plugin installation.

Change Log

[2.1.7] - 2021-09-08

  • Fixed issue that would filter internal fields even if internal field filteration was disabled

[2.1.6] - 2021-08-04

  • Documentation update

[2.1.5] - 2021-07-30

  • Fixed issue where alerting queries don't filter out internal fields
  • Results that return data in multiple different types are typed as string

[2.1.4] - 2021-06-02

Bug Fixes

  • #278: 'No data' displayed for Grafana 8

[2.1.3] - 2021-05-27

Bug Fixes

  • #265: Errors returned when in polling mode cause the driver to crash
  • #226: Filter fields prefixed with _ if filtration is on

[2.1.2] - 2021-4-12

  • Chore: Update SDK

[2.1.1] - 2020-04-06

Bug Fixes

  • #254: To template variables propery, splunk should be passing in scopedvars into the templating engine
  • #224: Splunk executes queries that have hide = true (the eye icon)
  • #222: Get the splunk changelog to publish to GCOM by upgrading to the "next" toolkit"
  • #246: Splunk should not refresh the query immediately after every pause in typing/editing the query
  • #239: Enable E2E tests
  • #215: "Skip TLS Verify" option not working
  • #213: "Post forbidden" error after upgrading to 2.0.4
  • #208: Update to new SDK to support individual plugin licensing

[2.1.0] - 2020-01-25

Bug fixes

  • #212: Return all indexes
  • #235: Better handle edge cases
  • #236: Better handle an out-of-bounds error

Enhancements

  • #187: Update README
  • #210: Link from logs to traces using data links or derived fields

Known issues

  • You must use // around regular expressions in the datalinks configuration or any plugin health check will fail. When a regular expression is optional, use an empty //.

[2.0.5] - 2020-12-11

Bug fixes

  • #214: Time filters not working for Splunk queries in dashboard variables
  • #210,#209: Plugin crashes a lot while trying out queries

[2.0.4] - 2020-10-27

  • Fix: If session key is empty, still valid
  • Enhancement: When using splunk in clustered mode, allow user to revert to 'basic auth'

[2.0.3] - 2020-10-19

  • Fix for proxy settings, now will recognize standard "http_proxy" "https_proxy" and "no_proxy" environment variables
  • TLS Skip Verification now performs verification when not checked

[2.0.2] - 2020-09-10

  • Fix for time range issue #184

[2.0.1] - 2020-06-01

Issues Resolved

  • Issues with timeseries/table processing #164
  • Needs to return raw logging as first frame #171

[2.0.0] - 2020-05-22

  • Rewritten as a backend plugin with native alerting support
  • Removed distinction between table & time_series results
  • Signed version for grafana 7.x

[1.5.0] - 2020-05-17

  • Signed release for Grafana 7.x

[1.4.1] - 2020-01-10

Fixed

  • "Unbalanced quotes" error when using variables in some scenarios
  • Logs panel does not show data in logs format
  • Query returns only first 100 points

[1.4.0] - 2019-11-26

Added

  • Explore logs integration

[1.3.4] - 2019-11-22

Fixed

  • Plugin failed with "Unexpected token <" error

[1.3.3] - 2019-11-21

Fixed

  • Query in template variable with no result data is running forever #82

[1.3.2] - 2019-10-03

Fixed

  • Plugin failed in Grafana 6.4 with error "Unexpected token <"

[1.3.1] - 2019-10-01

Fixed

  • Streaming support in Grafana 6.4 (incompatible with previous version)

[1.3.0] - 2019-07-30

Added

  • Variable query editor with namespace support, #63

Fixed

  • Streaming support in Grafana 6.3

[1.2.2] - 2019-06-29

Fixed

  • Bug with semicolon in query, #57
  • Variable expansion when concatenated with litteral string, #58
  • Variable expansion when there is a period character in the name, #59

[1.2.1] - 2019-03-18

Fixed

  • Query fails when search contains curly braces, #45
  • Unresponsive query field size in Grafana 6, #47

[1.2.0] - 2019-02-01

Added

  • Search editor with syntax highlighting, #33
  • Option for disabling internal fields filtration, #37

Fixed

  • Table panel is empty after update to v1.1.0, #34
  • Panel is not refreshing on leaving search field, #40

[1.1.0] - 2018-11-14

Added

  • Multi-value template variables support, #2
  • Use time range for variable queries if Refresh option set to On Time Range Change
  • Option for variables search mode in datasource config (fast/smart/verbose)

Fixed

  • Variable query returns only first value, #22
  • Grafana shows only first 100 points, #6
  • Use oneshot or async search for template variables query depending on datasource config