Splunk

Data SourceENTERPRISE

Splunk datasource
Enterprise

Enterprise plugins, along with support and assistance from the core team behind Grafana, are available with Grafana Enterprise.

Upgrade Now

Splunk data source

Running test environment

In order to get test environment up and running, see docker directory.

Table of contents

Configuration

Data source config

When configuring the Data Source, ensure the URL field utilizes https and points to the your configured Splunk port. The default Splunk API point is 8089, not 8000 (this is default web UI port). Enable Basic Auth and specify Splunk username and password.

Browser (direct) access mode and CORS

If you are using CORS, you'll need to configure the Splunk server to allow Grafana to communicate with it using a CORS connection. To do this, add your web site's address as a trusted HTTP origin to the crossOriginSharingPolicy attribute in the server.conf configuration file.

For example, add this stanza to the $SPLUNK_HOME/etc/system/local/server.conf configuration file, then restart Splunk:

[httpServer]
crossOriginSharingPolicy = http://localhost:3000

See more info in original article Communicate with the Splunk server for apps outside of Splunk Web

Note: we don't recommend to use browser (direct) access mode. It almost always is better to connect to Splunk via Grafana backend in server (proxy) access mode. Use browser (direct) only if you really need it and know how it works.

ds config

Advanced options

ds config advanced

Clustering strategy

The default is using a session key, but use basic authentication if you are load balancing your Splunk server.

Stream mode

Enable stream mode if you want to get search results as they become available. Note: This is an experimental feature; do not enable it until you really need it.

Auto cancel

If specified, the job automatically cancels after this many seconds of inactivity (0 means never auto-cancel). Default is 30.

Poll result

Run search and then periodically check for result. Under the hood this option runs search/jobs API call with exec_mode set to normal. In this case API request returns job SID, and then Grafana checks job status time to time, in order to get job result. This option may be helpful for slow queries. By default this option is disabled and Grafana sets exec_mode to oneshot which allows returning search result in the same API call. See more about search/jobs API endpoint in Splunk docs

Internal fields filtration

Do not display fields with names starting with '_'.

Time stamp field

The time stamp field contains a timestamp that is in UNIX time.

This is used to:

  • Correlate events by time
  • Create timeline histograms
  • Set time ranges for searches

More information on this field can be found here.

Fields search mode

When you use visual query editor, data source attempts to get list of available fields for selected source type.

  • quick - use first available result from preview
  • full - wait for job finish and get full result.

Default earliest time

Some searches can't use dashboard time range (such as template variable queries). This option helps to prevent search for all time, which can slow down Splunk. The syntax is an integer and a time unit [+|-]<time_integer><time_unit>. For example -1w. Time unit can be s, m, h, d, w, mon, q, y.

Variables search mode

Search mode for template variable queries. Possible values:

  • fast - Field discovery off for event searches. No event or field data for stats searches.
  • smart - Field discovery on for event searches. No event or field data for stats searches.
  • verbose - All event & field data.

Usage

Query editor

Editor modes

Query editor support two modes: raw and visual. To switch between these modes click hamburger icon at the right side of editor and select Toggle Editor Mode.

Raw mode

Use timechart command for timeseries data. For example:

index=os sourcetype=cpu | timechart span=1m avg(pctSystem) as system, avg(pctUser) as user, avg(pctIowait) as iowait
index=os sourcetype=ps | timechart span=1m limit=5 useother=false avg(cpu_load_percent) by process_name

Queries support template variables:

sourcetype=cpu | timechart span=1m avg($cpu)

Keep in mind that Grafana is timeseries-oriented application and your search should return timeseries data (timestamp and value) or single value. You can read about timechart command and find more search examples in official Splunk Search Reference

Splunk metrics and mstats

Splunk 7.x provides mstats command for analyzing metrics. In order to get charts working properly with mstats, it should be combined with timeseries command and prestats=t option must be set.

Deprecated syntax:
| mstats prestats=t avg(_value) AS Value WHERE index="collectd" metric_name="disk.disk_ops.read" OR metric_name="disk.disk_ops.write" by metric_name span=1m
| timechart avg(_value) span=1m by metric_name

Actual:
| mstats prestats=t avg(disk.disk_ops.read) avg(disk.disk_ops.write) WHERE index="collectd" by metric_name span=1m
| timechart avg(disk.disk_ops.read) avg(disk.disk_ops.write) span=1m

Read more about mstats command in Splunk Search Reference.

Format as

There are two supported result format modes - Time series (default) and Table. Table mode suitable for using with Table panel when you want to display aggregated data. That works with raw events (returns all selected fields) and stats search function, which returns table-like data. Examples:

index="_internal" sourcetype="scheduler" | fields host, source
index="_internal" sourcetype="splunkd_access" | stats avg(bytes) as bytes, avg(file) as file by status

Result is similar to Statistics tab in Splunk UI. Grafana table Splunk statistics

Read more about stats function usage in Splunk Search Reference

Visual mode

query editor

This mode provide easy to use step-by-step search creating. Note, that this mode creates timechart splunk search. Just select index, source type, and metrics, and set split by fields if you want.

Metric

You can add multiple metrics to search by clicking plus button at the right side of metric row. Metric editor contains list of frequently used aggregations, but you can specify here any other function. Just click on agg segment (avg by default) and type what you need. Select interested field from dropdonw (or type) and set alias if you want.

Split by and where

split by and where

If you set Split by field and use Time series mode, Where editor will be available. Click plus and select operator, aggregation and value, for example Where avg in top 10. Note, this Where clause is a part of Split by. See more at timechart docs.

Options

To change default timechart options, change the fields below the Split by row:

query options See more about these options in timechart docs.

Rendered splunk search

Click on target letter at the left to collapse editor and show rendered splunk search.

Annotations

annotations editor

Use annotations if you want to show Splunk alerts or events on graph. Annotation can be either predefined Splunk alert or regular splunk search.

Splunk alert

Specify alert name or leave field blank to get all fired alerts. Template variables are supported.

Splunk search

Use splunk search to get needed events, for example:

index=os sourcetype=iostat | where total_ops > 400
index=os sourcetype=iostat | where total_ops > $io_threshold

Template variables are supported.

Event field as text option suitable if you want to use field value as annotation text. For example, error message text from logs:

Event field as text: _raw
Regex: WirelessRadioManagerd\[\d*\]: (.*)

Regex allows to extract a part of message.

Template variables

Template variables feature supports Splunk queries which return list of values, for example with statscommand:

index=os sourcetype="iostat" | stats values(Device)

This query returns list of Device field values from iostat source. Then you can use these device names for timeseries queries or annotations.

There're two possible types of variable queries can be used in Grafana. First is a simple query (as present above) which retunrs a list of values. Second type is a query that can create a key/value variable. The query should return two columns that are named _text and _value. The _text column value should be unique (if it is not unique then the first value is used). The options in the dropdown will have a text and value that allows you to have a friendly name as text and an id as the value.

For instance, this search returns table with columns Name (Docker container name) and Id (container id):

source=docker_inspect | stats count latest(Name) as Name by Id | table Name, Id

In order to use container name as a visible value for variable and id as it's real value, query should be modified, like:

source=docker_inspect | stats count latest(Name) as Name by Id | table Name, Id | rename Name as "_text", Id as "_value"

Multi-value variables

It's possible to use multi-value variables in queries. An interpolated search will be depending on variable usage context. There are a number of that contexts which plugin supports. Assume there's a variable $container with selected values foo and bar:

  • Basic filter for search command

    source=docker_stats $container
    =>
    source=docker_stats (foo OR bar)
    
  • Field-value filter

    source=docker_stats container_name=$container
    =>
    source=docker_stats (container_name=foo OR container_name=bar)
    
  • Field-value filter with the IN operator and in() function

    source=docker_stats container_name IN ($container)
    =>
    source=docker_stats container_name IN (foo, bar)
    
    source=docker_stats | where container_name in($container)
    =>
    source=docker_stats | where container_name in(foo, bar)
    

Multi-value variables and quotes

If variable wrapped in quotes (both double or single), its values also will be quoted:

source=docker_stats container_name="$container"
=>
source=docker_stats (container_name="foo" OR container_name="bar")

source=docker_stats container_name='$container'
=>
source=docker_stats (container_name='foo' OR container_name='bar')

Data links

Datalinks

Datalinks allow you to look into Splunk log, parse out data, and link to that data.

Datalinks can be either internal or external.

Internal datalinks

Using internal datalinks, you can link one data source to another data source within Grafana. (Currently only tracing data sources tested, see the preceding screenshot) Datalinks * Field: The field from which to parse data. Logging data in Splunk is often returned in the _raw column. * Label: Given a key and value such as color=white, the label is the key in this key/value pair. * Regex: Given a key and value such as color=white, the regex parses out the value in the key value pair from the field. The regex is a matching regex; you must provide one set of brackets for the matching expression. For example, /(.*)/ to match everything. Important: You must wrap the regex in //. * Query: For now the only supported query is for tracing data sources, and it must be ${__value.raw}. This can, and will be expanded in the future.

  • External datalinks: Link to a URL that is based on the value parsed out of the Splunk logs. Datalinks

    • Field: The field from which to parse data. Logging data in Splunk is often returned in the _raw column.
    • Label: Given a key and value such as color=white, the label is the key in this key/value pair.
    • Regex: Given a key and value such as color=white, the regex parses out the value in the key value pair from the field. The regex is a matching regex; you must provide one set of brackets for the matching expression. For example, /(.*)/ to match everything.

    Important: You must have the regex wrapped in //.

    • URL: Use the variable ${__value.raw} to hold the value of the data parsed out with the regex. You can use this to construct a URL.

Datalinks

The data displays a link that allows you to associate data internally with other Grafana data sources, or externally with data via a URL.

Sign up Now

Version

Dependencies:
  • Grafana 7.0.0