Documentation Plugins Splunk Infrastructure Monitoring

Splunk Infrastructure Monitoring data source

The Splunk Infrastructure Monitoring data source plugin allows you to query and visualize Splunk Infrastructure Monitoring metrics using SignalFlow queries. You can also use template variables for dynamic dashboards and create annotations from alerts and events.

Supported Splunk environments

The Splunk Infrastructure Monitoring data source supports the following Splunk deployment types:

• Splunk Observability Cloud - Splunk’s cloud-hosted observability platform (formerly known as SignalFx).
• All regional realms - US (us0, us1, us2), EU (eu0), and AP (ap0) regions are supported.

Get started

The following sections will help you get started with the Splunk Infrastructure Monitoring data source:

• Configure the data source
• Query the data source
• Use template variables
• Troubleshoot

Additional features

Once you have configured the data source, you can:

• Add Annotations to overlay Splunk alerts and events on your graphs.
• Configure and use Templates and variables for dynamic dashboards.
• Add Transformations.
• Set up Alerting to monitor your Splunk metrics.

Before you begin

To configure the Splunk Infrastructure Monitoring data source, you need:

• Grafana Organization administrator role to add a data source.
• A Splunk Infrastructure Monitoring (previously SignalFx) account.
• An access token generated from your SignalFx account. To learn more about access token types, refer to authentication tokens.
• Your realm name, which you can find on your profile page when signed in to the SignalFx user interface.

Add the Splunk Infrastructure Monitoring data source

For general information on adding a data source, refer to Add a data source.

Complete the following steps to add a new Splunk Infrastructure Monitoring data source:

1. Click Connections in the left-side menu.
2. Click Add new connection.
3. Type Splunk Infrastructure Monitoring in the search bar.
4. Select the Splunk Infrastructure Monitoring data source.
5. Click Add new data source in the upper right.

Grafana takes you to the Settings tab, where you will set up your Splunk Infrastructure Monitoring configuration.

Configure the data source

The following table describes configuration options available in the Settings tab:

Custom URLs

Use this section only if you are using custom SignalFlow domains. Leave these fields blank for the default behavior.

Secure Socks Proxy

If you are running Grafana 10.0.0 or later and have a secure socks proxy configured, you can enable proxying the data source connection through the secure socks proxy to a different network.

For more details, refer to Configure a datasource connection proxy.

Provision the data source

You can configure data sources using config files with Grafana’s provisioning system. For more information, refer to the provisioning docs page.

The following example provisions a Splunk Infrastructure Monitoring data source:

YAML Copy

apiVersion: 1
datasources:
  - name: Splunk Infrastructure Monitoring
    type: grafana-splunk-monitoring-datasource
    access: proxy
    basicAuth: false
    editable: true
    enabled: true
    jsonData:
      realmName: us1
    secureJsonData:
      accessToken: <your-access-token>

Import a dashboard

The Splunk Infrastructure Monitoring data source includes a pre-built dashboard that you can import to get started quickly.

To import the pre-built dashboard:

1. Go to the data source’s configuration page.
2. Select the Dashboards tab.
3. Click Import next to the dashboard you want to import.

Query the data source

The query editor accepts a SignalFlow program or query. To learn more about SignalFlow, refer to SignalFlow Analytics Language.

SignalFlow query examples

The following examples demonstrate common SignalFlow query patterns:

Basic metric query:

signalflow Copy

data('cpu.utilization').publish()

Query with rollup and aggregation:

signalflow Copy

data('demo.trans.count', rollup='rate').sum().publish(label='Total Transactions')

Query with filter:

signalflow Copy

data('cpu.utilization', filter=filter('host', 'server1')).publish()

Query with time window aggregation:

signalflow Copy

data('demo.trans.latency').mean(over='5m').publish()

Use multiple queries

You can write multiple queries in a single panel and perform calculations between them. Assign each query to a variable and reference it in subsequent calculations:

signalflow Copy

A = data('demo.trans.latency').sum(by=['demo_customer']).publish(label='A', enable=False)
B = data('demo.trans.count', rollup='rate').sum(by=['demo_customer']).publish(label='B', enable=False)
C = (A / B).publish(label='Latency per Transaction')

In this example, queries A and B are calculated but hidden (enable=False), and only the result C is displayed.

Use SignalFlow labels

SignalFlow labels are applied as metadata to the results. For example, publish(label = 'foo') adds a label="foo" to the metadata.

Use ad-hoc filters

The Splunk Infrastructure Monitoring data source supports ad-hoc filters. Ad-hoc filters allow you to add filters to your SignalFlow queries dynamically without modifying the query itself.

When you add ad-hoc filters to a dashboard, the plugin automatically appends filter() clauses to your SignalFlow queries. For example, if you add an ad-hoc filter for region = us-west-1, the plugin modifies your query to include filter('region','us-west-1').

To use ad-hoc filters:

1. Add an ad-hoc filter variable to your dashboard.
2. Select your Splunk Infrastructure Monitoring data source.
3. Use the filter controls to add key-value pairs.

The filters are applied to all panels using the Splunk Infrastructure Monitoring data source on that dashboard.

Use template variables

To add a new Splunk Infrastructure Monitoring query variable, refer to Add a query variable. Use your Splunk Infrastructure Monitoring data source as your data source and select one of the following query types:

Metrics

Returns a list of available metrics. To learn more about metrics, refer to metric.

Tags

Returns a list of available tags. To learn more about tags, refer to tag.

Dimensions

Returns dimension keys or values. To learn more about dimensions, refer to dimension.

When you select Dimensions, you can optionally configure the following fields:

After creating a variable, you can use it in your Splunk Infrastructure Monitoring queries by using Variable syntax. For more information about variables, refer to Templates and variables.

Create annotations

Annotations allow you to overlay event data on your graphs. The Splunk Infrastructure Monitoring data source supports annotations using SignalFlow Alerts or Events queries.

To add an annotation:

1. Open a dashboard and click Settings (gear icon).
2. Select Annotations from the settings menu.
3. Click Add annotation query.
4. Select your Splunk Infrastructure Monitoring data source.
5. Enter a SignalFlow query for alerts or events.
6. Click Save dashboard.

Query alerts

Use the alerts() function to display detector alerts as annotations. Alerts are triggered when conditions defined in your Splunk detectors are met.

Example query for alerts from a specific detector:

signalflow Copy

alerts(detector_name='Deployment').publish()

The following fields are returned for alert annotations:

Query events

Use the events() function to display custom events as annotations. Custom events are user-defined events sent to Splunk Infrastructure Monitoring.

Example query for events by type:

signalflow Copy

events(eventType='simulated').publish()

The following fields are returned for event annotations:

Configure Grafana Alerting

This data source supports Grafana Alerting. You can create alert rules based on SignalFlow queries to monitor your Splunk metrics and receive notifications when conditions are met.

To create an alert rule:

1. Navigate to Alerting > Alert rules in Grafana.
2. Click New alert rule.
3. Select your Splunk Infrastructure Monitoring data source.
4. Enter a SignalFlow query to define the data you want to monitor.
5. Configure the alert condition, evaluation interval, and notification settings.

For more information, refer to Grafana Alerting.

Troubleshoot

For solutions to common issues, refer to Troubleshoot the Splunk Infrastructure Monitoring data source.

Plugin updates

Always ensure that your plugin version is up-to-date so you have access to all current features and improvements. Navigate to Plugins and data > Plugins to check for updates. Grafana recommends upgrading to the latest Grafana version, and this applies to plugins as well.