Documentation Plugins Splunk Infrastructure Monitoring

Splunk Infrastructure Monitoring data source

The Splunk Infrastructure Monitoring data source plugin lets you query and visualize Splunk Infrastructure Monitoring metrics using SignalFlow queries. You can also use template variables for dynamic dashboards and create annotations from alerts and events.

Supported features

The following table lists the features that the Splunk Infrastructure Monitoring data source supports:

Supported Splunk environments

The Splunk Infrastructure Monitoring data source connects to Splunk Observability Cloud, the cloud-hosted observability platform formerly known as SignalFx.

Splunk Observability Cloud runs in self-contained regional deployments called realms. The data source supports all realms, including the following:

• US: us0, us1, us2
• Europe: eu0, eu1, eu2
• Asia Pacific: au0, jp0, sg0

To find your realm, open the navigation menu in Splunk Observability Cloud, select Settings, and then select your username. The realm name appears in the Organizations section. For more information, refer to View your realm, API endpoints, and organization in the Splunk documentation.

Get started

The following sections help you get started with the Splunk Infrastructure Monitoring data source:

• Configure the data source
• Query the data source
• Use template variables
• Troubleshoot

Additional features

After you configure the data source, you can:

• Use Explore to run SignalFlow queries without building a dashboard.
• Add annotations to overlay Splunk alerts and events on your graphs.
• Configure and use template variables for dynamic dashboards.
• Add transformations to manipulate query results.
• Set up alerting to monitor your Splunk metrics.

Before you begin

To configure the Splunk Infrastructure Monitoring data source, you need the following:

• Grafana version 11.6.7 or later.
• The Grafana organization administrator role to add a data source.
• A Grafana Cloud Pro or Advanced plan or an activated self-managed Grafana Enterprise license.
• A Splunk Observability Cloud account, formerly SignalFx.
• An access token generated from your Splunk Observability Cloud account. To learn more about access token types, refer to Authentication tokens in the Splunk documentation.
• Your realm name. Refer to Supported Splunk environments for how to find it.

Add the Splunk Infrastructure Monitoring data source

For general information on adding a data source, refer to Add a data source.

Complete the following steps to add a new Splunk Infrastructure Monitoring data source:

1. Click Connections in the left-side menu.
2. Click Add new connection.
3. Type Splunk Infrastructure Monitoring in the search bar.
4. Select the Splunk Infrastructure Monitoring data source.
5. Click Add new data source in the upper right.

Grafana takes you to the Settings tab, where you set up your Splunk Infrastructure Monitoring configuration.

Configure the data source

The following table describes the configuration options available in the Settings tab:

Custom URLs

Use this section only if you connect through custom domains, such as the observability.splunkcloud.com endpoints. Leave these fields blank to use the default endpoints for your realm.

Secure Socks Proxy

If you have a secure socks proxy configured, you can enable proxying the data source connection through the secure socks proxy to a different network.

For more details, refer to Configure a data source connection proxy.

Provision the data source

You can configure data sources using configuration files with the Grafana provisioning system. For more information, refer to Provision the data source.

The following example provisions a Splunk Infrastructure Monitoring data source:

YAML Copy

apiVersion: 1
datasources:
  - name: Splunk Infrastructure Monitoring
    type: grafana-splunk-monitoring-datasource
    access: proxy
    basicAuth: false
    editable: true
    enabled: true
    jsonData:
      realmName: us1
      # Optional. Set these only when you use custom domains.
      # url_metrics_metadata: https://api.us1.observability.splunkcloud.com
      # url_signalflow: https://stream.us1.observability.splunkcloud.com
      # enableSecureSocksProxy: false
    secureJsonData:
      accessToken: <your-access-token>

Import a dashboard

The Splunk Infrastructure Monitoring data source includes a pre-built dashboard that you can import to get started quickly.

To import the pre-built dashboard:

1. Go to the data source’s configuration page.
2. Select the Dashboards tab.
3. Click Import next to the dashboard you want to import.

Query the data source

The query editor is a code editor with Python syntax highlighting that accepts a SignalFlow program or query. To learn more about SignalFlow, refer to SignalFlow analytics in the Splunk documentation.

SignalFlow query examples

The following examples demonstrate common SignalFlow query patterns:

Basic metric query:

signalflow Copy

data('cpu.utilization').publish()

Query with rollup and aggregation:

signalflow Copy

data('demo.trans.count', rollup='rate').sum().publish(label='Total Transactions')

Query with filter:

signalflow Copy

data('cpu.utilization', filter=filter('host', 'server1')).publish()

Query with time window aggregation:

signalflow Copy

data('demo.trans.latency').mean(over='5m').publish()

Use multiple queries

You can write multiple queries in a single panel and perform calculations between them. Assign each query to a variable and reference it in subsequent calculations:

signalflow Copy

A = data('demo.trans.latency').sum(by=['demo_customer']).publish(label='A', enable=False)
B = data('demo.trans.count', rollup='rate').sum(by=['demo_customer']).publish(label='B', enable=False)
C = (A / B).publish(label='Latency per Transaction')

In this example, queries A and B are calculated but hidden (enable=False), and only the result C is displayed.

Use SignalFlow labels

SignalFlow labels are applied as metadata to the results. For example, publish(label = 'foo') adds a label="foo" to the metadata.

Use the Ad hoc filters variable

The Splunk Infrastructure Monitoring data source supports the Ad hoc filters variable, which lets you add filters to your SignalFlow queries dynamically without modifying the query itself.

When you add the Ad hoc filters variable to a dashboard, the plugin automatically appends filter() clauses to your SignalFlow queries. For example, a filter for region = us-west-1 adds filter('region','us-west-1') to your query.

To use the Ad hoc filters variable:

1. Add an Ad hoc filters variable to your dashboard.
2. Select your Splunk Infrastructure Monitoring data source.
3. Use the filter controls to add key-value pairs.

The plugin applies the filters to all panels that use the Splunk Infrastructure Monitoring data source on that dashboard.

Use template variables

To add a new Splunk Infrastructure Monitoring query variable, refer to Add a query variable. Use your Splunk Infrastructure Monitoring data source and select one of the following query types:

Metrics

Returns a list of available metrics. To learn more about metrics, refer to metric.

Tags

Returns a list of available tags. To learn more about tags, refer to tag.

Dimensions

Returns dimension keys or values. To learn more about dimensions, refer to dimension.

When you select Dimensions, you can optionally configure the following fields:

After you create a variable, you can use it in your Splunk Infrastructure Monitoring queries with variable syntax. For more information about variables, refer to Template variables.

Create annotations

Annotations let you overlay event data on your graphs. The Splunk Infrastructure Monitoring data source supports annotations using SignalFlow alerts or events queries.

To add an annotation:

1. Open a dashboard and click Settings (gear icon).
2. Select Annotations from the settings menu.
3. Click Add annotation query.
4. Select your Splunk Infrastructure Monitoring data source.
5. Enter a SignalFlow query for alerts or events.
6. Click Save dashboard.

Query alerts

Use the alerts() function to display detector alerts as annotations. Alerts are triggered when conditions defined in your Splunk detectors are met.

Example query for alerts from a specific detector:

signalflow Copy

alerts(detector_name='Deployment').publish()

The following fields are returned for alert annotations:

Query events

Use the events() function to display custom events as annotations. Custom events are user-defined events sent to Splunk Infrastructure Monitoring.

Example query for events by type:

signalflow Copy

events(eventType='simulated').publish()

The following fields are returned for event annotations:

Configure Grafana Alerting

This data source supports Grafana Alerting. You can create alert rules based on SignalFlow queries to monitor your Splunk metrics and receive notifications when conditions are met.

To create an alert rule:

1. Navigate to Alerting > Alert rules in Grafana.
2. Click New alert rule.
3. Select your Splunk Infrastructure Monitoring data source.
4. Enter a SignalFlow query to define the data you want to monitor.
5. Configure the alert condition, evaluation interval, and notification settings.

For more information, refer to Grafana Alerting.

Troubleshoot

For solutions to common issues, refer to Troubleshoot Splunk Infrastructure Monitoring data source issues.

Plugin updates

Always keep your plugin version up to date so you have access to all current features and improvements. Navigate to Plugins and data > Plugins to check for updates. Grafana recommends upgrading to the latest Grafana version, and this applies to plugins as well.