NVIDIA DCGM

Health, reliability, utilization and Kubernetes allocation for NVIDIA data center GPUs. The app reads the metrics NVIDIA's dcgm-exporter publishes from a Prometheus data source you already have in Grafana.

Requirements

• Grafana 12.4 or later.
• A Prometheus data source over a store that holds dcgm-exporter metrics, such as Prometheus, Mimir, Grafana Cloud Metrics, Amazon Managed Service for Prometheus or Azure Monitor managed Prometheus.
• dcgm-exporter 4.8.3 or later is recommended. Older releases work, and the setup check names the panels they leave empty.

Getting started

1. Choose the data source. Go to Administration, then Plugins and data, then Plugins, and open NVIDIA DCGM. Enable the app, choose the Prometheus data source that holds your dcgm-exporter metrics, and select Save settings.

2. Run the setup check. Open NVIDIA DCGM in the navigation menu, then Setup check. It lists the DCGM fields and labels the data source holds, and the panels each missing field leaves empty.

3. Collect the recommended fields. The exporter's default field list leaves out most of the XID, ECC, health and throttling fields that Health and errors reads. The setup check offers the recommended list for the exporter versions it finds, as a CSV and as a GPU Operator ConfigMap with the command that applies it. A custom field list replaces the default one, and the recommended list includes every default field. A metric allowlist or relabeling rule downstream of the exporter, such as the Kubernetes Monitoring chart's allowlist, can still drop the fields it adds.

4. Open Fleet overview.

The app queries as the signed-in user. Viewers need query permission on the data source. If Grafana reaches the store through its secure SOCKS proxy or Private Data Source Connect, set that up on the Prometheus data source. The app's queries go through it.

Collection paths

The app reads the labels each of these produces:

• The NVIDIA GPU Operator with its ServiceMonitor
• Grafana's Kubernetes Monitoring Helm chart
• A self-managed dcgm-exporter on GKE with Google Cloud Managed Service for Prometheus
• Amazon Managed Service for Prometheus
• Azure Monitor managed Prometheus, scraping the exporter with a PodMonitor
• Oracle Kubernetes Engine

GKE's managed DCGM package reports no host, and its fixed field list has no XID, ECC, row remap or throttling fields. For those, run the exporter yourself with Google Cloud Managed Service for Prometheus, as Google describes. Google warns against running both on the same nodes. Turn the managed package off first.

The default scrape job for AKS-managed GPU node pools drops the UUID label. Every page identifies a GPU by it. Scrape the exporter with a PodMonitor instead, as Microsoft describes.

An exporter run with --use-old-namespace labels each GPU uuid in place of UUID. Remove the flag from the exporter. The setup check detects each of these three cases.

Pages

Page	Shows
Fleet overview	Which GPUs need attention, and how busy the fleet is
Health and errors	What is failing, and what to do about it
Kubernetes workloads	Which pods hold the GPUs, and whether they use them
GPU detail	One GPU in depth
Setup check	Which fields and labels the store holds, and the field list to change

Each page checks which fields the store holds. A panel whose field is not collected names the field to add. A panel that does not apply, such as MIG instances on a fleet without MIG, is hidden. XID tables add a description, severity and recommended action to each code from a catalog bundled with the app.

Tables that can list the whole fleet show their first rows and say how many. Open the panel in Explore for the full result.

Kubernetes attribution

Kubernetes workloads and the Allocated GPU idle alert rule read the pod and namespace labels the GPU Operator adds to each GPU series. A standalone dcgm-exporter does not add them, and the page then shows one message in place of its panels. The setup check shows which labels arrive.

Dashboards

Dashboard	Default time range
NVIDIA DCGM Fleet overview	Last 6 hours
NVIDIA DCGM Health and errors	Last 24 hours
NVIDIA DCGM Kubernetes workloads	Last 6 hours
NVIDIA DCGM GPU detail	Last 6 hours

Each dashboard carries the panels of the page with the same name, and each page has an Open as dashboard button. On a dashboard every panel is always shown. Its description names the fields it reads. Fleet overview and Health and errors add panels that count and name the recommended fields with no series. Label overrides from the app's settings do not apply to dashboards or alert rules.

A plugin update overwrites the dashboards. To customize one, save a copy and edit that.

Settings

Setting	Default	Description
Prometheus data source	None	The data source that holds your dcgm-exporter metrics
Label overrides	None	One more label name to read as the host, pod, namespace, container or cluster, for metrics that spell it in a way the app does not already read, such as a host relabeled to node
Exporter pod pattern	.*dcgm-exporter.*	A regular expression matching dcgm-exporter's own pod. That pod is not the workload using the GPU.
Resolution	The collection interval the setup check detects, or 1m	The step every page query uses, as a Prometheus duration of 30s or more

To provision the app, set DCGM_PROMETHEUS_UID in Grafana's environment to the data source's uid and add this to Grafana's provisioning/plugins directory:

apiVersion: 1

apps:
  - type: integrationplumbers-nvidiadcgm-app
    org_id: 1
    disabled: false
    jsonData:
      datasourceUid: ${DCGM_PROMETHEUS_UID}

Time range

Figures marked "now" use each GPU's latest sample from the last five minutes. A GPU that missed a scrape still appears. Figures marked "in range" cover the whole selected range. Event times are accurate to the collection interval. The default is 30s. XIDs from exporters older than 4.8.3 are placed within five minutes. Nothing refreshes automatically until you choose a refresh interval.

Alert rules

The plugin includes the thirteen rules below, checked every minute. Download them from the app's Configuration page, under Alert rules, once a data source is saved.

On Grafana Cloud, download prometheus-rules.yaml. In Alerting, open Alert rules and use Import to Grafana-managed rules: upload the file and choose the Prometheus data source.

On a Grafana you run yourself, download grafana-alert-rules.yaml. It already names the saved data source. Copy it into Grafana's provisioning/alerting directory and restart Grafana. The rules appear in an NVIDIA DCGM folder. The same file is in the plugin's alerting directory, where it reads the data source's uid from the DCGM_PROMETHEUS_UID environment variable instead.

Rules installed either way are provisioned, and cannot be edited in the UI.

Rule	Fires when	For	Severity
GPU near slowdown temperature	The GPU is within 5 °C of its slowdown temperature, or of 85 °C where it does not report one	5m	warning
Critical XID	An XID whose recommended action concerns the GPU or the node, in the last 10 minutes	0s	critical
Application XID	An XID whose recommended action concerns the application, in the last 10 minutes	0s	info
Uncorrectable memory error	The double-bit ECC count rose in the last 10 minutes	0s	critical
Row remap failed	The GPU could not remap a failing memory row	5m	critical
Row remap pending	A memory row remap is waiting for a GPU reset	15m	warning
NVLink errors growing	More than 100 NVLink CRC and replay errors in each 15-minute window	15m	warning
Throttled under load	Thermal and power throttling take more than 5% of the time while utilization is above 50%	15m	warning
Allocated GPU idle	A GPU held by the same pod stays under 1% utilization for 30 minutes	30m	info
GPU stopped reporting	A GPU that reported in the last 6 hours has not reported in the last 5 minutes	5m	warning
GPU health failing	A DCGM health watch reports a failure	5m	critical
Alert inputs not collected	A field another rule needs has no series in the store	1h	info
NVLink recovery error	An NVLink link needed recovery in the last 15 minutes	0s	warning

The 100 errors in NVLink errors growing and the 5% in Throttled under load are defaults to tune. When a host goes down, GPU stopped reporting fires once for each of its GPUs. Group its notifications by hostname to receive one per host. A GPU removed on purpose keeps it firing for six hours, unless you silence it.

Alert inputs not collected also fires for a field the GPUs cannot report, such as ECC counts with ECC disabled. Silence it for that field.

Support

Email helpdesk@integrationplumbers.io.

Use of the plugin is subject to the Integration Plumbers acceptable use policy.