--- title: "NVIDIA DCGM app plugin for Grafana" description: "GPU health, reliability and utilization from NVIDIA DCGM metrics in Prometheus. NVIDIA DCGM app plugin for Grafana by integrationplumbers." --- > This is the Markdown version of https://grafana.com/grafana/plugins/integrationplumbers-nvidiadcgm-app/ # NVIDIA DCGM - Developer: integrationplumbers - Version: 1.0.0 - [Change log](https://grafana.com/grafana/plugins/integrationplumbers-nvidiadcgm-app/changelog.md) ![Fleet overview](https://grafana.com/api/plugins/integrationplumbers-nvidiadcgm-app/versions/1.0.0/images/img/screenshots/fleet-overview.png) ![Health and errors](https://grafana.com/api/plugins/integrationplumbers-nvidiadcgm-app/versions/1.0.0/images/img/screenshots/health-and-errors.png) ![Kubernetes workloads](https://grafana.com/api/plugins/integrationplumbers-nvidiadcgm-app/versions/1.0.0/images/img/screenshots/kubernetes-workloads.png) ![GPU detail](https://grafana.com/api/plugins/integrationplumbers-nvidiadcgm-app/versions/1.0.0/images/img/screenshots/gpu-detail.png) ![Setup check](https://grafana.com/api/plugins/integrationplumbers-nvidiadcgm-app/versions/1.0.0/images/img/screenshots/setup-check.png) Health, reliability, utilization and Kubernetes allocation for NVIDIA data center GPUs. The app reads the metrics NVIDIA's dcgm-exporter publishes from a Prometheus data source you already have in Grafana. ## Requirements - Grafana 12.4 or later. - A Prometheus data source over a store that holds dcgm-exporter metrics, such as Prometheus, Mimir, Grafana Cloud Metrics, Amazon Managed Service for Prometheus or Azure Monitor managed Prometheus. - dcgm-exporter 4.8.3 or later is recommended. Older releases work, and the setup check names the panels they leave empty. ## Getting started 1. **Choose the data source.** Go to **Administration**, then **Plugins and data**, then **Plugins**, and open **NVIDIA DCGM**. Enable the app, choose the Prometheus data source that holds your dcgm-exporter metrics, and select **Save settings**. 2. **Run the setup check.** Open **NVIDIA DCGM** in the navigation menu, then **Setup check**. It lists the DCGM fields and labels the data source holds, and the panels each missing field leaves empty. 3. **Collect the recommended fields.** The exporter's default field list leaves out most of the XID, ECC, health and throttling fields that Health and errors reads. The setup check offers the recommended list for the exporter versions it finds, as a CSV and as a GPU Operator ConfigMap with the command that applies it. A custom field list replaces the default one, and the recommended list includes every default field. A metric allowlist or relabeling rule downstream of the exporter, such as the Kubernetes Monitoring chart's allowlist, can still drop the fields it adds. 4. **Open Fleet overview.** The app queries as the signed-in user. Viewers need query permission on the data source. If Grafana reaches the store through its secure SOCKS proxy or [Private Data Source Connect](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/), set that up on the Prometheus data source. The app's queries go through it. ## Collection paths The app reads the labels each of these produces: - The NVIDIA GPU Operator with its ServiceMonitor - Grafana's Kubernetes Monitoring Helm chart - A self-managed dcgm-exporter on GKE with Google Cloud Managed Service for Prometheus - Amazon Managed Service for Prometheus - Azure Monitor managed Prometheus, scraping the exporter with a PodMonitor - Oracle Kubernetes Engine GKE's managed DCGM package reports no host, and its fixed field list has no XID, ECC, row remap or throttling fields. For those, run the exporter yourself with Google Cloud Managed Service for Prometheus, as [Google describes](https://docs.cloud.google.com/stackdriver/docs/managed-prometheus/exporters/nvidia-dcgm). Google warns against running both on the same nodes. Turn the [managed package](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/dcgm-metrics) off first. The default scrape job for AKS-managed GPU node pools drops the `UUID` label. Every page identifies a GPU by it. Scrape the exporter with a PodMonitor instead, as [Microsoft describes](https://learn.microsoft.com/azure/azure-monitor/containers/prometheus-dcgm-integration). An exporter run with `--use-old-namespace` labels each GPU `uuid` in place of `UUID`. Remove the flag from the exporter. The setup check detects each of these three cases. ## Pages | Page | Shows | | -------------------- | --------------------------------------------------------------------- | | Fleet overview | Which GPUs need attention, and how busy the fleet is | | Health and errors | What is failing, and what to do about it | | Kubernetes workloads | Which pods hold the GPUs, and whether they use them | | GPU detail | One GPU in depth | | Setup check | Which fields and labels the store holds, and the field list to change | Each page checks which fields the store holds. A panel whose field is not collected names the field to add. A panel that does not apply, such as MIG instances on a fleet without MIG, is hidden. XID tables add a description, severity and recommended action to each code from a catalog bundled with the app. Tables that can list the whole fleet show their first rows and say how many. Open the panel in **Explore** for the full result. ### Kubernetes attribution Kubernetes workloads and the Allocated GPU idle alert rule read the pod and namespace labels the GPU Operator adds to each GPU series. A standalone dcgm-exporter does not add them, and the page then shows one message in place of its panels. The setup check shows which labels arrive. ## Dashboards | Dashboard | Default time range | | -------------------------------- | ------------------ | | NVIDIA DCGM Fleet overview | Last 6 hours | | NVIDIA DCGM Health and errors | Last 24 hours | | NVIDIA DCGM Kubernetes workloads | Last 6 hours | | NVIDIA DCGM GPU detail | Last 6 hours | Each dashboard carries the panels of the page with the same name, and each page has an **Open as dashboard** button. On a dashboard every panel is always shown. Its description names the fields it reads. Fleet overview and Health and errors add panels that count and name the recommended fields with no series. Label overrides from the app's settings do not apply to dashboards or alert rules. A plugin update overwrites the dashboards. To customize one, save a copy and edit that. ## Settings | Setting | Default | Description | | ---------------------- | ------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Prometheus data source | None | The data source that holds your dcgm-exporter metrics | | Label overrides | None | One more label name to read as the host, pod, namespace, container or cluster, for metrics that spell it in a way the app does not already read, such as a host relabeled to `node` | | Exporter pod pattern | `.*dcgm-exporter.*` | A regular expression matching dcgm-exporter's own pod. That pod is not the workload using the GPU. | | Resolution | The collection interval the setup check detects, or 1m | The step every page query uses, as a Prometheus duration of 30s or more | To provision the app, set `DCGM_PROMETHEUS_UID` in Grafana's environment to the data source's uid and add this to Grafana's `provisioning/plugins` directory: ```yaml apiVersion: 1 apps: - type: integrationplumbers-nvidiadcgm-app org_id: 1 disabled: false jsonData: datasourceUid: ${DCGM_PROMETHEUS_UID} ``` ## Time range Figures marked "now" use each GPU's latest sample from the last five minutes. A GPU that missed a scrape still appears. Figures marked "in range" cover the whole selected range. Event times are accurate to the collection interval. The default is 30s. XIDs from exporters older than 4.8.3 are placed within five minutes. Nothing refreshes automatically until you choose a refresh interval. ## Alert rules The plugin includes the thirteen rules below, checked every minute. Download them from the app's **Configuration** page, under **Alert rules**, once a data source is saved. On Grafana Cloud, download `prometheus-rules.yaml`. In **Alerting**, open **Alert rules** and use **Import to Grafana-managed rules**: upload the file and choose the Prometheus data source. On a Grafana you run yourself, download `grafana-alert-rules.yaml`. It already names the saved data source. Copy it into Grafana's `provisioning/alerting` directory and restart Grafana. The rules appear in an **NVIDIA DCGM** folder. The same file is in the plugin's `alerting` directory, where it reads the data source's uid from the `DCGM_PROMETHEUS_UID` environment variable instead. Rules installed either way are provisioned, and cannot be edited in the UI. | Rule | Fires when | For | Severity | | ----------------------------- | -------------------------------------------------------------------------------------------- | --- | -------- | | GPU near slowdown temperature | The GPU is within 5 °C of its slowdown temperature, or of 85 °C where it does not report one | 5m | warning | | Critical XID | An XID whose recommended action concerns the GPU or the node, in the last 10 minutes | 0s | critical | | Application XID | An XID whose recommended action concerns the application, in the last 10 minutes | 0s | info | | Uncorrectable memory error | The double-bit ECC count rose in the last 10 minutes | 0s | critical | | Row remap failed | The GPU could not remap a failing memory row | 5m | critical | | Row remap pending | A memory row remap is waiting for a GPU reset | 15m | warning | | NVLink errors growing | More than 100 NVLink CRC and replay errors in each 15-minute window | 15m | warning | | Throttled under load | Thermal and power throttling take more than 5% of the time while utilization is above 50% | 15m | warning | | Allocated GPU idle | A GPU held by the same pod stays under 1% utilization for 30 minutes | 30m | info | | GPU stopped reporting | A GPU that reported in the last 6 hours has not reported in the last 5 minutes | 5m | warning | | GPU health failing | A DCGM health watch reports a failure | 5m | critical | | Alert inputs not collected | A field another rule needs has no series in the store | 1h | info | | NVLink recovery error | An NVLink link needed recovery in the last 15 minutes | 0s | warning | The 100 errors in NVLink errors growing and the 5% in Throttled under load are defaults to tune. When a host goes down, GPU stopped reporting fires once for each of its GPUs. Group its notifications by `hostname` to receive one per host. A GPU removed on purpose keeps it firing for six hours, unless you silence it. Alert inputs not collected also fires for a field the GPUs cannot report, such as ECC counts with ECC disabled. Silence it for that field. ## Support Email . Use of the plugin is subject to the Integration Plumbers [acceptable use policy](https://www.integrationplumbers.io/legal/acceptable-use).