Slurm Exporter - Node Detail

Per-node CPU and memory utilization. Designed to scale from small clusters to 100k+ nodes via a `$partition` filter variable.

Slurm Exporter - Node Detail screenshot 1
Slurm Exporter - Node Detail screenshot 2
Slurm Exporter - Node Detail screenshot 3

Per-node CPU and memory utilization. Designed to scale from small clusters to 100k+ nodes via a $partition filter variable.

PanelTypeDescription
Total / Idle / Mixed / Alloc / Down / DrainstatCluster-wide node counts
Node States per PartitionbarchartStacked distribution by partition
Partition SummarytableAlloc / Idle / Mix / Down / Drain per partition
Down & Drain NodestableAlways scalable — only shows degraded nodes
Node CPU & Memory DetailtablePer-node: CPU alloc/total/%, Mem alloc/total/% — filtered by $partition
CPU Allocated per Node (top 10)timeseriesMost loaded nodes over time
Memory Allocated per Node (top 10)timeseriesTop memory consumers

Scale note: The per-node detail table is filtered by the $partition variable. On clusters with 100k+ nodes, always select a specific partition to limit results. The partition summary and "Down & Drain Nodes" panels are always O(partitions).

Requirements

  • Prometheus Slurm Exporter, scraped by Prometheus
  • Grafana 12+ with a Prometheus datasource
  • The dashboard uses a $datasource template variable, so it works with any Prometheus datasource name

Getting started

  1. Run the exporter on a host with a working Slurm client + munge setup (slurmctld host, login node or monitoring VM):
docker run -d --name slurm_exporter \
  -p 9341:9341 \
  -v /etc/slurm:/etc/slurm:ro \
  -v /var/run/munge:/var/run/munge:ro \
  -v /etc/munge/munge.key:/etc/munge/munge.key:ro \
  sckyzo/slurm-exporter:latest

Pre-compiled binaries, systemd unit and Kubernetes options are described in the project README.

  1. Add a scrape job to Prometheus:
scrape_configs:
  - job_name: slurm
    static_configs:
      - targets: ['slurm-exporter-host:9341']
  1. Import this dashboard and select your Prometheus datasource.

The full dashboard suite

This dashboard is part of a set of 10 dashboards maintained with the exporter:

#DashboardFocus
01Cluster OverviewSlurm HPC cluster overview: nodes, jobs, CPU/GPU utilization, partition breakdown.
02Jobs & QueueJob states, queue status, top users and accounts by CPU/job usage.
03Node Detail (this one)Per-node CPU and memory utilization by partition. Designed to scale from small clusters to 100k+ nodes.
04Cluster Usage StatisticsCluster utilization (CPU/Memory/GPU/Node), fairshare, jobs, users, accounts, scheduler. Based on gnetId 4323.
05Schedulerslurmctld scheduler cycle times, backfill stats, RPC operation breakdown.
06Reservations & LicensesReservation status, timeline, node health per reservation, and software license usage.
07AccountingHPC accounting: user and account resource consumption, FairShare analysis, top consumers, and priority diagnostics.
08Exporter HealthCollector success/failure status, scrape duration, and Slurm binary version info.
09Exporter PerformanceInternal performance dashboard for slurm_exporter: command durations, cache freshness, error rates, and scrape health. Use this to validate optimisations and detect slowdowns before they cause scrape failures.
10All Metrics ReferenceAll metrics exposed by slurm_exporter — reference dashboard for admins/users to copy panels into their own dashboards. GPU panels included (show 0 if no GPUs).

All JSON files, screenshots and Prometheus alert/recording rules are in monitoring/ on GitHub. Issues and contributions welcome.

Source file: 03-slurm-nodes.json | UID: slurm-nodes

Revisions
RevisionDescriptionCreated
Linux Server

Linux Server

by Grafana Labs
Grafana Labs solution

Monitor Linux with Grafana. Easily monitor your Linux deployment with Grafana Cloud's out-of-the-box monitoring solution.

Learn more

Get this dashboard

Import the dashboard template

or

Download JSON

Datasource
Dependencies