Slurm Exporter - Cluster Overview

Global cluster snapshot: node states, CPU/GPU utilization gauges, job totals, per-partition table (CPU allocation, running/pending jobs), and job state history.

Slurm Exporter - Cluster Overview screenshot 1
Slurm Exporter - Cluster Overview screenshot 2
Slurm Exporter - Cluster Overview screenshot 3

Global cluster snapshot: node states, CPU/GPU utilization gauges, job totals, per-partition table (CPU allocation, running/pending jobs), and job state history.

PanelTypeDescription
Total Nodes / Active / Running / Pending / CPU % / Down+DrainstatQuick health indicators
CPU Utilizationgaugeslurm_cpus_alloc / slurm_cpus_total * 100
Node Utilizationgauge(alloc + mix) / total * 100
Jobs Over TimetimeseriesRunning / Pending / Completing trend
Node State BreakdownbargaugeIdle / Mixed / Alloc / Drain / Down / Maint
Node States Over TimetimeseriesStacked node state history
CPUs Over TimetimeseriesAllocated / Idle / Total
PartitionstableCPU Alloc / Total / % / Running Jobs / Pending Jobs per partition
Job States Over TimetimeseriesRunning / Pending / Completing / Failed / Suspended
Cores Requested Over TimetimeseriesCores running vs pending
Collector StatusstatOK/FAIL per collector
Collector DurationtimeseriesScrape duration history

Requirements

  • Prometheus Slurm Exporter, scraped by Prometheus
  • Grafana 12+ with a Prometheus datasource
  • The dashboard uses a $datasource template variable, so it works with any Prometheus datasource name

Getting started

  1. Run the exporter on a host with a working Slurm client + munge setup (slurmctld host, login node or monitoring VM):
docker run -d --name slurm_exporter \
  -p 9341:9341 \
  -v /etc/slurm:/etc/slurm:ro \
  -v /var/run/munge:/var/run/munge:ro \
  -v /etc/munge/munge.key:/etc/munge/munge.key:ro \
  sckyzo/slurm-exporter:latest

Pre-compiled binaries, systemd unit and Kubernetes options are described in the project README.

  1. Add a scrape job to Prometheus:
scrape_configs:
  - job_name: slurm
    static_configs:
      - targets: ['slurm-exporter-host:9341']
  1. Import this dashboard and select your Prometheus datasource.

The full dashboard suite

This dashboard is part of a set of 10 dashboards maintained with the exporter:

#DashboardFocus
01Cluster Overview (this one)Slurm HPC cluster overview: nodes, jobs, CPU/GPU utilization, partition breakdown.
02Jobs & QueueJob states, queue status, top users and accounts by CPU/job usage.
03Node DetailPer-node CPU and memory utilization by partition. Designed to scale from small clusters to 100k+ nodes.
04Cluster Usage StatisticsCluster utilization (CPU/Memory/GPU/Node), fairshare, jobs, users, accounts, scheduler. Based on gnetId 4323.
05Schedulerslurmctld scheduler cycle times, backfill stats, RPC operation breakdown.
06Reservations & LicensesReservation status, timeline, node health per reservation, and software license usage.
07AccountingHPC accounting: user and account resource consumption, FairShare analysis, top consumers, and priority diagnostics.
08Exporter HealthCollector success/failure status, scrape duration, and Slurm binary version info.
09Exporter PerformanceInternal performance dashboard for slurm_exporter: command durations, cache freshness, error rates, and scrape health. Use this to validate optimisations and detect slowdowns before they cause scrape failures.
10All Metrics ReferenceAll metrics exposed by slurm_exporter — reference dashboard for admins/users to copy panels into their own dashboards. GPU panels included (show 0 if no GPUs).

All JSON files, screenshots and Prometheus alert/recording rules are in monitoring/ on GitHub. Issues and contributions welcome.

Source file: 01-slurm-overview.json | UID: slurm-overview

Revisions
RevisionDescriptionCreated

Get this dashboard

Import the dashboard template

or

Download JSON

Datasource
Dependencies