Monitor Apache Spark easily with Grafana
Easily monitor Apache Spark, a unified analytics engine for large-scale data processing, with Grafana Cloud’s out-of-the-box monitoring solution. The Grafana Cloud forever-free tier includes 3 users and up to 10k metrics series to support your monitoring needs.
Key metrics included
metrics_master_workers_Number
metrics_spark_app_driver_BlockManager_disk_diskSpaceUsed_MB_Number
metrics_spark_app_driver_BlockManager_memory_maxMem_MB_Number
metrics_spark_app_driver_BlockManager_memory_maxOffHeapMem_MB_Number
metrics_spark_app_driver_BlockManager_memory_maxOnHeapMem_MB_Number
metrics_spark_app_driver_DAGScheduler_job_activeJobs_Number
metrics_spark_app_driver_DAGScheduler_job_allJobs_Number
metrics_spark_app_driver_DAGScheduler_messageProcessingTime_Max
metrics_spark_app_driver_DAGScheduler_messageProcessingTime_Mean
metrics_spark_app_driver_DAGScheduler_messageProcessingTime_Min
metrics_spark_app_driver_DAGScheduler_messageProcessingTime_StdDev
metrics_spark_app_driver_DAGScheduler_stage_failedStages_Number
metrics_spark_app_driver_DAGScheduler_stage_runningStages_Number
metrics_spark_app_driver_DAGScheduler_stage_waitingStages_Number
metrics_spark_app_driver_LiveListenerBus_listenerProcessingTime_org_apache_spark_HeartbeatReceiver_Count
metrics_spark_app_driver_LiveListenerBus_listenerProcessingTime_org_apache_spark_HeartbeatReceiver_Max
metrics_spark_app_driver_LiveListenerBus_listenerProcessingTime_org_apache_spark_HeartbeatReceiver_Mean
metrics_spark_app_driver_LiveListenerBus_listenerProcessingTime_org_apache_spark_HeartbeatReceiver_Min
metrics_worker_coresFree_Number
metrics_worker_coresUsed_Number
metrics_worker_memFree_MB_Number
metrics_worker_memUsed_MB_Number