Hadoop HDFS FSImage

About

Initial dashboard for hadoop in kubernetes (wait what?)

This is mainly a test of hdfs + nn + zk + jn in k8s:

• k8s 1.15.x
• prometheus v2.17.2 via prometheus-operator
• grafana v6.7.3 (a04ef6cefc) with prometheus as Data Source
• Hadoop 3.1.3 in HA setup ( zookeeper cluster + 3 journalnodes + 3 namenodes, which means 1 active nn and 2 standby)
• marcelmay/hadoop-hdfs-fsimage-exporter v1.3 as sidecar in namenode pods, version 1.3 required for hadoop 3.x
• ServiceMonitor to scrape metrics - you must add ti on your own.

Known limitations

• You need to create service monitor on your own.
• You may need to adjust dashboard to match your prometheus labels
• Vast majority of metrics is not yet exposed on the graphs, but you can add them on your own

Changelog

• added variable for DS_PROMETHEUS so that json file can be directly added to grafana data directory and solves errors such as Templating init failed Datasource named ${DS_PROMETHEUS} was not found `

• initial release

Contact

• contact author if you have any suggestions etc