Kafka Cluster Monitoring [Elasticsearch]

Dashboard

A comprehensive Kafka cluster monitoring dashboard with Elasticsearch as the datasource. This dashboard gives real time monitoring in Broker health, consumer group stats, consumer lags and much more.
Last updated: 4 months ago

Downloads: 187

Reviews: 1

  • Screenshot_2020-04-27 Kafka Monitoring - Grafana.png
    Screenshot_2020-04-27 Kafka Monitoring - Grafana.png
  • Screenshot_2020-04-27 Kafka Monitoring - Grafana(1).png
    Screenshot_2020-04-27 Kafka Monitoring - Grafana(1).png
  • Screenshot_2020-04-27 Kafka Monitoring - Grafana(2).png
    Screenshot_2020-04-27 Kafka Monitoring - Grafana(2).png
  • Screenshot_2020-04-27 Kafka Monitoring - Grafana(3).png
    Screenshot_2020-04-27 Kafka Monitoring - Grafana(3).png

A comprehensive Kafka cluster monitoring dashboard with Elasticsearch as the data source. This dashboard gives real time monitoring in Broker health, consumer group stats, consumer lags, errors and much more. The panels are grouped as below metric rows.

  1. Kafka Brokers
  2. Consumers
  3. Topics
  4. Errors

This dashboard uses Elasticsearch 7.6 or higher as the datasource. This dashboard work with metrics collected using metricbeat 7.6 or higher. Elasticsearch introduced the consumer lag collect feature in 7.6 and that's the reason for my recommendation on this version or higher. Configure metricbeat in each Kafka broker instances and send the metrics to an Elasticsearch. Dashboard has been tested with a 3 node Kafka cluster running in AWS ubuntu 18.0LTS.

Collector Configuration Details

This dashboard work with metrics collected using metricbeat 7.6 or higher. Elasticsearch introduced the consumer lag collect feature in 7.6 and that's the reason for my recommendation on this version or higher.