Kafka Cluster Monitoring [Elasticsearch]

Dashboard

A comprehensive Kafka cluster monitoring dashboard with Elasticsearch as the datasource. This dashboard gives real time monitoring in Broker health, consumer group stats, consumer lags and much more.
Last updated: a year ago

Start with Grafana Cloud and the new FREE tier. Includes 10K series Prometheus or Graphite Metrics and 50gb Loki Logs

Downloads: 487

Reviews: 1

  • Screenshot_2020-04-27 Kafka Monitoring - Grafana.png
    Screenshot_2020-04-27 Kafka Monitoring - Grafana.png
  • Screenshot_2020-04-27 Kafka Monitoring - Grafana(1).png
    Screenshot_2020-04-27 Kafka Monitoring - Grafana(1).png
  • Screenshot_2020-04-27 Kafka Monitoring - Grafana(2).png
    Screenshot_2020-04-27 Kafka Monitoring - Grafana(2).png
  • Screenshot_2020-04-27 Kafka Monitoring - Grafana(3).png
    Screenshot_2020-04-27 Kafka Monitoring - Grafana(3).png

A comprehensive Kafka cluster monitoring dashboard with Elasticsearch as the data source. This dashboard gives real time monitoring in Broker health, consumer group stats, consumer lags, errors and much more. The panels are grouped as below metric rows.

  1. Kafka Brokers
  2. Consumers
  3. Topics
  4. Errors

This dashboard uses Elasticsearch 7.6 or higher as the datasource. This dashboard work with metrics collected using metricbeat 7.6 or higher. Elasticsearch introduced the consumer lag collect feature in 7.6 and that's the reason for my recommendation on this version or higher. Configure metricbeat in each Kafka broker instances and send the metrics to an Elasticsearch. Dashboard has been tested with a 3 node Kafka cluster running in AWS ubuntu 18.0LTS.

Collector Configuration Details

This dashboard work with metrics collected using metricbeat 7.6 or higher. Elasticsearch introduced the consumer lag collect feature in 7.6 and that's the reason for my recommendation on this version or higher.