Kafka Cluster Monitoring [Elasticsearch]

A comprehensive Kafka cluster monitoring dashboard with Elasticsearch as the datasource. This dashboard gives real time monitoring in Broker health, consumer group stats, consumer lags and much more.

Overview
Revisions
Reviews
Edit →

A comprehensive Kafka cluster monitoring dashboard with Elasticsearch as the data source. This dashboard gives real time monitoring in Broker health, consumer group stats, consumer lags, errors and much more. The panels are grouped as below metric rows.

Kafka Brokers
Consumers
Topics
Errors

This dashboard uses Elasticsearch 7.6 or higher as the datasource. This dashboard work with metrics collected using metricbeat 7.6 or higher. Elasticsearch introduced the consumer lag collect feature in 7.6 and that’s the reason for my recommendation on this version or higher. Configure metricbeat in each Kafka broker instances and send the metrics to an Elasticsearch. Dashboard has been tested with a 3 node Kafka cluster running in AWS ubuntu 18.0LTS.