Kafka Streams Dashboard

Dashboard

Dashboard for a Kafka Streams application using Prometheus JMX-Exporter running in Kubernetes
Last updated: 2 months ago

Start with Grafana Cloud and the new FREE tier. Includes 10K series Prometheus or Graphite Metrics and 50gb Loki Logs

Downloads: 41

Reviews: 0

  • Screenshot 2021-02-23 102152.png
    Screenshot 2021-02-23 102152.png
  • Screenshot 2021-02-25 091107.png
    Screenshot 2021-02-25 091107.png

A Dashboard for Kafka Streams Applications running in Kubernetes.

Setup

Option A:

  1. Deploy a Kafka Streams App with Kubernetes. Make sure that the App exports JMX metrics (example is from sbt):
javaOptions in Universal += Seq(
  "-Dcom.sun.management.jmxremote.port=9186",
  "-Dcom.sun.management.jmxremote.rmi.port=9186",
  "-Dcom.sun.management.jmxremote.ssl=false",
  "-Dcom.sun.management.jmxremote.local.only=false",
  "-Dcom.sun.management.jmxremote.authenticate=false"
).mkString(" ")
  1. Start a sidecar prometheus exporter. You can use sscalling/jmx-prometheus-exporter.
  2. Check if prometheus is scraping your application. You can test this by checking the query result of kafka_streams_kafka_metrics_count_count.
  3. If prometheus is scraping correctly, the dashboard should work.

Option B:

  1. Deploy your application with the prometheus-jmx-exporter as java agent (see here
  2. No need for an additional sidecar container, your app exports prometheus metrics directly.
  3. Check if prometheus is scraping your application. You can test this by checking the query result of kafka_streams_kafka_metrics_count_count.
  4. If prometheus is scraping correctly, the dashboard should work.

Kubernetes Config

To automatically add your application to prometheus, use the following annotations in a kubernetes service:

annotations:
  prometheus.io/scrape: 'true'
  prometheus.io/port: '5556'
  prometheus.io/path: "/metrics"
Dependencies: