GCP Managed Kafka

Managed Kafka Grafana Dashboard

Overview

This Grafana dashboard is designed to monitor and visualize key metrics of a Managed Kafka cluster running on Google Cloud Platform (GCP) using Google Cloud Monitoring (Stackdriver).

Features

• Broker Health Monitoring: Tracks memory usage, CPU utilization, and request count for Kafka brokers.
• Cluster Performance Metrics: Displays partition statuses, offline partitions, and long-term storage usage.
• Traffic & Throughput Insights: Provides per-topic byte in/out and request count metrics.
• Latency Monitoring: Tracks Kafka request latencies (P50, P99, P999).
• Consumer Metrics: Monitors consumer lag, stable/rebalancing consumer groups, and dead/empty consumer groups.
• Custom Alerts & Annotations: Built-in support for monitoring annotations and alerts.
• Kafka UI Quick Access: Direct link to the Kafka UI for deeper analysis.

Data Source

The dashboard uses Google Cloud Monitoring (Stackdriver) as the primary data source, configured via:

• Datasource: ${DS_GCP}
• Project: $cluster

Panels

The dashboard includes the following panels:

1. Memory Usage (Stat Panel)
2. CPU Usage (Stat Panel)
3. Request Count Usage (Gauge Panel)
4. Offline Partitions (Stat Panel)
5. Total Partitions Per Broker (Stat Panel)
6. Long-Term Storage Usage (Stat Panel)
7. Byte In/Out Per Topic (Stat Panels)
8. Topic Request Count & Error Count (Stat Panels)
9. Messages Per Topic (Time Series Panel)
10. Request Latency Per Broker (P50, P99, P999) (Time Series Panels)
11. Consumer Lag Per Topic (Time Series Panel)
12. Stable, Rebalancing, Dead, and Empty Consumer Groups (Stat Panels)

Refresh & Time Settings

• Refresh Rate: Every 30 seconds
• Time Range: Last 1 hour

Usage Instructions

1. Ensure Data Source is Configured: Update the ${DS_GCP} variable to match your GCP project.
2. Modify Thresholds if Needed: Adjust alert thresholds for memory, CPU, and consumer lag based on your operational requirements.
3. Use Kafka UI for Deep Dive: Click on the Kafka UI link to access additional insights.
4. Set Up Alerts: Utilize Grafana alerting rules to notify on critical metrics.

Tags

• Cloud Monitoring
• Kafka
• GCP

Dashboard UID

• aeff2g49qgc8wa

Notes

• Make sure the correct GCP project permissions are granted to access Kafka metrics.
• Use Grafana Variables ($cluster) for multi-cluster monitoring support.
• For further customization, modify the PromQL queries in the panel configurations.

Created with Grafana v11.2.0