--- title: "Kafka data source plugin for Grafana" description: "Kafka Datasource Plugin. Kafka data source plugin for Grafana by hamedkarbasi93. Available in Grafana Cloud and self-managed Grafana." --- > This is the Markdown version of https://grafana.com/grafana/plugins/hamedkarbasi93-kafka-datasource/ # Kafka - Developer: hamedkarbasi93 - Version: 1.10.0 - [Change log](https://grafana.com/grafana/plugins/hamedkarbasi93-kafka-datasource/changelog.md) ![Streaming Dashboard](https://grafana.com/api/plugins/hamedkarbasi93-kafka-datasource/versions/1.10.0/images/img/stream-json-screenshot.png) ![Datasource Configuration](https://grafana.com/api/plugins/hamedkarbasi93-kafka-datasource/versions/1.10.0/images/img/config-basic.png) ![JSON Query Builder](https://grafana.com/api/plugins/hamedkarbasi93-kafka-datasource/versions/1.10.0/images/img/query-json.png) ![Avro Inline Schema](https://grafana.com/api/plugins/hamedkarbasi93-kafka-datasource/versions/1.10.0/images/img/query-avro-inline-schema.png) ![SASL and TLS Authentication](https://grafana.com/api/plugins/hamedkarbasi93-kafka-datasource/versions/1.10.0/images/img/config-authentication.png) # Kafka Datasource for Grafana [![License](https://img.shields.io/github/license/hoptical/grafana-kafka-datasource)](https://grafana.com/grafana/plugins/hamedkarbasi93-kafka-datasource/) [![CI](https://github.com/hoptical/grafana-kafka-datasource/actions/workflows/ci.yml/badge.svg)](https://github.com/hoptical/grafana-kafka-datasource/actions/workflows/ci.yml) [![Release](https://github.com/hoptical/grafana-kafka-datasource/actions/workflows/release.yml/badge.svg)](https://github.com/hoptical/grafana-kafka-datasource/actions/workflows/release.yml) [![Go Version](https://img.shields.io/badge/go-1.26.5-blue?logo=go)](https://golang.org/doc/go1.26) [![Grafana v10.2+](https://img.shields.io/badge/grafana-10.2%2B-orange?logo=grafana)](https://grafana.com/) --- **Visualize real-time Kafka data in Grafana dashboards.** --- ## Why Kafka Datasource? - **Live streaming:** Monitor Kafka topics in real time. - **Flexible queries:** Select topics, partitions, offsets, and timestamp modes. - **Rich JSON support:** Handles flat, nested, and array data. - **Avro support:** Integrates with Schema Registry for Avro messages. - **Plaintext support:** Reads raw Kafka payload bytes without schema decoding. - **Line Protocol support:** Parses InfluxDB Line Protocol messages into a single long-format Grafana frame per message, with tags carried as columns. - **Secure:** SASL authentication (PLAIN, SCRAM, OAUTHBEARER/OIDC, GSSAPI/Kerberos) & SSL/TLS encryption. - **Easy setup:** Install and configure in minutes. ## How It Works This plugin connects your Grafana instance directly to Kafka brokers, allowing you to query, visualize, and explore streaming data with powerful time-series panels and dashboards. ## Requirements - Apache Kafka v0.11+ - Grafana v10.2+ > Note: Without PDC, the Grafana server must access the Kafka broker and Schema Registry directly. With PDC enabled, Grafana must access the PDC proxy, and the PDC agent must access the Kafka broker and Schema Registry. ## Features - Real-time monitoring of Kafka topics - Kafka authentication (SASL: PLAIN, SCRAM, OAUTHBEARER/OIDC client credentials, GSSAPI/Kerberos) & encryption (SSL/TLS) - Query all or specific partitions - Autocomplete for topic names - Flexible offset options (latest, last N, earliest) - Timestamp modes (Kafka event time, dashboard received time) - Advanced JSON support (flat, nested, arrays, mixed types) - Avro support with Schema Registry integration (inline schema or Schema Registry) - Protobuf support with Schema Registry integration (inline schema or Schema Registry) - Plaintext support for raw byte payloads (no schema required) - Line Protocol support (InfluxDB format) with per-query timestamp-precision selection (auto / ns / µs / ms / s) - Transactional topic support (committed messages only; control records are filtered) - Message key support (None, String, JSON, Base64 formats) - Configurable flattening depth (default: 5) - Configurable max fields per message (default: 1000) - Customizable query aliases with placeholders - Grafana Alerting on a snapshot of recent topic messages ## Installation ### Via grafana-cli ```bash grafana-cli plugins install hamedkarbasi93-kafka-datasource ``` ### Via zip file Download the [latest release](https://github.com/hoptical/grafana-kafka-datasource/releases/latest) and unpack it into your Grafana plugins directory (default: `/var/lib/grafana/plugins`). ### Provisioning You can automatically configure the Kafka datasource using Grafana's provisioning feature. For a ready-to-use template and configuration options, refer to `provisioning/datasources/datasource.yaml` in this repository. ## Usage ### Configuration 1. Add a new data source in Grafana and select "Kafka Datasource". 2. Configure connection settings: - **Broker address** (e.g. `localhost:9094` or `kafka:9092`) - **Private Data Source Connect (PDC)** (optional; requires PDC to be configured in Grafana) - **Authentication** (SASL — PLAIN, SCRAM, OAUTHBEARER/OIDC client credentials, or GSSAPI/Kerberos — SSL/TLS, optional) - **Avro Schema Registry** (if using Avro format) - **Timeout settings** (default: two seconds) [![Kafka basic datasource configuration](https://raw.githubusercontent.com/hoptical/grafana-kafka-datasource/ba3d10342aca7512c5679fcd7761e5b394094598/src/img/config-basic.png)](https://raw.githubusercontent.com/hoptical/grafana-kafka-datasource/ba3d10342aca7512c5679fcd7761e5b394094598/src/img/config-basic.png) ### Build the Query 1. Create a new dashboard panel in Grafana. 2. Select your Kafka data source. 3. Configure the query: - **Topic**: Enter or select your Kafka topic (autocomplete available). - **Fetch Partitions**: Click to retrieve available partitions. - **Partition**: Choose a specific partition or "all" for all partitions. - **Message Format**: - `JSON`: For JSON messages - `Avro`: For Avro messages (requires schema registry or inline schema) - `Protobuf`: For Protobuf messages (requires schema registry or inline schema) - `Plaintext`: For raw payload bytes (no schema required) - `Line Protocol`: For InfluxDB Line Protocol messages. All lines in a Kafka message are combined into the rows of a single long-format frame; tags become columns; field types (float/int/uint/bool/string) are preserved per spec. A **Timestamp Precision** selector (`Auto-detect` by default) controls how inline timestamps are interpreted. - **Offset Reset**: - `latest`: Only new messages - `last N messages`: Start from the most recent N messages (set N in the UI) - `earliest`: Start from the oldest message - **Transactional Topics**: The datasource reads committed messages and automatically skips Kafka transaction control records (COMMIT/ABORT markers). - **Timestamp Mode**: Choose between Kafka event time or dashboard received time. - **Alias**: Optional custom name for the query series. Supports template placeholders: - `{{topic}}`: The Kafka topic name - `{{field}}`: The field name (for field display names) - `{{partition}}`: The partition number - `{{refid}}`: The query RefID (e.g., A, B) - **Key Format**: Choose how to process Kafka message keys: - `None`: Ignore keys (default, backward compatible) - `String`: Single "key" column with UTF-8 decoded key - `JSON`: Flatten JSON keys with "key." prefix (e.g., key.userId, key.region) - `Base64`: Single "key" column with raw binary key encoded as a base64 string **Tip:** Numeric fields become time series, string fields are labels, arrays and nested objects are automatically flattened for visualization. ![JSON query editor example](https://raw.githubusercontent.com/hoptical/grafana-kafka-datasource/ba3d10342aca7512c5679fcd7761e5b394094598/src/img/query-json.png) ### Grafana Alerting Dashboard panels still stream over Grafana Live. Alert rules cannot subscribe to Live channels, so this plugin evaluates alerts from messages inside the rule's time range. QueryData resolves the partition offsets selected by the rule's time range (`ListOffsets` by timestamp). It returns the freshest bounded tail: up to 1000 messages across all selected partitions, with at most eight concurrent partition readers. Keeping the newest messages makes Reduce → Last evaluate the freshest available value instead of a value from the beginning of a busy range. The 1000-message cap is global and is divided across selected partitions. When a range contains more messages than that cap, Reduce → Min, Max, Mean, and Count describe only the returned tail, not the entire time range. This bound keeps alert evaluation memory, CPU, and broker reads predictable. A producer with no messages in the range evaluates as **No data**. If the query deadline arrives before the selected end offset, the evaluation is **Error** instead of a partial snapshot. Latest, last N, and earliest still control dashboard streaming. They do not define the alert window. Topics written by transactional (exactly-once) producers are supported. The snapshot reads committed data up to the partition's last stable offset, so an open transaction or a trailing COMMIT/ABORT marker does not stall the evaluation. Line protocol is not supported for alerting yet. JSON, Avro, Protobuf, and plaintext return **wide time series**. String fields become series labels, and Kafka `offset`/`partition` columns are removed, so Reduce does not fail with `must be a wide series but got type long`. Set **Alert field** (`selectedField`) to the numeric payload field (for example `value1` or `metrics.cpu.load`), then Reduce → Last. That field is shown only in the alerting query editor. Dashboard Live streaming is unchanged. ## Supported JSON Structures - Flat objects - Nested objects (flattened) - Top-level arrays - Mixed types **Examples:** Simple flat object: ```json { "temperature": 23.5, "humidity": 65.2, "status": "active" } ``` Nested object (flattened as `user.name`, `user.age`, `settings.theme`): ```json { "user": { "name": "John Doe", "age": 30 }, "settings": { "theme": "dark" } } ``` Top-level array (flattened as `item_0.id`, `item_0.value`, `item_1.id`, etc.): ```json [ { "id": 1, "value": 10.5 }, { "id": 2, "value": 20.3 } ] ``` ## Avro Support Avro messages are supported using either: - **Inline schema**: paste your Avro schema (JSON format) in the UI - **Schema Registry**: the plugin fetches the latest schema by subject from the configured Schema Registry ![Avro inline schema editor](https://raw.githubusercontent.com/hoptical/grafana-kafka-datasource/ba3d10342aca7512c5679fcd7761e5b394094598/src/img/query-avro-inline-schema.png) **Example Avro schema:** ```json { "type": "record", "name": "SensorReading", "fields": [ { "name": "sensor_id", "type": "string" }, { "name": "temperature", "type": "double" }, { "name": "timestamp", "type": "long" } ] } ``` Both flat and nested structures are supported. When using Schema Registry, messages are encoded in Confluent wire format with schema ID prefix for efficient deserialization. ## Protobuf Support Protobuf messages are supported using either: - **Inline schema**: paste your `.proto` schema definition in the UI - **Schema Registry**: the plugin extracts the schema ID from the Confluent wire format header and fetches the schema from the configured Schema Registry **Example Protobuf schema:** ```protobuf syntax = "proto3"; message SensorReading { string sensor_id = 1; double temperature = 2; int64 timestamp = 3; } ``` Both flat and nested structures are supported. Confluent wire format uses a magic byte (0x00) followed by a 4-byte schema ID, then a variable-length message-indexes array (varint-encoded), and finally the Protobuf payload. The message-indexes identify which message type in the schema the payload encodes. ### Known Protobuf Limitations - **Inline schemas do not support imports** - Well-known types like `google.protobuf.Timestamp` won't compile when provided inline - Use inline schema for simple schemas, or provide the schema via Schema Registry if imports are needed - **Default message selection**: the first top-level message in the `.proto` file is used as the default - **Schema caching**: schemas are cached by ID/subject to avoid repeated registry requests If you encounter issues with complex Protobuf schemas, open an issue so we can prioritize improvements. ## Live Demo ![Kafka dashboard live demo](https://raw.githubusercontent.com/hoptical/grafana-kafka-datasource/ba3d10342aca7512c5679fcd7761e5b394094598/src/img/graph.gif) More screenshots and recordings (click to open): - [Streaming dashboard (high-resolution screenshot)](https://raw.githubusercontent.com/hoptical/grafana-kafka-datasource/ba3d10342aca7512c5679fcd7761e5b394094598/src/img/stream-json-screenshot.png) - [Authentication and TLS configuration](https://raw.githubusercontent.com/hoptical/grafana-kafka-datasource/ba3d10342aca7512c5679fcd7761e5b394094598/src/img/config-authentication.png) ## Sample Data Generator Want to test the plugin with realistic Kafka messages? Use the included sample producers to generate JSON, Avro, or Protobuf messages with various structures and schema configurations. For detailed usage, see the [example README](https://github.com/hoptical/grafana-kafka-datasource/blob/main/example/README.md). To test Plaintext mode, produce any bytes (for example JSON bytes) and select `Plaintext` in the query editor to view raw payload text. ## Performance Benchmarks Production builds always use the optimized path. The old pre-fix behavior is still available for measurement through dedicated benchmark variants instead of runtime environment variables. Useful benchmark pairs: - `BenchmarkDecodeAvroMessage` vs `BenchmarkDecodeAvroMessage_NoCache` - `BenchmarkDecodeProtobufMessage_Plain` vs `BenchmarkDecodeProtobufMessage_Plain_NoCache` - `BenchmarkProcessMessage_JSON_Wide100` vs `BenchmarkProcessMessage_JSON_Wide100_FieldOrderCacheDisabled` - `BenchmarkWorkflow` vs `BenchmarkWorkflow_NoOptimizations` Example commands: ```bash # Run all focused performance benchmarks go test -run '^$' -bench 'BenchmarkDecodeAvroMessage|BenchmarkDecodeProtobufMessage_Plain|BenchmarkProcessMessage_JSON_Wide100|BenchmarkWorkflow' -benchmem ./pkg/kafka_client/... ./pkg/plugin/... # Compare cached Avro decode vs the explicit no-cache benchmark path go test -run '^$' -bench '^BenchmarkDecodeAvroMessage$' -benchmem -count=6 ./pkg/kafka_client/... > after.txt go test -run '^$' -bench '^BenchmarkDecodeAvroMessage_NoCache$' -benchmem -count=6 ./pkg/kafka_client/... > before.txt sed 's/BenchmarkDecodeAvroMessage_NoCache/BenchmarkDecodeAvroMessage/' before.txt > before.norm.txt benchstat before.norm.txt after.txt ``` The Avro and Protobuf schema caches remain bounded with a hardcoded default size of `256` entries. For benchmark methodology, before/after numbers, profiling commands, and architecture notes, see [Performance Optimizations](https://github.com/hoptical/grafana-kafka-datasource/blob/main/docs/PERFORMANCE_OPTIMIZATIONS.md). ## FAQ & Troubleshooting - **Can I use this with any Kafka broker?** Yes, supports Apache Kafka v0.11+ and compatible brokers. - **Does it support secure connections?** Yes, SASL (PLAIN, SCRAM-SHA-256/512, OAUTHBEARER/OIDC client credentials, GSSAPI/Kerberos) and SSL/TLS are supported. See [OAuth 2.0 Authentication](https://github.com/hoptical/grafana-kafka-datasource/blob/main/docs/OAUTH2_AUTH.md) for OAUTHBEARER setup, and [Kerberos Authentication](https://github.com/hoptical/grafana-kafka-datasource/blob/main/docs/KERBEROS_AUTH.md) for GSSAPI setup. - **What JSON formats are supported?** Flat, nested, arrays, mixed types. - **What is Plaintext format?** It bypasses schema decoding and renders raw payload bytes in a single `message` field. - **What is Line Protocol format?** It parses [InfluxDB Line Protocol](https://docs.influxdata.com/influxdb/v2/reference/syntax/line-protocol/) messages — `measurement,tag=val field=val timestamp`. Each Kafka message produces a single Grafana frame in **long format**, one row per LP field, with a fixed streaming-friendly schema: `Time | _measurement | _field | value | value_str | | offset`. This shape works correctly with Grafana Live streaming (consistent schema per channel), and you can pivot it into Influx-style per-series frames with Grafana's **Transform → Partition by values** on `_measurement` + `_field` for dashboards that were built around the InfluxDB datasource. The **Timestamp Precision** dropdown decides how inline timestamps are interpreted; `Auto-detect` picks ns/µs/ms/s from the magnitude. - **Can I use this with Grafana Alerting?** Yes. Alert rules evaluate a snapshot of recent messages (see Grafana Alerting above). Use Reduce → Last on a numeric field. Dashboard live streaming is unchanged. - **How do I generate test data?** Use the included Go or Python producers. - **Where do I find more help?** See this README or open an issue. ## Documentation & Links - [Sample Producers](https://github.com/hoptical/grafana-kafka-datasource/blob/main/example/README.md) - [OAuth 2.0 Authentication](https://github.com/hoptical/grafana-kafka-datasource/blob/main/docs/OAUTH2_AUTH.md) - [Kerberos Authentication](https://github.com/hoptical/grafana-kafka-datasource/blob/main/docs/KERBEROS_AUTH.md) - [Performance Optimizations](https://github.com/hoptical/grafana-kafka-datasource/blob/main/docs/PERFORMANCE_OPTIMIZATIONS.md) - [Changelog](https://github.com/hoptical/grafana-kafka-datasource/blob/main/CHANGELOG.md) - [Contribution Guidelines](https://github.com/hoptical/grafana-kafka-datasource/blob/main/CONTRIBUTING.md) - [Code of Conduct](https://github.com/hoptical/grafana-kafka-datasource/blob/main/CODE_OF_CONDUCT.md) - [Articles Series](https://hamedkarbasi.com/series/grafana-kafka-datasource-guides) --- ## Support & Community If you find this plugin useful, please consider giving it a ⭐ on GitHub or supporting development: [![Buy Me a Coffee](https://img.shields.io/badge/buy%20me%20a%20coffee-donate-yellow?logo=buy-me-a-coffee\&style=flat)](https://www.buymeacoffee.com/hoptical) For more information, see the documentation files above or open an issue/PR.