Plugins 〉Kafka

Data Source
community

Kafka

  • Overview
  • Installation
  • Change log
  • Related content

Kafka Datasource for Grafana

License CI Release

The Kafka data source plugin allows you to visualize streaming Kafka data from within Grafana.

Reqirements

  • Apache Kafka v0.9+
  • Grafana v8.0+

Note: This is a backend plugin, so the Grafana server should've access to the Kafka broker.

Getting started

Installation via grafana-cli tool

Use the grafana-cli tool to install the plugin from the commandline:

grafana-cli plugins install hamedkarbasi93-kafka-datasource

The plugin will be installed into your grafana plugins directory; the default is /var/lib/grafana/plugins. More information on the cli tool.

Installation via zip file

Alternatively, you can manually download the latest release .zip file and unpack it into your grafana plugins directory; the default is /var/lib/grafana/plugins.

Configure the data source

Add a data source by filling in the following fields:

Basic fields

FieldDescription
NameA name for this particular AppDynamics data source
ServersThe URL of the Kafka bootstrap servers separated by comma. E.g. broker1:9092, broker2:9092

Query the Data source

To query the Kafka topic, you have to config the below items in the query editor.

FieldDescription
TopicTopic Name
PartitionPartition Number
Auto offset resetStarting offset to consume that can be from latest or last 100.
Timestamp ModeTimestamp of the message value to visualize; It can be Now or Message Timestamp

Note: Make sure to enable the streaming toggle.

kafka dashboard

Known limitations

  • The plugin currently does not support any authorization and authentication method.
  • The plugin currently does not support TLS.
  • Plugin is based on confluent-kafka-go, hence it only supports Linux-based operating systems as discussed in #6. However, we're cosidering changing the base package to support all operating systems.

This plugin supports topics publishing very simple JSON formatted messages. Note that only the following structure is supported as of now:

{
    "value1": 1.0,
    "value2": 2,
    "value3": 3.33,
    ...
}

We plan to support more complex JSON data structures, Protobuf and AVRO in the upcoming releases. Contributions are highly encouraged!

Compiling the data source by yourself

A data source backend plugin consists of both frontend and backend components.

Frontend

  1. Install dependencies

    yarn install
    
  2. Build plugin in development mode or run in watch mode

    yarn dev
    

    or

    yarn watch
    
  3. Build plugin in production mode

    yarn build
    

Backend

  1. Update Grafana plugin SDK for Go dependency to the latest minor version:

    go get -u github.com/grafana/grafana-plugin-sdk-go
    go mod tidy
    
  2. Build backend plugin binaries for Linux:

    mage build:backend
    

Contributing

Thank you for considering contributing! If you find an issue or have a better way to do something, feel free to open an issue or a PR.

License

This repository is open-sourced software licensed under the Apache License 2.0.

Learn more

Installing Kafka on Grafana Cloud:

For more information, visit the docs on plugin installation.

Changelog

1.0.0 (Unreleased)

Initial release.