RabbitMQ Grafana Datasource Plugin

The RabbitMQ Streaming Datasource plugin for realtime data updates in Grafana Dashboards.

What is RabbitMQ Stream?

If you are not fimiliar with RabbitMQ Stream Plugin, here are some good resources to read from about it:

• RabbitMQ official page about streams
• RabbitMQ Stream Core vs Stream Plugin
• RabbitMQ Stream Practical Guide Part 1
• RabbitMQ Stream Practical Guide Part 2
• RabbitMQ Stream Practical Guide Part 3

Getting Started

Reqirements

• RabbitMQ v3.12.10+ with rabbitmq_stream plugin enabled (should work on RabbitMQ v3.9+ but never was tested)
• Grafana v9.4.3+

Note: This is a backend plugin, so the Grafana server should've access to the RabbitMQ broker.

Configuration Editor - Configure your RabbitMQ Stream

Add a data source by filling in the following fields (TIP: if you are not sure some field means just leave it with the default value).

Connection

Basic connection section to connect the RabbitMQ.

Authentication

Basic authentication section to connect the RabbitMQ.

Stream Settings

Stream decleration settings section, where you define the settings of your wanted RabbitMQ stream and the settings of its consumer.

Exchanges

Optional Section. Array section for multiple exchanges that should be created in the RabbitMQ (they will also be recreated in case of connection lost to the RabbitMQ / Stream deletion in the RabbitMQ). If the exchange already exists in the RabbitMQ, the plugin will not recreate the exchange but if the same exchange name already exists in the RabbitMQ with different settings that you gave, you might encounter some problems creating the RabbitMQ Datasource connections.

Bindings

Optional Section. Array section for multiple bindings that should be created in the RabbitMQ (they will also be recreated in case of connection lost to the RabbitMQ / Stream deletion in the RabbitMQ). If the binding already exists in the RabbitMQ, the plugin will not recreate the binding. There is a support of binding of exchange to queue (or a stream) type of binding AND binding of exchange to exchange.

Query Editor

Please pay attention that there is no real query editor in the RabbitMQ plugin. Once you set the datasource settings you ready to go. You can still change the time range query in the default Grafana query editor which will impact what data is being showen and how fast the query interval is. This plugin was planned and deisgned to work with the Plotly by nline panel plugin. Feel free to check this awesome plugin (and if you wish not to use plotly you can still use transformations together with this RabbitMQ plugin).

Reconnecting to the RabbitMQ

The plugin handle most chaos scenarios automatically:

• If the stream is deleted in the RabbitMQ itself - the stream, its consumer, and the pre-configured exchanges and bindings will be recreated.
• If the Grafana is down once it goes up again, the consumer will be recreated.
• If the RabbitMQ is down the plugin will try to keep reconnecting to the RabbitMQ (it will only stop if the datasource is deleted by the user).

Important Note about the Deletion of RabbitMQ Datasource

The consumer of the stream is created once the user created a panel of the RabbitMQ datasource. If the user decides to remove the RabbitMQ Datasource before the consumer was created, the stream, exchanges and bindings that were created from the RabbitMQ Datasource will still exists in the RabbitMQ even if you set them to be disposed. So if you wish for them to be deleted, you must do it manually. If the consumer was already created, the stream, exchanges and bindings will be disposed (deleted) based on the settings of the RabbitMQ datasource you had set.

Known limitations

• This plugin does not support advanced TLS Configuration for RabbitMQ connections.
• This plugin only supports JSON based messages that and throws away any non JSON message. The JSON can contain numbers, strings, booleans, and JSON formatted values. Nested object values can be extracted using the Extract Fields transformation (or being processed by the Plotly by nline panel plugin).
• This plugin automatically attaches timestamps to the messages when they are received. Timestamps included in the message body can be parsed using the Convert field type transformation. The key name of the added timestamp is: RmqMsgConsumedTimestamp
• Currently the only supported offsets are First and Last (no specific offset or timestamp support when consuming messages from the RabbitMQ streams).

Known Errors and Causes When Trying To Connect To RabbitMQ

• dial tcp <ip/hostname>:<port>: connect: connection refused
  • There is no RabbitMQ up with the given <ip/hostname>.
  • Wrong Stream Port.

• timeout 10000 ms - waiting Code, operation: commandPeerProperties
  • The Stream Plugin is not enabled in the RabbitMQ.
  • Wrong AMQP Port.
  • Wrong VHost.

• There could be more errors, but the rest of them are pretty intuitive.

Credits

Creating the actual Streaming Backend Plugin

The streaming feature in Grafana is still experimental and the documntation to how to create a streaming backend datasource is lacking. There are not a lot of plugins that support this feature yet, so in order to create this RabbitMQ Datasource plugin I read and used some parts from all of the existing Grafana Streaming Datasources:

• MQTT Datasource (by GrafanaLabs)
• Kafka Datasource (by hamedkarbasi93)
• Websocket Backend Datasource Example (by GrafanaLabs)
• Websocket Datasource (by Golioth)

Speical credit for the framer.go which was copied from the MQTT Datasource and modified a little bit.

Communication with the RabbitMQ

The plugin is using 2 main core packages to be able to communicate with the RabbitMQ, both by the RabbitMQ core team:

• amqp091-go
• rabbitmq-stream-go-client

Contributing

Thank you for considering contributing! If you find an issue or have a better way to do something, feel free to open an issue or a PR.

License

This repository is open-sourced software licensed under the Apache License 2.0.