Plugins 〉Rabbitmq


Developer
maormil

Sign up to receive occasional product news and updates:



Data Source
community

Rabbitmq

  • Overview
  • Installation
  • Change log
  • Related content

RabbitMQ Grafana Datasource Plugin

The RabbitMQ Streaming Datasource plugin for realtime data updates in Grafana Dashboards.

What is RabbitMQ Stream?

If you are not fimiliar with RabbitMQ Stream Plugin, here are some good resources to read from about it:

Getting Started

Reqirements

  • RabbitMQ v3.12.10+ with rabbitmq_stream plugin enabled (should work on RabbitMQ v3.9+ but never was tested)
  • Grafana v9.4.3+

Note: This is a backend plugin, so the Grafana server should've access to the RabbitMQ broker.

Configuration Editor - Configure your RabbitMQ Stream

New RabbitMQ Datasource

Add a data source by filling in the following fields (TIP: if you are not sure some field means just leave it with the default value).

Connection

Basic connection section to connect the RabbitMQ.

Connection Section

FieldTypeIs RequiredDefault ValueDescription
HoststringYes"localhost"Hostname (or the IP) of the RabbitMQ server
AMQP PortintYes5672The AMQP port of the RabbitMQ server
Stream PortintYes5552The stream port of the RabbitMQ server
VHoststringYes"/"The virtual host the RabbitMQ server

Authentication

Basic authentication section to connect the RabbitMQ.

Authentication Section

FieldTypeIs RequiredDefault ValueDescription
TLS ConnectionboolYesfalseShould use TLS to connect to the RabbitMQ server
UsernamestringYes"guest"Username to connect to the RabbitMQ server
PasswordstringYes"guest"Password to connect to the RabbitMQ server

Stream Settings

Stream decleration settings section, where you define the settings of your wanted RabbitMQ stream and the settings of its consumer.

Stream Settings Section

FieldTypeIs RequiredDefault ValueDescription
Should Dispose StreamboolYestrueShould delete this stream (in the RabbitMQ) when the RabbitMQ datasource is deleted
Stream NamestringYes"rabbitmq.stream"The stream name that will be created
Consumer NamestringNo""The consumer name that will be created
Offset from StartboolYestrueShould the consumer consume messages from the start or the end of the stored messages in the stream
Max AgeintYes3,600,000,000,000The max age of messages in the stream in nano-seconds (set to 0 to disable the max-age limit)
Max Length BytesintYes2,000,000,000The max length of messages in bytes in the stream (set to 0 to disable the max-length-bytes limit)
Max Segment Size BytesintYes500,000,000The max segment size in bytes in the stream
CRCboolYesfalseWhen CRC control is disabled, the perfomance is increased

Exchanges

Optional Section. Array section for multiple exchanges that should be created in the RabbitMQ (they will also be recreated in case of connection lost to the RabbitMQ / Stream deletion in the RabbitMQ). If the exchange already exists in the RabbitMQ, the plugin will not recreate the exchange but if the same exchange name already exists in the RabbitMQ with different settings that you gave, you might encounter some problems creating the RabbitMQ Datasource connections.

Exchanges Section

FieldTypeIs RequiredDefault ValueDescription
Should Dispose ExchangeboolYestrueShould delete this exchange when the RabbitMQ datasource is deleted
Dispose if UnusedboolYestrueDelete this exchange only if it doesn't have bindings (and if 'Should Dispose Exchange' is set ON)
Exchange NamestringYes"rabbitmq.exchange"The exchange name that should exist in the RabbitMQ
Exchange TypestringYes"fanout"The exchange type
Is DurableboolYestrueShould the exchange be durable
Is Auto DeletedboolYesfalseShould the exchange be auto deleted
Is InternalboolYesfalseShould the exchange be internal
Is NoWaitboolYesfalseShould the exchange be noWait

Bindings

Optional Section. Array section for multiple bindings that should be created in the RabbitMQ (they will also be recreated in case of connection lost to the RabbitMQ / Stream deletion in the RabbitMQ). If the binding already exists in the RabbitMQ, the plugin will not recreate the binding. There is a support of binding of exchange to queue (or a stream) type of binding AND binding of exchange to exchange.

Bindings Section

FieldTypeIs RequiredDefault ValueDescription
Should Dispose BindingboolYestrueShould unbind when the RabbitMQ datasource is deleted
Is Queue BindingboolYestrueShould binding be from Exchange to queue/stream (if disabled, the binding will be from exchange to exchange)
Sender NamestringYes"rabbitmq.exchange"The exchange to bind from
Routing KeystringYes"/"The routing key to bind between the sender exchange and the receiver
Receiver NamestringYes"rabbitmq.stream"The stream/queue/exchange to bind to
Is No WaitboolYesfalseShould binding be noWait

Query Editor

Bindings Section

Please pay attention that there is no real query editor in the RabbitMQ plugin. Once you set the datasource settings you ready to go. You can still change the time range query in the default Grafana query editor which will impact what data is being showen and how fast the query interval is. This plugin was planned and deisgned to work with the Plotly by nline panel plugin. Feel free to check this awesome plugin (and if you wish not to use plotly you can still use transformations together with this RabbitMQ plugin).

Reconnecting to the RabbitMQ

The plugin handle most chaos scenarios automatically:

  • If the stream is deleted in the RabbitMQ itself - the stream, its consumer, and the pre-configured exchanges and bindings will be recreated.
  • If the Grafana is down once it goes up again, the consumer will be recreated.
  • If the RabbitMQ is down the plugin will try to keep reconnecting to the RabbitMQ (it will only stop if the datasource is deleted by the user).

Important Note about the Deletion of RabbitMQ Datasource

The consumer of the stream is created once the user created a panel of the RabbitMQ datasource. If the user decides to remove the RabbitMQ Datasource before the consumer was created, the stream, exchanges and bindings that were created from the RabbitMQ Datasource will still exists in the RabbitMQ even if you set them to be disposed. So if you wish for them to be deleted, you must do it manually. If the consumer was already created, the stream, exchanges and bindings will be disposed (deleted) based on the settings of the RabbitMQ datasource you had set.

Known limitations

  • This plugin does not support advanced TLS Configuration for RabbitMQ connections.
  • This plugin only supports JSON based messages that and throws away any non JSON message. The JSON can contain numbers, strings, booleans, and JSON formatted values. Nested object values can be extracted using the Extract Fields transformation (or being processed by the Plotly by nline panel plugin).
  • This plugin automatically attaches timestamps to the messages when they are received. Timestamps included in the message body can be parsed using the Convert field type transformation. The key name of the added timestamp is: RmqMsgConsumedTimestamp
  • Currently the only supported offsets are First and Last (no specific offset or timestamp support when consuming messages from the RabbitMQ streams).

Known Errors and Causes When Trying To Connect To RabbitMQ

  • dial tcp <ip/hostname>:<port>: connect: connection refused
    • There is no RabbitMQ up with the given <ip/hostname>.
    • Wrong Stream Port.

Exchanges Section

  • timeout 10000 ms - waiting Code, operation: commandPeerProperties
    • The Stream Plugin is not enabled in the RabbitMQ.
    • Wrong AMQP Port.
    • Wrong VHost.

Exchanges Section

  • There could be more errors, but the rest of them are pretty intuitive.

Credits

Creating the actual Streaming Backend Plugin

The streaming feature in Grafana is still experimental and the documntation to how to create a streaming backend datasource is lacking. There are not a lot of plugins that support this feature yet, so in order to create this RabbitMQ Datasource plugin I read and used some parts from all of the existing Grafana Streaming Datasources:

Speical credit for the framer.go which was copied from the MQTT Datasource and modified a little bit.

Communication with the RabbitMQ

The plugin is using 2 main core packages to be able to communicate with the RabbitMQ, both by the RabbitMQ core team:

Contributing

Thank you for considering contributing! If you find an issue or have a better way to do something, feel free to open an issue or a PR.

License

This repository is open-sourced software licensed under the Apache License 2.0.

Installing Rabbitmq on Grafana Cloud:

For more information, visit the docs on plugin installation.

Changelog

1.0.0

  • Initial release approved by Grafana Labs as a signed plugin.